In the original game (the hardtype game, mind you), Dark Elf and Evil Wall always gave me a hard time. (Though I thought the trap doors were alot tougher then him...) He's pretty easy, now, after the countless times I've played through the game though.
Yeah, in FFIVA they nerfed the difficulty, like with DoS. Even the optional bosses really aren't that hard. (They have a noticeable attack pattern, which you can use to your advantage.) Though it took me a few times to beat Lunar Odin and Lunar Ifrit.
Of course, once you stock up on Hero's Shields...