Problem is that they make very light work of melee and ranged units, and have a high damage resistance. They deal so much damage that they invalidate many units in their host civilizations. Also, they are cheap enough where training a huge amount of them is very reasonable.
With like 20 elephants, buildings fall very quickly too. Maybe even faster than using say battering rams