40 sword cavalry costs a lot of money. They should be able to take down building pretty fast. If you've let the enemy get to your civ center with 40 sword cavalry unopposed, then you deserve to lose that civ center. The balance is all kind of screwed up right now with units, but 40 melee cavalry quickly destroying a civ center is not imbalanced itself. It should be easier for, say, spear infantry to kill those cavalry though. With hard counters removed this is difficult right now.