We had siege that wasn't OP. Catapults were weaker and it took forever for them to bring the Fortress down. That is without repairing it.
The trebuchets in AoE2 will bring the Castles down easily. Fort's job is to slow down the enemy, and give you the advantage. Fort shouldn't stop the push, only delay it.