A fully upgraded swordsman champion still does 1 damage per hit to a CC. It's a waste of time to damage buildings with units, unless you got mace men or axe men.
If I remember correctly, we had a change in A27 community mod that lowered building armor. Why didn't that change stick?
So, either do one or the other. Some people want to see buildings razed to the ground without going to the trouble of making siege rams.