I believe that unarmed towers and fortresses without a garrison add more strategic planning to the game and you will have to find compromises for each important point on the map - send a garrison to the castle and weaken the invading army, or gather as many units as possible on a campaign, thereby exposing the rear. As for me, this will deepen the strategic component very much, and will also serve as a source of tantrums for the losers.