if your opponent has 20 units when he attacks you and you only have 5 units to defend, then that means that your build is unsafe. you should be using a build that has more units out to defend when his attack hits.
You have defender's advantage to work with as well, since your units have no distance to travel once they're finished being trained. We also have the border mechanic in 0 AD, which prevents people from building forward bases too early in the game; this acts as a secondary defensive mechanism. On top of that, soldiers which are trained for defense can be tasked to gathering resources*, so you don't even lose out on economy if you make defensive units.
If, on top of all the defense mechanisms I just talked about, you also have very strong buildings (CCs and towers), then a match of 0 AD quickly becomes a match of "who can take the map with CCs the fastest", which isn't actually very fun. It's the primary reason I don't play the game as much as I did when I discovered it. 0 AD as of right now is all about taking the map with buildings that are much more cost-efficient than actual units. It's not very interesting no matter how you look at it.
That said, units being vulnerable even when they are garrisoned in buildings is a mechanic that helps alleviate that problem. A few more nerfs to CCs and towers and the game's design will become quite sound.
*I actually have an issue with the fact that the unit rank upgrade is now gone. before, it meant that if you wanted units which were strong in combat, you had to sacrifice your economy. so if you wanted to rush, you would get the upgrade asap, if you wanted to play defensively and economically heavy, it made sense to wait until you could safely support a house boom or caravan boom before getting that upgrade. now, that line is kind of gone and i'm not sure if that's good or not, since it makes defensive, economic play the "best", on paper anyway. well, this would take play testing on my part to confirm, just wanted to throw this thought out there.