A lot of players do this trick :
Garrison rams in a building (fort most likely but possible to make it from workshop)
Ungarrison them on approaching siege, then just after they land a hit, garrison them to safety.
It's annoying because you have no way to react fast enough to kill the trickster ram. You can still :
Surround your own rams with enough soldiers so that the opponent rams won't be able to unload too close to them.
Surround your own rams with melee on stand-ground stance (need less then for 1 but also less safe for you own rams, you might be able to kill enemy rams though).
Give up on ramming the fort and find an alternative (catapults, capture the fort, ignore the fort..).
The odds that your opponent use anything else then the default garrison hotkey to do this are extremely low.