CallToArms unit action was meant to be usable in combination with the 'unit only' hotkey modifier. However it doesn't work anymore / never worked.
I find CallToArms to be extremely useful, but when your units start attacking buildings, that can be very annoying. Most players I know remap the most hotkeys possible to avoid units attacking buildings in any circumstances.
Here Is a mod I use to make CallToArms to by default attack units only.
CallToArmsUnitsOnly.zip
Note: