Still working on this. So far I have implemented the following changes:
When a unit tries to move and is obstructed by another unit, instead of stopping and asking for a short-range path, it will just walk all the way up to the second unit. From there it will attempt to "slide" along the side of the other unit's obstruction square. This "sliding" allows units to walk around each other without having to invoke the short-range pathfinder, which means they don't have to stop and can keep walking. Units only ask for short-term paths if the "sliding" is failing to make progress (unit is stuck and cannot slide).
When a unit is following a long-range path, it only attempts to move parallel to the path instead of passing through every waypoint. This reduces the tendency for groups of units to travel single-file over long distances. Units will still follow short-range paths exactly, and will travel to the destination exactly.
The combination of these two changes allows groups of units to get moving faster, and as they move they group together much better instead of narrowing to a single-file line. It also improves performance, because the "sliding" dramatically reduces the number of calls to the short-range pathfinder. I tested performance in the "combat demo (huge)" scenario.
There are also some bugs where units get stuck - I've been working on figuring out how this happens and preventing it. It happens rarely.
There's another issue. The groups of units are still not sticking together closely enough for my liking. They still bump into each other and don't all start moving instantly. To make units move together as closely as Starcraft marines, it will be necessary to have the units in front of the group move first, to open a space so that the ones behind them can walk forward without bumping into them.
My plan for that is to let units wait for up to one other unit before they start moving. Here's a scenario illustrating how it would work.
Unit A wants to move
If A can freely move, it does so, and is finished for this turn.
If A is obstructed by a unit B, and B is moving but has not yet
moved this turn, then B remembers "A waits for B."
Then A is done for now, but will try again later.
Now unit B wants to move
If B can freely move, it does so. Then B remembers "A waits for B"
and tells A to try moving again.
Unit A tries to move again (on the same turn, because B told it
to try again).
If A can freely move, it does so.
If A still can't freely move because a unit C obstructs it, then
A just moves as far as it can. It will not wait for C; it may
wait for only one other unit per turn.
However, I don't know if this kind of detailed ordering is possible with the component message passing framework. There is an alternative, which would be to sort units by their distance from the goal, and have units that are closer to their goal move first. However, that has its own problems and might be slower because it's O(n log n) instead of O(n).