lag investigation thread

[maroder]

Hey all,

mostly out of curiosity I tried to simulate a few different scenarios to find out what parts of the code contribute the most to the lag and should be considered first for future optimizations.

All this was tested with svn rev 25963 and about 500 to 600 units for each test.

TLDR: Most of the things I tested worked better than I thought, but the main problem on my system seems to be unit death (and potentially overkill) and range queries of attacks and especially of missed projectiles. Also: the current AI implementation: See https://code.wildfiregames.com/D3769

_____________________________________________________________________________________________

1st test: long range pathfinding with one type of unit

result for me: not that bad actually very little lag 1/10

----------------------------------------------------

2nd test: long range pathfinding with multiple unit types (graphic influence)

result for me: also good, very little lag 1/10

----------------------------------------------------

test 3: melee combat with 600 units |

result for me: noticeable lag 7/10

----------------------------------------------------

test 4: ranged combat with 600 units

result for me: without having a good measurement I would say the most lag so far 10/10

----------------------------------------------------

test 5: ranged combat with 600 units but emphasis on the projectile calculation

result for me: nearly no lag 1/10

----------------------------------------------------

test 6: ranged combat with 600 units but emphasis on the overkill factor

result for me: huge lag 10/10

----------------------------------------------------

test 7: gathering & short pathfinding

result for me: nearly no lag 1/10

----------------------------------------------------

test 8: BuildingAI

result for me: nothing 0/10

To gather all the information of this thread in one place:

see explanation by @wraitii to why this may be:

18 hours ago, wraitii said:

For what it's worth, I'm pretty sure that the speed difference is because missing arrows query units around them to try and find alternative targets, whereas successful arrows don't. The former is obviously much slower. Thus why it seems like 'overkill' is the problem, but it's in fact just that missing arrows are computationally expensive (and every arrow after the target has been killed misses).

You'd probably notice different results if you used an attack with splash, since splash does range queries anyways (as in, just much more lag all around lol).
See https://code.wildfiregames.com/source/0ad/browse/ps/trunk/binaries/data/mods/public/simulation/components/DelayedDamage.js$71

I think you're also compounding the results by having a massive blob of units nearby, since that probably slows down the C++ range query.

Overall, there is little easy gain to make here unless we got much smarter about immediate range queries.

 

Edited January 12, 2022 by maroder

[Freagarach]

One should replay the non-visual replays for consistent results, especially when one wants to investigate the pathfinder.

[Player of 0AD]

And why do ranged units lag the most? Is it because they calculate which target is the closest to them? Maybe they should just attack a random unit if that helps...

[maroder]

I updated the post

What I would conclude from all that (no guarantee on correctness) is that the main reason for lag (at least for me) is the overkill factor.

So the engine seems to dislike it if the health of a unit is so much reduced that it dies, but there are still other attacks incoming, which are now targeting an entity that is already dead. And this happens much more often when ranged units attack vs when melee units attack.

All other parts, be it graphics, short and long range pathfinder ect, seem not too bad from these tests.

Edited October 13, 2021 by maroder

[alre]

probably because they actually deal more attacks. also there's the random number generator for the spread and the rendering of the projectiles.

In any case, the attack/damage routines are those that suck the most resources it seems.

[Stan`]

One could rewrite them in C++ but with regards to moddability and amount of refactoring, it would be a no-no

[maroder]

Yeah, rewriting all that seems like a huge project, so I get that part of the argument, but personally I don't think the modability factor should play a role here (and I like to make mods).

The question is who wants to mod a game that is not functioning correctly? not doing optimization only because someday someone might want to change it seems to me like a bad tradeoff. If there are ways to optimize the javascript part that should of course be preferred, but if the only way to fix that would be to reduce some of the modability, then that is the sacrifice that has to be done (imo).

Edited October 13, 2021 by maroder

[maroder]

And the other question is: would it really need a complete rewrite? From what I saw I would say that that the problematic part is the unit death / attacking combination and not all of the attacking routine. But I also would really like to hear what conclusions other people draw from the videos

[real_tabasco_sauce]

Aside from re-writing code, one could still implement changes to reduce overkill (and therefore, lag) such as attack-ground. It would still be the player's decision whether or not to use attack-ground, but it would also be more effective in large battles due to less overkill.

However, this does looks laggy on its own right, maybe it just appears bad since its shown in 0.5x speed.

Attack-ground:

 

[real_tabasco_sauce]

@maroderIn your experiments, how much did map size influence lag?

[Stan`]

35 minutes ago, maroder said:

Yeah, rewriting all that seems like a huge project, so I get that part of the argument, but personally I don't think the modability factor should play a role here (and I like to make mods).

The question is who wants to mod a game that is not functioning correctly? not doing optimization only because someday someone might want to change it seems to me like a bad tradeoff. If there are ways to optimize the javascript part that should of course be preferred, but if the only way to fix that would be to reduce some of the modability, then that is the sacrifice that has to be done (imo).

The problem is that attack.js is linked to so many components that you would have to freeze them too (eg, health resistance unitai and possibly others) rendering the game a lot less moddable not just a bit. I'm hoping there is a better alternative but mayber there isn't.

 

What's slow when Units are dying are the onownership messages and another one. You can see it pretty clearly using profiler2. Making the messages faster might help a lot. Could even be threaded maybe.

 

[maroder]

36 minutes ago, real_tabasco_sauce said:

n your experiments, how much did map size influence lag?

did not test

[real_tabasco_sauce]

@maroder everyone online says large maps cause lag but I think thats just an assumption that game hosts have taken as fact for a long time.

[maroder]

I think it defenitly plays a role to some degree, but I posted already enough videos   feel free to investigate further tho.

[Kokomal]

On 13/10/2021 at 10:03 AM, maroder said:

Yeah, rewriting all that seems like a huge project, so I get that part of the argument, but personally I don't think the modability factor should play a role here (and I like to make mods).

The question is who wants to mod a game that is not functioning correctly? not doing optimization only because someday someone might want to change it seems to me like a bad tradeoff. If there are ways to optimize the javascript part that should of course be preferred, but if the only way to fix that would be to reduce some of the modability, then that is the sacrifice that has to be done (imo).

I agree

[wraitii]

On 13/10/2021 at 5:30 PM, maroder said:

So the engine seems to dislike it if the health of a unit is so much reduced that it dies, but there are still other attacks incoming, which are now targeting an entity that is already dead. And this happens much more often when ranged units attack vs when melee units attack.

For what it's worth, I'm pretty sure that the speed difference is because missing arrows query units around them to try and find alternative targets, whereas successful arrows don't. The former is obviously much slower. Thus why it seems like 'overkill' is the problem, but it's in fact just that missing arrows are computationally expensive (and every arrow after the target has been killed misses).

You'd probably notice different results if you used an attack with splash, since splash does range queries anyways (as in, just much more lag all around lol).
See https://code.wildfiregames.com/source/0ad/browse/ps/trunk/binaries/data/mods/public/simulation/components/DelayedDamage.js$71

I think you're also compounding the results by having a massive blob of units nearby, since that probably slows down the C++ range query.

Overall, there is little easy gain to make here unless we got much smarter about immediate range queries.

[maroder]

Thanks for the insights! That explains a lot.

38 minutes ago, wraitii said:

Overall, there is little easy gain to make here unless we got much smarter about immediate range queries.

Sad. Cause as the videos show, at least on my system it feels like this is the main reason for the lag.

41 minutes ago, wraitii said:

I think you're also compounding the results by having a massive blob of units nearby

you mean like in every fight situation in every game? I would say that's gold standard.

[wowgetoffyourcellphone]

Question. So, in a Total War battle, you can also have 100s of arrows in a volley, but have less lag. What are they doing differently? 

[Grapjas]

Do individuals specifically get hit and die by arrows in total war? Never looked that closely to it. Otherwise it's probably just calculating the entire battalion as 1 hitbox and lets random soldiers die when the health drops of the battalion entity.

[maroder]

might also be worth asking if @badosu has any knowledge or insight how the range queries/ attacking is handled over in BAR (and if it is also performance critical there)

[maroder]

17 hours ago, wraitii said:

Overall, there is little easy gain to make here unless we got much smarter about immediate range queries

After having thought about it a bit more: You're insights explain why ranged attack causes more lag then melee and why this is hard to improve, but even the melee attacking (and especially the unit death part) causes much lag. Do you think this could be a better area for improvements?

[Stan`]

The problem with the death part is the number of messages it generates. And all those messages need and are copied for the AI.

Sometimes sending those messages can take more than 500ms per frame.

See yesterday's discussion on #0ad-dev.

https://irclogs.wildfiregames.com/%230ad-dev/2022-01-11-QuakeNet-%230ad-dev.log

[wowgetoffyourcellphone]

12 hours ago, Grapjas said:

Do individuals specifically get hit and die by arrows in total war? Never looked that closely to it. Otherwise it's probably just calculating the entire battalion as 1 hitbox and lets random soldiers die when the health drops of the battalion entity.

Each soldier has his own hitbox in TW. Deaths occur per soldier with the soldier's stats (armor and life points) vs. the enemy's attack (attack value plus a dice roll). The battalion as a group only has high level things like morale, coherence, direction (affects armor), range, and stamina. 

[wraitii]

14 hours ago, wowgetoffyourcellphone said:

Question. So, in a Total War battle, you can also have 100s of arrows in a volley, but have less lag. What are they doing differently? 

I would assume they are much smarter about collision detection  

They probably have much more persistent data structures for collision detection in such a manner, because otherwise the game just wouldn't work. For example, they can probably first consider all formations (of which there are very few) to decide a new target to pick, where we must consider all individual units.

Or another example: for the most part, they don't have to consider new units joining the battle (I think this may be somewhat different in TW WarHammer but it remains limited). That probably enables them to use some clever hacks that we can't so easily use.

2 hours ago, maroder said:

After having thought about it a bit more: You're insights explain why ranged attack causes more lag then melee and why this is hard to improve, but even the melee attacking (and especially the unit death part) causes much lag. Do you think this could be a better area for improvements?

There is a lot of stuff happening in combat. If the target dies, UnitAI needs to choose a new target, which is by itself rather slow -> range query + filtering for preferred targets, sorting, etc. That's not even mentioning the fact that yes, dying by itself could be a bit slow.
This is indeed an area of possible improvements, and some work has been done (e.g. rP25102)

To be honest, the arrows range query could perhaps be improved by having some in-turn caching or some other logic to batch missile hits (so that multiple missiles landing in the same area don't do multiple range queries or something), but that doesn't sound entirely trivial to do without breaking in some edge-cases. It's always a question of code complexity, accuracy, edge-cases with these kind of optimisations.

---

Edit: Also, while such "in a vacuum" performance profiling is useful when you're working to optimise a specific function, be careful about generalising. In an actual MP replay, you'll most likely see a much more nuanced picture.

[badosu]

On 12/01/2022 at 5:09 AM, maroder said:

might also be worth asking if @badosu has any knowledge or insight how the range queries/ attacking is handled over in BAR (and if it is also performance critical there)

Am I correct in thinking that 0AD performs these calculations in JS?

I'm looking at how BAR handles it and asking engine people. But I think in BAR its C engine code, with hooks/callins to allow for customization.