Late-game Lag

[Crynux]

Hi all, so lag has been a topic for a bit. I noticed in my rounds that in the beginning, I usually get around 60fps, with everything maxed. Late game can dip to about 15-20 fps (my most recent game did so; 'normal' map, 2 player vs 2 AI).

Anyways, I decided to look into this. First, I went into the scenario editor, and spammed a bunch of houses and units on the map. Opened it up in-game, and there was very little difference. So then, I heard you can 'pit' AI's against each other. So I setup a tiny map, 4 AI's, and let'r rip on 20x. For the most part it was "OK".

Then ... I slowed it down, because I noticed the performance dipping. To figure out what was doing this, I zoomed in as far as possible, and moved around the map. I quickly noticed that in the main city-centers of the AI's that weren't attacked, they were getting ~75fps (unlocked). However in the battleground ... it dipped to below 20fps.

That made me think - maybe there's something we're not seeing... and that's the unit decay.

So after some testing, it looks like the unit decay rate (at least for Slingers [I tested with those units only]) may be too low.

The defaults that I can see in the template_unit.xml are as follows:

    <DelayTime>30.0</DelayTime>
    <SinkRate>0.01</SinkRate>
    <SinkAccel>0.0</SinkAccel>

In game, a depth is calculated based on some bounding box (I assume unit height? This may be inaccurate as when a unit is 'dead' they're shorter than when standing - not sure if it accounts for that). That number happens to be 3.877156. With that, at a rate of 0.01, and 60fps, it'll take ~3.87 minutes for the unit to decay and be removed (not including the 30 second delay).

Taking that into account, for the unit I tested with, to re-create one of them it takes 10 seconds. To create a batch of 5, it takes 30 seconds. So potentially in the time it takes 1 unit to decay, you could have replaced it with 38 - that's if you used only one barracks. With 2+ players and multiple barracks/unit types ... this can become a problem quickly.

So my thinking is - in late game, part of the lag is due to all of these 'decaying' units all over the ground, which seems to only increase as time passes.

Anyways, is there a reason for such a long decay time? I'm thinking to speed things up - it'd be good to either increase the SinkAccel value, or increase the SinkRate in general. Another option would be to have some code that could dynamically increase the SinkAccel or SinkRate values based on how many units are on the field. On a side note, it may be useful to have a setting in the UI as well, since if it does slow so much as I've seen - then it'd be good to turn it off in some cases, or at least have a manual throttle for it.

Thoughts?

Thanks in advance,

Crynux

 

[wowgetoffyourcellphone]

Interesting analysis , but I really doubt lag comes from decaying dead units. It's most likely that it's pathfinding. Late game has tons of units, so pathfinding calculations grow exponentially.

[Crynux]

It does add some degree of lag - mainly in the form of rendering. In my tests, I could be on one side of the map getting 75fps, and on the other where there's a bunch of dead units, I'd get 20fps. So even if pathfinding does contribute to lag in late game, the increased render load due to having to render so many corpses (even when underground?) seems to have a negative effect.

[Stan`]

@Crynux There is no real justification for the number itself. The only part where it may be important is to know there has been a battle. To see potentially which of your enemies is the weakest.

One of the reason of the lag might be the blood decals which alpha transparency which will have a performance hit. To test that theory just download the no blood mod.

[Crynux]

1 minute ago, Stan` said:

@Crynux There is no real justification for the number itself. The only part where it may be important is to know there has been a battle. To see potentially which of your enemies is the weakest.

One of the reason of the lag might be the blood decals which alpha transparency which will have a performance hit. To test that theory just download the no blood mod.

Interesting, I'll have to give that a try. At the moment I'm testing with the decay rates turned up a bit, and all the same... will see how it performs; large map, 4 AI's, unlimited units.

[fatherbushido]

@Crynux: see

https://code.wildfiregames.com/rP21732 (I didn't find the related diff proposal)

https://code.wildfiregames.com/rP21783 (see https://code.wildfiregames.com/D1467 for the work before)

Edited May 22, 2019 by fatherbushido

[Crynux]

Hmmm, so there seems to be a bit of an improvement, there's still the usual unit stutter every now and then, but that might be the pathfinding lag. Overall I get a better fps that before it seems, spikes of 100fps, and lower-end values of 40 or so fps in 'populated' areas. I'll have to give it a try tomorrow in an actual game. I haven't taken a look at that mod yet ... I'll do that soon after.

On a side note, I noticed in-game there's quite a few arrows on the ground (due to archers and such); so I figured I'd take a look at an arrow mesh to see how many triangles were in it; looking at props/weap_arrow_front.dae in blender, it looks like it has 62 triangles. I'm not a 3d modeling person by any means, but is that high for such a small prop? It just gets me thinking - these slowdowns I'd imagine are a result of the renderer being somewhat outdated, so any triangles we can trim would help. My concern with the arrow is that if you get a good few archers in some area, they can spawn triangles quite quickly.

5 minutes ago, fatherbushido said:

@Crynux: see

https://code.wildfiregames.com/rP21732 (I didn't find the related diff proposal)

https://code.wildfiregames.com/rP21783 (see https://code.wildfiregames.com/D1467 for the work before)

Very interesting. Thanks for the links, skimmed there quickly while writing this response ... but it looks like more justification is needed? Maybe I'll come up with some tests of sorts.

Thanks again,

Crynux

[wraitii]

With things like arrows, I think triangles are a lesser concern than draw calls. 60 polygons is nothing in 2019, but 60 draw calls is something.

Reducing the # of polygons would help, possibly after instancing of some kind, but it won't fundamentally speed things up right now imo.

[fatherbushido]

6 minutes ago, Crynux said:

My concern with the arrow is that if you get a good few archers in some area, they can spawn triangles quite quickly.

It's even worse if you have bad archers as all the projectiles will remain on the ground if they "miss" their (or a) target.

6 minutes ago, Crynux said:

but it looks like more justification is needed?

Yes! iirc they did that before alpha23 release so it was a bit done on the fire.

[Crynux]

9 minutes ago, wraitii said:

With things like arrows, I think triangles are a lesser concern than draw calls. 60 polygons is nothing in 2019, but 60 draw calls is something.

Reducing the # of polygons would help, possibly after instancing of some kind, but it won't fundamentally speed things up right now imo.

Hmmm yeah that's true, instancing would help it seems in a lot of cases. I was just thinking that in the meantime, if we could reduce anything to increase render speed, it would help.

On that note (models), are there any kind of stats on how many triangles we have per model 'class'? Like is there an average polygon or triangle count for 'normal' units like slingers, or one for 'houses', 'civic-centers', etc? It might be something valuable to have - then we could keep the counts within a certain range based on classification... a 3d resolution for lack of a better term.

10 minutes ago, fatherbushido said:

It's even worse if you have bad archers as all the projectiles will remain on the ground if they "miss" their (or a) target.

Yes! iirc they did that before alpha23 release so it was a bit done on the fire.

Yeah lol the case I was looking at specifically were a few archers that just seemed to fling the arrows wherever, they didn't even try... just a pile of arrows on the ground.

As for justification - I think it's less the actual decaying, and more the implications of having the decay there. So like, having 230 units, and then killing all 230 units... it's the same amount of polys/tris when alive vs decaying, so there isn't much to be saved there. It's when you have 4 minutes of n units hiding underground, with the next wave of units fighting on top of it - then you get the performance decrease. I guess that could be called replacement rate? The units are replaced faster than they can decay, so it steadily increases unless battle stops entirely ... which usually never happens late game, unless someone gets destroyed.

I mean, I like seeing the corpses there ... it makes the game feel more realistic, but I'd rather see no corpses than be playing a slideshow.

If anything - I think the idea of having a slider in the Graphics menu or something - to multiply the decay by some factor, manually set the duration, or just disable it entirely, would be of value.

On a side note, I just had another thought. Does anyone remember playing older shooters, where you'd shoot a wall for example, and the bullet holes would only 'trail' so far? Maybe we could do something like that - if your unit cap is 200 units, then you should only really be allowed to have 200 or so decaying.

Along the same lines, I remember in those shooters as well, when you turned away, and looked back, sometimes the decals on the wall for the bullet holes would disappear. I guess it's a more harsh approach - as you'd only see corpses in-view, but if anything it's a possible (probably not favorable) solution.

Thanks again!

-Crynux

[Guest]

Or just redesign the decaying system. The plan was to remove the corpses slowly. Like the real world. There are far better ways of doing that.

[Stan`]

55 minutes ago, Crynux said:

 

On that note (models), are there any kind of stats on how many triangles we have per model 'class'? Like is there an average polygon or triangle count for 'normal' units like slingers, or one for 'houses', 'civic-centers', etc? It might be something valuable to have - then we could keep the counts within a certain range based on classification... a 3d resolution for lack of a better term.

The word you are looking for is polycount (though triscount would be more accurate)

You can find some guidelines here 

https://trac.wildfiregames.com/wiki/ArtPolyCountGuidelines