Re-Release A27 or RC builds?

[Stan`]

@real_tabasco_sauce did you try the rc1 before the sm update ?

[real_tabasco_sauce]

yeah "rc1" in those profiles is the one @Gurken Khan shared.

[Stan`]

That means the problem happened between A26 and the RC1. Do you experience the stutter with rC1 ?

[guerringuerrin]

46 minutes ago, Acero said:

Devs (with the help of players like guerrin and seleucids) seem to be developing and testing patches that are promising in terms of getting back some performance (full hashes, dead bodies).

But let's not forget the elephant in the room: why did performance get worse for so many in this alpha?

Correcting inefficiencies present in A26 as well will not explain what happened in this alpha.

for the record: my only contribution to this has been making myself available to @real_tabasco_sauce, who has been carrying out testing tasks on these potential patches for quite some time and guiding volunteers through the process of compiling patched versions of the game to be tested in a controlled environment.

I believe the best approach in a situation like this is to keep bringing in more volunteers so we can conduct 4v4 performance tests in controlled environments while ensuring everyone uses the same client. The real challenge is reaching an agreement and working as a team because jumping to conclusions and spreading "performance solutions" that may not work as expected will only lead to more frustration.

[real_tabasco_sauce]

it still shows up in profiles with the same magnitude. here is a 4v4 on foothills from a26

I'm going to watch this replay on normal speed and see if any stutters coincide with jumps in state hash check. For the record, in a27, I have seen the state hash check value surpass 20MS in the f11 profiler, which I have not yet seen in a26 or the rc

[Stan`]

1 minute ago, real_tabasco_sauce said:

it still shows up in profiles with the same magnitude. here is a 4v4 on foothills from a26

I mean does it show up as stuttering while it didn't in A26 or not. The question is whether there might be something else hiding

[real_tabasco_sauce]

ok yeah I can already tell that they still occur in a26 and that they still disrupt the GUI. Its easiest to tell by panning. Indeed, even early on, there is a slight freeze every 4 sec.

state_hash_check is unlogged in a26, but I can see 4MS jumps coinciding with the stutter every 4 sec. I'll see how large they get

[Stan`]

Another interesting thing might be to drop them in a special a26 version and see if the game still performs better.

[real_tabasco_sauce]

14 minutes ago, Stan` said:

Another interesting thing might be to drop them in a special a26 version and see if the game still performs better.

maybe, from the above, I am starting to think the bottleneck theory is pretty well supported.

[Stan`]

1 minute ago, real_tabasco_sauce said:

maybe, from the above, I am starting to think the bottleneck theory is pretty well supported.

Indeed. But it's a bit too good to be true. E.g. it doesn't explain this https://gitea.wildfiregames.com/0ad/0ad/issues/7589#issuecomment-112230

But that's the only SP report.

[real_tabasco_sauce]

2 minutes ago, Stan` said:

Indeed. But it's a bit too good to be true. E.g. it doesn't explain this https://gitea.wildfiregames.com/0ad/0ad/issues/7589#issuecomment-112230

But that's the only SP report.

Interesting. We don't really have a comparison with a26, but seems like pretty bad performance.

@MetaPhyZic, @Atrik, and @Seleucids all mentioned low fps upon reaching p3/late game in addition or in combination with stuttering.

In my case, I have good FPS but notice the stutter. All of these players are on a linux distro while I play on windows. @0 calories what operating system do you use?

So maybe indeed there is more than one issue at hand. @Atrik even said that he gets to 3fps even with quick hash only. Maybe can @Seleucids and/or @Atrik profile a single player combat demo huge and compare between a26 and a27?

[Atrik]

9 minutes ago, real_tabasco_sauce said:

@Atrik profile a single player combat demo huge and compare between a26 and a27

I only have a snap install for a26 and seems like it makes using the profiler more complicated.

[0 calories]

41 minutes ago, real_tabasco_sauce said:

Interesting. We don't really have a comparison with a26, but seems like pretty bad performance.

@MetaPhyZic, @Atrik, and @Seleucids all mentioned low fps upon reaching p3/late game in addition or in combination with stuttering.

In my case, I have good FPS but notice the stutter. All of these players are on a linux distro while I play on windows. @0 calories what operating system do you use?

So maybe indeed there is more than one issue at hand. @Atrik even said that he gets to 3fps even with quick hash only. Maybe can @Seleucids and/or @Atrik profile a single player combat demo huge and compare between a26 and a27?

Linux

Let me test another game haven't had time yet... I compiled previously incorrectly  so i can report if improvement..

[Seleucids]

1 hour ago, real_tabasco_sauce said:

Maybe can @Seleucids and/or @Atrik profile a single player combat demo huge and compare between a26 and a27?

Sadly A26 doesn't build itself on Linux anymore. I am happy to try it on A27 though. 

Profiling before start (during loading etc)

 

There was a fair bit of stuttering and low fps once I reached about 400 units per side. 

commands.txt

profiler2.html  

 

 

Edited Wednesday at 20:34 by Seleucids

[Stan`]

@Atrik @Seleucids @0 calories @MetaPhyZic Would you be so kind as to upload your userreport_hwdetect.txt there might be a hardware pattern. IIRC for instance @Dunedan only has an IGPU no dedicated. Maybe the performance improvement affected dedicated GPUs and not IGPUs.

@Seleucids This is an SP game right ? So the stuttering cannot come from hash checks, since there aren't any.

Quote

 

I only have a snap install for a26 and seems like it makes using the profiler more complicated.

 

The snap will also run slower than the native package.

 

[Seleucids]

13 minutes ago, Stan` said:

This is an SP game right ?

This is a LAN game, because i can't get a large battle with just 1 player. Atlas doesn't work for me (crashes on start)

My user report is attached below. 

userreport_hwdetect.txt

I am using an integrated GPU (Intel Arc) and is not ideal. However, frame drops and stutter also occur on my much stronger machine running RTX 3060, except the minimum fps on there is 20 and there aren't complete GUI crashes. 

 

Looking at profiler2 results, the rendering is the most costly. Could we only render 1 texture layer, not all 3 materials? (just the .png file not the spec tex and AO tex norm tex etc)

Edited Wednesday at 20:56 by Seleucids

[Seleucids]

A video recording:

 

[Stan`]

10 minutes ago, Seleucids said:

This is a LAN game, because i can't get a large battle with just 1 player. Atlas doesn't work for me (crashes on start)

I have a combat demo huge auto map here -> https://github.com/0ADMods/trailer_tools/tree/main/maps/scenarios (You can also tweak the script to change the units or to make a bigger battle.)

Also if you want one player, you can just set the AI to unnasigned.

Quote

Looking at profiler2 results, the rendering is the most costly. Could we only render 1 texture layer, not all 3 materials? (just the .png file not the spec tex and AO tex norm tex etc)

IIRC just setting material quality to minimum will do that. Setting quality to lowest will also reduce the size of the textures by 8 or something.

 

[wraitii]

4 hours ago, Stan` said:

@wraitii why would the hash check freeze the UI ?

The hashing happens on the main thread so the CPU literally cannot do anything else. So it's a freeze of everything.

[Seleucids]

3 minutes ago, Stan` said:

IIRC just setting material quality to minimum will do that

I already have everything set to minimum. Could we prohibit it from the engine somehow?

[wraitii]

7 minutes ago, Seleucids said:

A video recording:

 

This video seems to show "regular sim lag" to me - I would expect A26 to behave similarly. If you upload the profile2 JSON file we could look more in-depth.

[Seleucids]

8 minutes ago, Seleucids said:

A video recording:

 

The profiler 2 excerpt is attached profile2.jsonphere:

 

[Stan`]

1 minute ago, Seleucids said:

I already have everything set to minimum. Could we prohibit it from the engine somehow?

I don't think it would make a big difference (Especially given you already replaced most of the textures in your mod). Do you have reason to believe the converse? The reason all meshes need all the maps now (though most of them use a very light 2x2 norm and spec), is because shader switching is expensive, and the game needed to switch many variants of the shame shaders depending on the texture maps. If you look at Godot or Unity (Godot is another FOSS engine) materials for instance, they all have all the maps. Since our engine is a bit different we're just a bit more explicit about it.

Another big performance hit is the entity variation. Many different units means many different textures to load and unload on the GPU. In your case, if you force all the entities to use the same textures (not identical ones, the same file) you might see improved performance.

 

 

 

[wraitii]

7 minutes ago, Seleucids said:

The profiler 2 excerpt is attached profile2.jsonphere:

I suspect the report is showing data at the end of the game when things are smoother. You should try clicking "save live report to file" in the middle of the game.

But based on that profiling, your render frames take exactly 16ms which is consistent with targeted 60 fps.
The "sim" frame every 200 ms take about 50ms, which should be potentially noticeable, but not dramatic. Breaking that down ,the "gui sim update" takes about 10ms, and the sim update is probably variable but 5-15ms is expected.

The full state hash check in that replay is 111ms which is definitely noticeable.

---

One possibility is that the lag is GUI lag. We've bumped the max selection size to 300 units I believe, that might affect lag?

[Seleucids]

15 minutes ago, wraitii said:

300 units I believe, that might affect lag?

If there are 300 units to select, then yes. However, in A26, I raised the selection size to 1000 with my own GUI mod and it was quite fine. Although it's to be tested when the area we are drawing over contains many allied and enemy units which will not be added to our controllable selection but nevertheless be considered by the engine.

Since you mentioned selection, there is something strange I noticed: if I draw the selection area from top left to bottom right, the selection works as usual. However, if I move my cursor from bottom right to top left, I cannot select anything. Not sure if this is related. 

profile2.jsonp

This file here is saved from the huge battle demo map during the middle of a huge fight, corresponding scene:

I must say, this setup felt much smoother than my previous LAN game, even though the units present have been doubled. 

 

Edited Wednesday at 21:32 by Seleucids