Re-Release A27 or RC builds?

[Acero]

15 hours ago, Itms said:

If A27 is not playable for some people, then it means it was even worse for them in A26. Indeed, full hashes were performed in A26 as well. The numerous performance improvements we released are just making that bottleneck prominent. It is alarmist and totally incorrect to insinuate that A27 would have higher spec requirements than A26.

I think this kind of comments from a leading developer is the reason why a bunch of old school players have the feeling that 0ad devs, which are almost never seen in the lobby, live a bit disconnected from reality and disregard players opinions.

Here Itms says in a dismissive tone and with categorial certainty that a27 MUST be more preformant than a26 given all the performance improvements released on the latest alpha. Guess what Itms:  For a lot of players a27 performs WAY worse than a26. And devs have been degrading 0ad perceived performance over the years even as computers have been getting more powerful. You should be asking yourself why, instead of denying reality.

While a handful of players reported an increase in performance, and some did not notice a change, i would say half or more feel the performance of 27 is worse in 4v4 games. Even veteran players and hosts like Said have expressed they lost a lot of motivation to play 4v4s anymore due to the extreme lag we experience in a27. Chrsgtr went out to say that he thinks this alpha is the one that will kill 0ad due to the lag.

Related question:

Is there any way i can install alpha23 on linux without recompiling? Any appimage or other source is available. I would love to do some tests, because i am pretty confident I will make some very interesting discoveries of how performance has evolved over the last 5 alphas. Alpha 23 was the last alpha that had 2 turns per second among other things, and the last alpha i remember ran decently well on rather older CPUs. I would appreciate if someone could point me to official executables for linux for alpha 23. I will prob not download a compilation from a random player tho probably, for security reasons. If the official executables are not available, i would also appreciate some instructions on how to compile myself.

Thanks.

[Seleucids]

2 minutes ago, Acero said:

i would also appreciate some instructions on how to compile myself.

Try what I just posted above with the A23 zips. I will try A23 tomorrow as well. 

[Stan`]

5 hours ago, Acero said:

 

Is there any way i can install alpha23 on linux without recompiling?

There is an appimage for it, but it will not work on recent AMD CPUs :/

https://www.appimagehub.com/p/1438982/

The big difference between A23 and A24 was the huge art update that happened during the the two years. Ideally we'd use the old horse an unit models as LOds for people with weaker configuration, but don't have the bandwidth.

[Feldfeld]

6 hours ago, Acero said:

For a lot of players a27 performs WAY worse than a26

I think it could help a lot if one, or multiple, of these players could bisect to try to pinpoint how the issue got introduced. I imagine it appeared in a very subtle way.

[Stan`]

Does anyone have a rc1 installer laying around from last year ? Could be a good starting point since it was before the Spidermonkey Update. That's the biggest thing that changed aside from Vulkan

[wraitii]

Quote

 

For me, my frame drops to 10 fps as soon as I reach P3. Then freezes during fights. 

Good news @0 calories, I managed to recompile A27 with the wraitii quick hash and it reduces lag to near A26 levels!

See, the thing is that this is inconsistent with what should happen, so I find it a little sus. My patch should basically remove the every-4s-freeze entirely, but this should happen mostly right from the start and just get slowly worse over the game. If it slows down during fights it's something else and the patch should not be changing that.

With regards to running with the patch - it's safe, but you will OOS right from the start and you then might miss out on actual OOS, or cheaters and such. So that's not ideal. We have been looking into it and there's really no way to make it compatible with A27 as released, unfortunately. So we need a patch release, and it's going to be incompatible, and that's all very annoying. But we're not a big team and we don't have a lot of free time - and we do this on our free time - so you'll have to take things in your own hand in the very short term.

6 hours ago, Acero said:

I think this kind of comments from a leading developer is the reason why a bunch of old school players have the feeling that 0ad devs, which are almost never seen in the lobby, live a bit disconnected from reality and disregard players opinions.

We'd like some help, actually. When you play MP games, please upload your commands.txt so we can replay and profile your games. Please upload your profile.txt so we can get some idea of what's slow. Please profile using profiler2. Please profile using perf or samply or something else. Please bisect. It's not that complicated, and if you need some help, at this point I think deepseek or chatgpt or mistral or llama or gemini or grok can help you out.

And don't try to pull rank by talking old school. I've been around longer than the lobby has.

[strat0spheric]

Yesterday i had 3 4v4 games. All of them ended up at around min 13 or 14, when the battle started with 1! fps for me and the rest of the players. Thats basically not playable. I would have been happy about 10 fps. Since a27 this is always the case. I had at least around 6 to 8 fps in a26 in comparable situations. PC: Razer blade stealth laptop on i7.

Here is my commands.txt of one of the games yesterdaycommands.txt

Edit: i have to figure out the profiler thing ...tbc

Edited Wednesday at 09:28 by strat0spheric

[Itms]

8 hours ago, Acero said:

I think this kind of comments from a leading developer is the reason why a bunch of old school players have the feeling that 0ad devs, which are almost never seen in the lobby, live a bit disconnected from reality and disregard players opinions.

Here Itms says in a dismissive tone and with categorial certainty that a27 MUST be more preformant than a26 given all the performance improvements released on the latest alpha. Guess what Itms:  For a lot of players a27 performs WAY worse than a26. And devs have been degrading 0ad perceived performance over the years even as computers have been getting more powerful. You should be asking yourself why, instead of denying reality.

I think this kind of comments from an unknown user is the reason why a bunch of old timers in the team have the feeling that 0ad players, especially those who have never written a line of code, live a bit disconnected from reality and disregard the major part of their free time (which is scarce in an adult's life) the devs dedicate here.

Here Acero says in a aggressive tone and with utmost disrespect for our work that devs MUST deliver a flawless (but still free) product on each and every alpha, fix every bug immediately while also finding time and motivation to play the game in a competitive multiplayer setting. Guess what Acero: that leaves me the choice between giving up OR avoiding interaction with players. And players have been degrading my motivation over the years as the expectations have been getting higher. You should be asking yourself why, instead of disparaging us.

 

Now I hope you understand why your tone is unacceptable Please delete your post which doesn't help, and I'll do the same with my bad taste parody.

Instead we would be grateful to have profiling data and numbers instead of vague feelings of performance degradation. I do not question the fact that there is a performance issue, but right now it is unexplainable. As wraitii points out, it is not possible that the hashing is the sole culprit if you maintain that A26 was objectively faster. We need to understand that issue before we can think of fixing it.

[Gurken Khan]

59 minutes ago, Stan` said:

Does anyone have a rc1 installer laying around from last year ?

It's from 2023, but here you go.

(I'll probably check later if it destroys my upload quota and delete it eventually...)

0ad-0.0.27-rc1-27645-alpha-win32.exe

[Stan`]

@real_tabasco_sauce Could you take this version for a spin see how it feels ?

[Seleucids]

1 hour ago, Itms said:

As wraitii points out, it is not possible that the hashing is the sole culprit if you maintain that A26 was objectively faster. We need to understand that issue before we can think of fixing it.

Lets think back to changes to the engine and the public mod between A26 and A27. Which additional features could introduce extra load during game session? 

I plan to do these 2 experiments: 

1. Run A27 public mod with A26 engine

2. Run A26 public mod with A27 engine

The difficult part is there were many changes in function names and that might cause many compatibility issues.

I personally think that public mod is unlikely to be the culprit, but I stand corrected. 

Pathfinder is usually a big contributor to lag.

[guerringuerrin]

9 hours ago, Acero said:

Is there any way i can install alpha23 on linux without recompiling? Any appimage or other source is available. I would love to do some tests, because i am pretty confident I will make some very interesting discoveries of how performance has evolved over the last 5 alphas. Alpha 23 was the last alpha that had 2 turns per second among other things, and the last alpha i remember ran decently well on rather older CPUs. I would appreciate if someone could point me to official executables for linux for alpha 23. I will prob not download a compilation from a random player tho probably, for security reasons. If the official executables are not available, i would also appreciate some instructions on how to compile myself.

Thanks.

I've sent you a .deb file from the official Debian repositories via PM. It was just a four-word Google search. I can't say whether you'll run into the same issue that @Stan` pointed out with the AppImage, but you might at least face some outdated dependency problems.

When RC1 was released, I offered to help you with the compilation, which is a relatively straightforward task, but you chose to wait for an AppImage or the final version. It would have been valuable to have you involved in the tests.

[0 calories]

13 hours ago, Seleucids said:

Originally they did:

if (turn % 20 == 0)
        return true;

Which means 1 full has per 20 turns. We can do maybe turn % 300 == 0 which means 1 full hash check per minute. 

• return true = do full hash
• return false = don't do full hash

 

For everyone else who want to do this or play with us:

The OOS is false alarm. The error is thrown because everyone else is doing a full hash at 20 turns, but we did not respond to their check request. The error does not affect gameplay and can be royally ignored. Just play on and the simulation will be in sync because it is deterministic in nature and player commands are synced over network. 

 

I personally have noticed significant improvements, especially in early game and in large fights. However, it doesn't solve everything because other parts of simulation are still poorly optimised. 

Is it in this file? + recompilation?

 

# grep -R "turn % 20 == 0" 
simulation2/system/TurnManager.cpp:    if (turn % 20 == 0)

 

I will try it, if it gives me normal responses I'm ok temporary using it this way, so others will just ignore OOS. 

[Stan`]

1 hour ago, Seleucids said:

Pathfinder is usually a big contributor to lag.

Using profiler2 you can see the time spent in the pathfinder.

3 minutes ago, 0 calories said:

s it in this file? + recompilation?

 

# grep -R "turn % 20 == 0" 
simulation2/system/TurnManager.cpp:    if (turn % 20 == 0)

Yes.

[0 calories]

I took one team game and does not have feeling it would be helpfull - still getting loss in FPS and also GUI responses so slow.

 

How can I make sure my recompilation really using 300 insteead of 20?

[Stan`]

13 minutes ago, 0 calories said:

I took one team game and does not have feeling it would be helpfull - still getting loss in FPS and also GUI responses so slow.

How can I make sure my recompilation really using 300 insteead of 20?

You could check the date at which it was compiled  

[Acero]

6 hours ago, Itms said:

Instead we would be grateful to have profiling data and numbers instead of vague feelings of performance degradation.

Itms, as players we now only have one very crude but objective number at our disposal to measure performance degradation, which is comparing 'elapsed game time' vs 'elapsed lobby time'. When you see that a 25 min game has been running for 50 mins in the lobby (assuming no pauses) then you know this game is running horribly slow. You can convert this ratio to a speed factor, like 0.5x. As most games get slower as the game progresses, this game is actually running probably at 0.3x in late game.

This is why i have proposed in the past a few features you could add in order to measure performance improvement/degradation objectively between alphas, between types of maps, etc. Here are two ideas i already proposed:

1. Next to each turn in the replay file, write a miliseconds elapsed timestamp. This way you can know how long each turn took in real time. Maybe also record pause and unpause events in order to remove the pauses from the time keeping. With this simple change, combined with a site like replay palace, you could have an objective historical record of how performance in real tg games has been evolving.
2. During team games we have warning in the top-right of the screen that shows players that are having a slow network connection. But we dont have the same for players who have unsually slow CPUs, and lag the game this way. Make every client send to the host the average FPS over the last second in order to warn other players of clients with slow CPUs.  This way we can isolate who is causing the most lag in network games and keep investigating from there.

I can think of many more ideas like these, but you get the point.

I you dont want to rely on "vague feelings" from players you can help fix this by providing some simple tools to capture basic performance data like the onces i mention.

 

7 hours ago, wraitii said:

We'd like some help, actually. When you play MP games, please upload your commands.txt so we can replay and profile your games.

wraitii, by just profiling replays you are omitting the whole multiplayer aspect of lag. Am i missing something?

Edited Wednesday at 15:45 by Acero

[Stan`]

21 minutes ago, Acero said:

Itms, as players we now only have one very crude but objective number at our disposal to measure performance degradation, which is comparing 'elapsed game time' vs 'elapsed lobby time'. When you see that a 25 min game has been running for 50 mins in the lobby (assuming no pauses) then you know this game is running horribly slow. You can convert this ratio to a speed factor, like 0.5x. As most games get slower as the game progresses, this game is actually running probably at 0.3x in late game.

Actually you have much more information than this. By pressing F11 you see the in game profiler aka Profiler1 which will give you a sense of what can be slow. If you press it multiple times, you'll see different informations. If you press Shift+F11, you take a snapshot of all this data in the form of a txt file called profile.txt you can send to us. If you press Shift F11 multiple times, you can send us multiple measurements.

It gives you for instance the number of triangles for 3D models, but also the how many milliseconds the renderer takes per turn or the simulation.

21 minutes ago, Acero said:

wraitii, by just profiling replays you are omitting the whole multiplayer aspect of lag. Am i missing something?

The lag shouldn't be affecting the game so much. It's not really network lag, that's the problem e.g you don't have a bandwidth problem since we're sending very little data. What you have is more of a ping issue where every player is expecting to give you news every 200ms and some fail to meet that deadline, so they have up to three turns to catch up (600ms)

[wraitii]

29 minutes ago, Acero said:

wraitii, by just profiling replays you are omitting the whole multiplayer aspect of lag. Am i missing something?

Yeah as Stan said the "lag" is not actually lag, it's the simulation update (or the turn hash) taking too long. This should also show in replays from commands.txt.

[Atrik]

7 minutes ago, Stan` said:

The lag shouldn't be affecting the game so much. It's not really network lag, that's the problem e.g you don't have a bandwidth problem since we're sending very little data. What you have is more of a ping issue where every player is expecting to give you news every 200ms and some fail to meet that deadline, so they have up to three turns to catch up (600ms)

I think what @Acero meant is that the performance issue isn't affecting everybody the same, not even at the same moment in the simulation. The best example would be, moving the camera, dezooming etc, can have a lot of impact of your performances, and subsequently very likely slow the game for all players. Making a headless profiling of a multiplayer replay might or might not hint to what performances felt like for this game.

[Acero]

1 minute ago, wraitii said:

Yeah as Stan said the "lag" is not actually lag, it's the simulation update (or the turn hash) taking too long. This should also show in replays from commands.txt.

I understand this. Will 0ad do full hashes in replays too for example? Are you triggering all and each of the CPU intensive functions used during a team game when running a replay? That's why I asked you if i was missing something.

And regarding the word "lag" it is not only related to networking. There can be all kinds of lag. For example input lag with a mouse/keyboard, display lag with slow refresh screens, network lag, but in our case also lag caused the CPU not able to keep up with the speed of the game. For a player who owns a fast CPU  the lag that a slow CPU player will induce into a TG will feel exactly the same as if there was a comparable network lag. Correct me if i am wrong on that.

[real_tabasco_sauce]

My guess is what registers for faster PC players as a stutter, a brief hiccup, could be far, far worse for players with a lower spec PC. The 'gaps' you see below represent the former:

This is what it looks like for a full TG, you can see the effect of the stutter for sure. This is a 19 minute foothills TG. the blue lines are normal turns that take very little time to compute, and the gaps are the full hashing. We have done these for a long time, but I suspect that in addition to improvements in other areas revealing the stutter, we might also have more to hash overall. 

So yeah there is full hashing in replays. I'll look at earlier versions and see if the hashing gap remains large like this.

if you want to profile stuff, there is a really excellent guide on how to do it here: https://gitea.wildfiregames.com/0ad/0ad/wiki/Profiler2

[Stan`]

29 minutes ago, Acero said:

understand this. Will 0ad do full hashes in replays too for example? Are you triggering all and each of the CPU intensive functions used during a team game when running a replay? That's why I asked you if i was missing something.

Mp replays will trigger hashes yes. Sp ones no.

29 minutes ago, Acero said:

And regarding the word "lag" it is not only related to networking. There can be all kinds of lag. For example input lag with a mouse/keyboard, display lag with slow refresh screens, network lag, but in our case also lag caused the CPU not able to keep up with the speed of the game. For a player who owns a fast CPU  the lag that a slow CPU player will induce into a TG will feel exactly the same as if there was a comparable network lag. Correct me if i am wrong on that.

We agree on this

Are those replays on replay pallas ? It could be a nice source for @wraitii and @Itms

@wraitii why would the hash check freeze the UI ?

[Acero]

15 hours ago, Acero said:

If A27 is not playable for some people, then it means it was even worse for them in A26. Indeed, full hashes were performed in A26 as well. The numerous performance improvements we released are just making that bottleneck prominent. It is alarmist and totally incorrect to insinuate that A27 would have higher spec requirements than A26.

Devs (with the help of players like guerrin and seleucids) seem to be developing and testing patches that are promising in terms of getting back some performance (full hashes, dead bodies).

But let's not forget the elephant in the room: why did performance get worse for so many in this alpha?

Correcting inefficiencies present in A26 as well will not explain what happened in this alpha.

[real_tabasco_sauce]

32 minutes ago, Acero said:

Correcting inefficiencies present in A26 as well will not explain what happened in this alpha.

From what I understand, we are still figuring out why it seems to be worse. The unfortunate thing is that aside from this, the performance really is strikingly better:

https://gitea.wildfiregames.com/0ad/0ad/issues/7589#issuecomment-111582 (red is a26, green is a27). So this is essentially holding back the benefits of the work that was done, at least for MP.

Ok, so here are 3 profiles of combat demo huge in multiplayer, along with one with quick hash only:

fullhashCDH.jsonp

nofullhashCDH.jsonp

a26CDH.jsonp

rc1CDH.jsonp

I can't tell any significant difference between the magnitude of the full hash across these alphas, they all seem to be 120 to 150MS for me.