Re-Release A27 or RC builds?

[Seleucids]

As most people are aware, A27 has some technical bugs: extreme stuttering and OOS on rejoin. The extreme stutter makes many players drop down to 3fps or even freeze and crash the game completely. We have determined that  it is caused by full hash. Wraitii has a fixed version with quick hash that solves this problem. I and @real_tabasco_sauce have tested it out and large battles with 4 players were able to run smoothly. 

My question is, can we do a quick re-release like we had in A25-b or just call a new version A27.5 / A28, where we at least fix the hashing stutter. 

If this is too much to ask, all we need to do is compile this: https://gitea.wildfiregames.com/wraitii/0ad/src/branch/quick_hash_only/ into .exe or appimages then distribute it to the player base so that people who aren't capable of building the solution can also play. Of course it might not be compatible with the rest of A27 players but if everyone could update easily, then it won't be a problem. 

The solution can be popularised by advertising on the forum and the lobby heading message. We can do a release candidate every time a bug is fixed, just to get games going. This current version is almost unplayable for many medium spec users and a few players I know have quit because of the stutter issue. 

 

[ffm2]

This is a performance improvement by removing a feature. Before one removes a feature one should consider what this feature is there for in the first place. The feature is there to detect OOS. OOS still occure even in vanilla but also when broken mods are used. This shows the necessity of this feature. Without this feature https://gitea.wildfiregames.com/0ad/0ad/issues/7634 would not have been caught.

[Genava55]

5 hours ago, Seleucids said:

We have determined that  it is caused by full hash.

What is "full hash" in this context? The control of all values in the simulation?

[hyperion]

Someone else would allow the reader to have context, so https://gitea.wildfiregames.com/0ad/0ad/pulls/7610

[Itms]

8 hours ago, Seleucids said:

As most people are aware, A27 has some technical bugs: extreme stuttering and OOS on rejoin. The extreme stutter makes many players drop down to 3fps or even freeze and crash the game completely. We have determined that  it is caused by full hash. Wraitii has a fixed version with quick hash that solves this problem. I and @real_tabasco_sauce have tested it out and large battles with 4 players were able to run smoothly.

This is an oversimplification of the situation. As ffm correctly points out, the full hashing is a feature that cannot be removed lightly. We are working actively on the issue, as you can see from the linked technical discussions.

8 hours ago, Seleucids said:

My question is, can we do a quick re-release like we had in A25-b or just call a new version A27.5 / A28, where we at least fix the hashing stutter.

What we will not do is hurriedly release a new version. We are carefully fixing a few bugs the right way which will take time. Thanks to our new git workflow, we are able to prepare a 0.27.1 release in an orderly way, something that wasn't possible in the past. It will also take some time to get the process right, as it will be our first patch release.

The future release 28 is in active development and should be out in Q4 of 2025.

I cannot give you an estimate for 27.1, but it will be as fast as humanly possible on our 100% volunteer free time.

8 hours ago, Seleucids said:

This current version is almost unplayable for many medium spec users and a few players I know have quit because of the stutter issue.

If A27 is not playable for some people, then it means it was even worse for them in A26. Indeed, full hashes were performed in A26 as well. The numerous performance improvements we released are just making that bottleneck prominent. It is alarmist and totally incorrect to insinuate that A27 would have higher spec requirements than A26.

[guerringuerrin]

9 hours ago, Seleucids said:

The extreme stutter makes many players drop down to 3fps or even freeze and crash the game completely. We have determined that  it is caused by full hash. Wraitii has a fixed version with quick hash that solves this problem. I and @real_tabasco_sauce have tested it out and large battles with 4 players were able to run smoothly. 

If I recall correctly, we ran a 2v2 large-army combat test two or three days ago with you and @real_tabasco_sauce . During that test, I used two computers: a high-spec PC and a mid-range laptop. Despite having all graphics settings on low, the laptop still struggled, running at just 3 FPS for most of the large battles.

Some improvements have been observed, such as the absence of stuttering and state hash check spikes when panning the camera. While these changes may indicate a performance boost overall, I would be cautious about promoting them as a definitive "performance improvement," as players might find the gains imperceptible, leading to potential frustration.

To ensure a more accurate assessment, it would be beneficial to involve more players in the testing environment, allowing us to conduct full 4v4 matches under "typical multiplayer conditions".

Edited Tuesday at 10:47 by guerringuerrin

[Seleucids]

1 hour ago, Itms said:

If A27 is not playable for some people, then it means it was even worse for them in A26.

I'm afraid this isn't always the case. In A26, people with low spec computers also had low frames and simulation delays. However, the stuttering was much less of an issue as it only gets extreme in very very large fights. In most cases, low graphics cards just gave 10-20 fps, which was not comfortable but still playable. We have almost never seen anyone having to pause the game because their GUI is crashing, whereas this happens in almost every game nowadays. 

On average, the frame rate in A27 is only 70% of that in A26. I did some fps recording and obtained this data. The important part is the minimum fps has dropped a lot since A27 and GUI  fails to respond. A26 could run smoothly with just 1 stutter per game on powerful hardware, whereas A27 struggles even when your hardware is maxed out. The difference is, A27 fails to utilise 100% of any resource: a single CPU core is at 60% when the game is stuttering and the GPU usage is merely 50%. 

However, with the build which removed full hashing, the usage of system resources is near 100% and the bottleneck now is really the hardware. Another solution is to async the GUI so that at least the player can hang in the game, but not sure if that is trivially doable. 

 

44 minutes ago, guerringuerrin said:

To ensure a more accurate assessment, it would be beneficial to involve more players in the testing environment, allowing us to conduct full 4v4 matches under "typical multiplayer conditions".

If nobody shows up, we can make a few alt accounts and join with multiple instances (Linux OP !). Since the game is only a single core process, the performance of 4 instances is the same as 1 instance. Another idea is I do unofficial builds and distribute that out to players.

[0 calories]

6 hours ago, Itms said:

This is an oversimplification of the situation. As ffm correctly points out, the full hashing is a feature that cannot be removed lightly. We are working actively on the issue, as you can see from the linked technical discussions.

What we will not do is hurriedly release a new version. We are carefully fixing a few bugs the right way which will take time. Thanks to our new git workflow, we are able to prepare a 0.27.1 release in an orderly way, something that wasn't possible in the past. It will also take some time to get the process right, as it will be our first patch release.

The future release 28 is in active development and should be out in Q4 of 2025.

I cannot give you an estimate for 27.1, but it will be as fast as humanly possible on our 100% volunteer free time.

If A27 is not playable for some people, then it means it was even worse for them in A26. Indeed, full hashes were performed in A26 as well. The numerous performance improvements we released are just making that bottleneck prominent. It is alarmist and totally incorrect to insinuate that A27 would have higher spec requirements than A26.

I can claim 0.26 fast smooth and fast.. while 0.27 I have 4 fps issue for sure.. I guess you assurance that A26 had problem is not correct from player perspective. 

Edited Tuesday at 15:51 by 0 calories

[0 calories]

Also i would rather play 0.27 patch 1 rather waiting anothe 2 years for another full alfa/beta release.. just fix it / revert it for players to be happy.. and you can fix  performance issues (or full checks or how you call it - with other features which will lead to much smooter gameplay I hope)  meanwhile.. that sounds most logical to me. If you want to lose player base just keep it sucks...

Edited Tuesday at 16:04 by 0 calories

[real_tabasco_sauce]

41 minutes ago, 0 calories said:

I can claim 0.26 fast smooth and fast.. while 0.27 I have 4 fps issue for sure.. I guess you assurance that A26 had problem is not correct from player perspective.

Can you describe your problem in more detail? Do you drop down to 4fps irregularly, like every 4 seconds, or is it just a constant laggy experience?

How quickly in a typical game do you get down to 4fps? And what fps would an equivalent game in a26 have? What steps to fix it have you tried, like switching to different graphics settings in options -> graphics (advanced). Have you tried combinations of Vulkan/openGL and with vs without GPU skinning?

It could be that you are experiencing a separate issue.

[hyperion]

34 minutes ago, 0 calories said:

I can claim 0.26 fast smooth and fast.. while 0.27 I have 4 fps issue for sure.. I guess you assurance that A26 had problem is not correct from player perspective. 

Great, that means you are one of the few that can bisect the issue, let us know which commit changed your experience for the worse.

You should be able by letting ai players play against each other in a LAN game (no need for team games) to reproduce the issue if it's related to state hashing, which is likely if you see a short simulation stall every 4s only but have high fps otherwise. Any other symptoms must have a different reason than what this thread is about.

[Seleucids]

3 minutes ago, real_tabasco_sauce said:

How quickly in a typical game do you get down to 4fps? And what fps would an equivalent game in a26 have? What steps to fix it have you tried, like switching to different graphics settings in options -> graphics (advanced). Have you tried combinations of Vulkan/openGL and with vs without GPU skinning?

For me, my frame drops to 10 fps as soon as I reach P3. Then freezes during fights. 

 

Good news @0 calories, I managed to recompile A27 with the wraitii quick hash and it reduces lag to near A26 levels! The game throws OOS errors at 10 seconds in but it is a false alarm and will not pop up again. 

[Stan`]

1 hour ago, 0 calories said:

or full checks or how you

Just for the record that full hash check has been there since there is multiplayer (it was there in a23 as well) it is ran every 20, 200ms turns or less if your computers are fast enough.

31 minutes ago, Seleucids said:

Good news @0 calories, I managed to recompile A27 with the wraitii quick hash and it reduces lag to near A26 levels! The game throws OOS errors at 10 seconds in but it is a false alarm and will not pop up again. 

Could you compare performance with Shift+F11 this will generate a profile.txt that can help us diagnose what's taking time. You can also encourage other players to do so.

Do note that since you don't have full hashes anymore you won't be able to see some real ooses like the Briton one.

[0 calories]

3 hours ago, real_tabasco_sauce said:

Can you describe your problem in more detail? Do you drop down to 4fps irregularly, like every 4 seconds, or is it just a constant laggy experience?

How quickly in a typical game do you get down to 4fps? And what fps would an equivalent game in a26 have? What steps to fix it have you tried, like switching to different graphics settings in options -> graphics (advanced). Have you tried combinations of Vulkan/openGL and with vs without GPU skinning?

It could be that you are experiencing a separate issue.

I tried any GFX setup vulkan/opengl/native soon depreceated no mater what always getting sluttering .. it start from beggining of the game it is visible... but not that moch as game progress at some point FPS goes down so much i need restart 0ad rejoin and get for short time again some UI responsivness...

I did not try any steps to fix it as Im not developer and have no idea how to fix it. I just changed gfx settings above.. 

 

yes it could be anything but im not alone.. 

[Seleucids]

 

2 hours ago, Stan` said:

Could you compare performance with Shift+F11 this will generate a profile.txt that can help us diagnose what's taking time. You can also encourage other players to do so.

This was taken while spectating a TG at minute 9, on my custom build:

profile.txt

Now with the normal one:

profile.txt

 

[0 calories]

3 hours ago, hyperion said:

Great, that means you are one of the few that can bisect the issue, let us know which commit changed your experience for the worse.

You should be able by letting ai players play against each other in a LAN game (no need for team games) to reproduce the issue if it's related to state hashing, which is likely if you see a short simulation stall every 4s only but have high fps otherwise. Any other symptoms must have a different reason than what this thread is about.

I had just little time and patience to test it but here is my observation (need more time to evolve game in late stage) i just keep few minutes..

8xAI hosted (no lobby) it seems more stable and less "shuttering"

7xAI 1xme (no mods single mod enabled!) I feel presence of shuttering every 4-10 secs 

 

[Stan`]

Hash only occur on MP games. They should happen in local lan games as well. They will not happen in SP games.

[real_tabasco_sauce]

So that does actually sound like the same issue @0 calories, just more severe. Was the one you did with 8xAI more similar to your experience from a26?

[0 calories]

18 minutes ago, real_tabasco_sauce said:

So that does actually sound like the same issue @0 calories, just more severe. Was the one you did with 8xAI more similar to your experience from a26?

I never did 8xAI in A26 and cant compare now as I dont have A26 anymore but i just know I never experienced such shuttering .. lags yeas.. but this is not lag..

[Atrik]

From a few games tested without hash check, both I and @Seleucids seems to have a VERY noticeable difference in performance. I was extremely skeptical that it would but it seems like it does somehow. The stuttering is just... gone. And in late game I don't get insane fps drops like 7 - 3 fps....

Would it be possible to return the other players hash by default every 20 turn but once in a while, maybe every minute, actually calculate the hash? This way we could have fully compatible version with the fix even if it's hacky? How could i do this?

[Seleucids]

40 minutes ago, Atrik said:

Would it be possible to return the other players hash by default every 20 turn but once in a while, maybe every minute, actually calculate the hash? This way we could have fully compatible version with the fix even if it's hacky? How could i do this?

Originally they did:

if (turn % 20 == 0)
        return true;

Which means 1 full has per 20 turns. We can do maybe turn % 300 == 0 which means 1 full hash check per minute. 

• return true = do full hash
• return false = don't do full hash

 

For everyone else who want to do this or play with us:

The OOS is false alarm. The error is thrown because everyone else is doing a full hash at 20 turns, but we did not respond to their check request. The error does not affect gameplay and can be royally ignored. Just play on and the simulation will be in sync because it is deterministic in nature and player commands are synced over network. 

 

I personally have noticed significant improvements, especially in early game and in large fights. However, it doesn't solve everything because other parts of simulation are still poorly optimised. 

Edited Tuesday at 22:59 by Seleucids

[Seleucids]

1 hour ago, 0 calories said:

I never did 8xAI in A26

https://releases.wildfiregames.com

Here you can download all past versions of the game and try them out if you are interested  

We did our custom builds here. 

[guerringuerrin]

33 minutes ago, Seleucids said:

For everyone else who want to do this or play with us:

The OOS is false alarm. The error is thrown because everyone else is doing a full hash at 20 turns, but we did not respond to their check request. The error does not affect gameplay and can be royally ignored. Just play on and the simulation will be in sync because it is deterministic in nature and player commands are synced over network. 

Shouldn't we wait to do the necessary testing on other implications of removing this feature before we go public and start having players running different versions of the game? It seems a bit hasty. Or maybe I'm missing something?

Edited Tuesday at 23:31 by guerringuerrin

[Atrik]

7 minutes ago, guerringuerrin said:

Shouldn't we wait to do the necessary testing on other implications of removing this feature before going public and starting players running different versions

Y I just had a game with climax lag and 3fps using no hash. Camera and gui were still responsive thoughts. It's clearly bad to play without the oos hash check but i guess most players are desperate for improvements rn

Edited Tuesday at 23:35 by Atrik

[Seleucids]

This video below shows you how to re-compile A27 and edit the engine code so that hash is removed. This is exactly what I did this afternoon:

By the way welcome to my new channel ! 

This one is for Linux users. I will look into Windows tomorrow.