Jump to content
Sign in to follow this  
Kuba386

Testing pathfinder update

Recommended Posts

I've merged D14 and D53 patches and it has improved pathfinder performance a lot (see this topic).

I've already made a few tests myself, however a real multiplayer game is needed.

The purpose of this test is to check:

- if there aren't any errors

- how does the patch improve performance

Common issue of multiplayer games with large number of players are lags, appearing when there are lots of units. In some cases it makes game unplayable. Goal of this patch is to fix these lags or at least decrease them.

We need to play such multiplayer game with lots of units and unlimited population.

We need at least 6 players.

We already have two: me and @Feldfeld

 

Download latest version of code, apply attached patch and compile. (See build instructions for more info)

EDIT: attached patch contains memory leak, newer version is available to download on D14.

pathfinderupdate.patch

Edited by Kuba386
  • Like 3

Share this post


Link to post
Share on other sites

How we get the patch? How to incorporate to our game? I’m using the Mac OS rebundle these days. 

Tonight im available if it works on me. 

Humnn that’s hard to build on Mac for me (svn). 

Edited by Servo
  • Thanks 1

Share this post


Link to post
Share on other sites
18 minutes ago, Servo said:

How we get the patch?

I've attached it to my post :)

 

18 minutes ago, Servo said:

How to incorporate to our game?

 

23 minutes ago, Kuba386 said:

Download latest version of code, apply attached patch and compile. (See build instructions for more info)

 

18 minutes ago, Servo said:

Humnn that’s hard to build on Mac for me (svn). 

I only have windows and linux on my computer so I probably can't help you much with compilation on mac :cry:. If people will have problems compiling this patch I will just host binaries somewhere.

I can only do it for windows and linux tough.

18 minutes ago, Servo said:

Tonight im available if it works on me. 

What does tonight mean for you? Could you give UTC time?

Edited by Kuba386

Share this post


Link to post
Share on other sites
14 hours ago, Kuba386 said:

The purpose of this test is to check:

- if there aren't any errors

- how does the patch improve performance

 

You are posting in wrong forum. I guess.

Quote

This forum is primarily for discussion for gameplay balance in format of:

Topic: Subject to be tested (e.g. Early game Cavalry Skirmisher rush) and description.

Replies to Topic: Replays of games (1v1s are best) that involve the subject of testing. Analyze the game (mistakes, well-executed aspects, etc.) and write a summary. Commentary of balance is encouraged, but must have a replay reference. Also, please include the revision of the game that was tested!

 

Proposed balance changes will be written by team members for visibility and comments in this forums as well. When a new balance-changing change occurs, topics of outdated revisions will gain a new post with the changes.

Do note that in order to keep this efficient, topics created may be edited/deleted/moved/merged to make the format of this subforum concise and consistent.

 

Share this post


Link to post
Share on other sites
Guest

I have tested this on multiplayer. With both having applied the patch and on different OSes. Linux and windows. The game was in sync, so I suppose there arent issues with determinism. At least, the kind that’s easy to spot and fix.

However, there were freezes. Which I haven’t investigated yet. May or may not be related. There were around 200 units moving. Could be the thing elexis posted on phab. 

Share this post


Link to post
Share on other sites

@Servo I could distribute a modified version of the game with that patch applied but you'd have to install it manually. would that be okay ?

(Could make a new bundle but it takes around 6h so meh)

  • Like 1

Share this post


Link to post
Share on other sites

@(-_-) thank you for testing. I've also done some testing with @nani

This pathfinder patch does not make game lagless, but it stops pathfinder from being a bottleneck. Next bottlenecks are network latency for MP games and AI for SP games.

Lags you have seen were most probably caused by network latency and not by pathfinder.

Yes, that means MP game on local network where all players have multi-core CPUs(no network latency, no AI, high number of threads for all players) is almost lagless.

Share this post


Link to post
Share on other sites
6 minutes ago, Kuba386 said:

This pathfinder patch does not make game lagless, but it stops pathfinder from being a bottleneck.

Did you run the profiler in these matches? It may tell us, what time was spending for.

Share this post


Link to post
Share on other sites
1 hour ago, Kuba386 said:

 Lags you have seen were most probably caused by network latency and not by pathfinder. 

Network latency should not be an issue, the turn lengths are 500ms in multiplayer and you can test by simulating lag that if the meanRTT <= turnLength, there is no network lag perceivable.

On "Simulation interpolation lag":

However there is a different aspect that can make the simulation consume 550-600ms, even when there is no performance bottleneck: it's the approximation / interpolation of the turn events to the intended turn length. The computation may be done in 50ms, but the events should unfold in 500ms on screen. I think the approximation depends on the performance of the recent frames, is inaccurate and can thus overshoot often. At least that's the only theory that I can come up with that explains when one measures and displays the time between the "TurnStart" and "TurnEnd" network packets (#5323). If I recall correctly, even on a new_rms_test map with about 20 entities on the entire map and only the localclient, < 10ms ping  and no units moving there are often 550ms between the two packets.

Other things to thread:

On 12/13/2018 at 10:40 PM, Kuba386 said:

There are many other things to multithread. Like NetClient or map generation. I can try multithread them too, but I'm really not sure how much can I do here. 

There are some edge cases that need async calls or threading:

  • The NetClient threading doesn't really remove performance bottlenecks, but it is absolutely necessary to split it so as not to get disconnected when some C++ or JS code blocks the main thread for some seconds, like we saw in every single a23 4v4 if one didn't simulate lag to increase the timeout tolerance artifically. (ticket and patch somewhere)
  • Map generation is threaded already (one couldn't see the loading screen progress). But not all parts of the loading screen are threaded.
  • Running the renderer and simulation in different threads might or might not have potential (for both loading screen and ingame).
  • Rejoining clients that finished the loading screen simulate the missed turns - that can freeze the application for many seconds. This is the most important issue to be fixed. It would be probably better if the rejoining client simulates 3 turns, the other players one turn, rather than the others simulating 0 turns while the rejoining client simulates all missing turns.
  • XMPP connection startup freezes the GUI until the server responds or timeout (ticket somewhere)
  • STUN connection buildup freezes the GUI for some hardcoded seconds
  • replay menu loading thousands of new replays will freeze for that period
  • userreporter libCURL interaction was threaded, but if the server times out, then the main thread exists while the userreporter thread still waits (ticket somewhere)
  • Like 1

Share this post


Link to post
Share on other sites

Okay, so first I must say how astounding the performance is with that patch (I'm using 12 threads on an hexacore CPU I7-8750H) I can play a game at 20x with 4AIs at around 10-20 fps in mid game. (It went so well that I ran out of memory maybe the Leak @elexis is tracking down)

  1. I think you forgot to edit the default.cfg to set the default number of threads (By default no number is set)
  2. Every time I open the option menu I get a warning, likely related to 1.

Here is the log.

WARNING: JavaScript warning: gui/options/options.js line 228 reference to undefined property option.tooltip
WARNING: JavaScript warning: gui/options/options.js line 263 Script value conversion check failed: v.isString() || v.isNumber() (got type undefined)
WARNING: JavaScript warning: gui/options/options.js line 228 reference to undefined property option.tooltip
WARNING: JavaScript warning: gui/options/options.js line 263 Script value conversion check failed: v.isString() || v.isNumber() (got type undefined)
WARNING: JavaScript warning: gui/options/options.js line 263 Script value conversion check failed: v.isString() || v.isNumber() (got type undefined)

 

@Servo Here is a Mac version of the game with the patch:  system.zip

To install:

  1. Download the file
  2. Extract it (clicking on it should do the trick)
  3. Open Finder
  4. Go the Application folder
  5. Right click on 0 A.D.
  6. Click on show contents
  7. Open Contents
  8. Copy LibAtlasUI.dylib in Frameworks
  9. Copy LibAtlasUI.dylib in Frameworks
  10. Copy Pyrogenesis in MacOs
  11. Install pathfinder_mod.zip
  12. Launch the game
  13. Enable and Start the mod 
  14. In the game options set the number of threads you want

 

For those on windows that don't want to compile here is the modified version on Windows (Sorry linux guys but you can compile yourselves :P)

Here is the windows version patched exectuables: system.zip

  1. Go in your 0AD installation folder
  2. Go to binaries
  3. go to system
  4. Overwrite the files
  5. Install pathfinder_mod.zip
  6. Launch the game
  7. Enable and Start the mod 
  8. In the game options set the number of threads you want

 

  • Thanks 1

Share this post


Link to post
Share on other sites
34 minutes ago, wowgetoffyourcellphone said:

Should "pyrogenesis" file in system.zip be the exe or pdb or? It currently has no file type.

Try the other zip :)

EDIT: Made the link not bug.

  • Haha 1

Share this post


Link to post
Share on other sites

You can try one per core. :)

  • Intel I3 usually have 2 to 4
  • Intel I5 4
  • Intel I7 8 to 12
  • Intel I9 can go up to 32 I think

At least two to see a difference :)

Share this post


Link to post
Share on other sites

I tested it a few times. One time it crashes, the other times the game freezes. Without the patch I didn't have any "problems" (besides lagging). Til the game crashes it runs much smoother.

My Settings:

- Intel i5 8250u, 8GB RAM, UHD 620

- 500 vs 500 (AI) vs 500 (AI) vs 500 (AI) Units

- Settings to 6 Threads and higher

crashlog.dmp

  • Thanks 1

Share this post


Link to post
Share on other sites
32 minutes ago, (-_-) said:

Regarding the OOM, someone over at HC also had encountered an OOM. The common thing being multiple AIs.

Yes that's AIs.

However the fact that the game crashes with out of memory makes me think @Kuba386's patch triggers something faster.

I compiled the exe with the large address aware patch and it crashed at around 3.5 GB of RAM usage. 4 AIs hardest difficulty. No lag whatsoever though.

 

Share this post


Link to post
Share on other sites
1 hour ago, OptimusShepard said:

Same settings, 4 Threads it freezes because of error "out of memory".

Sad thing to hear. I will look at this issue. Thank you very much for reporting.

2 hours ago, stanislas69 said:

At least two to see a difference

At least one to see a difference. Using single thread gives big difference from not using threading at all.

2 hours ago, wowgetoffyourcellphone said:

How many "threads" should I set? 

Usually optimal value is around 2*(number_of_cores-1). At least from my experience.

Share this post


Link to post
Share on other sites
Spoiler

OS Name    Microsoft Windows 10 Home
Version    10.0.17134 Build 17134
Other OS Description     Not Available
OS Manufacturer    Microsoft Corporation
System Name    JUSTUSLAPTOP
System Manufacturer    Acer
System Model    Aspire V3-771
System Type    x64-based PC
System SKU    Aspire V3-771_0686_2.14
Processor    Intel(R) Core(TM) i5-3230M CPU @ 2.60GHz, 2601 Mhz, 2 Core(s), 4 Logical Processor(s)
BIOS Version/Date    Insyde Corp. V2.14, 12/7/2012
SMBIOS Version    2.7
Embedded Controller Version    0.00
BIOS Mode    UEFI
BaseBoard Manufacturer    Acer
BaseBoard Model    Not Available
BaseBoard Name    Base Board
Platform Role    Mobile
Secure Boot State    On
PCR7 Configuration    Binding Not Possible
Windows Directory    C:\WINDOWS
System Directory    C:\WINDOWS\system32
Boot Device    \Device\HarddiskVolume2
Locale    Earth, Sol system
Hardware Abstraction Layer    Version = "10.0.17134.471"
User Name    JustusLaptop\JustusAvramenko
Time Zone    Eastern European Standard Time
Installed Physical Memory (RAM)    6.00 GB
Total Physical Memory    5.82 GB
Available Physical Memory    1.49 GB
Total Virtual Memory    11.4 GB
Available Virtual Memory    4.79 GB
Page File Space    5.54 GB
Page File    C:\pagefile.sys
Kernel DMA Protection    Off
Virtualization-based security    Not enabled
Device Encryption Support    Reasons for failed automatic device encryption: TPM is not usable, PCR7 binding is not supported, Hardware Security Test Interface failed and device is not InstantGo, Un-allowed DMA capable bus/device(s) detected, Disabled by policy, TPM is not usable
Hyper-V - VM Monitor Mode Extensions    Yes
Hyper-V - Second Level Address Translation Extensions    Yes
Hyper-V - Virtualization Enabled in Firmware    Yes
Hyper-V - Data Execution Protection    Yes

So, threads = 2 for the system above?

Share this post


Link to post
Share on other sites

Since you have 4 logical processors I would try using 4-6 threads.

Would be nice if you could try using 1,2,4,6 and post results here.

Edited by Kuba386

Share this post


Link to post
Share on other sites
2 hours ago, OptimusShepard said:

Same settings, 4 Threads it freezes because of error "out of memory".

What if you use one thread or disable threading? (0 threads)

Share this post


Link to post
Share on other sites
6 hours ago, stanislas69 said:

I think you forgot to edit the default.cfg to set the default number of threads (By default no number is set)

Yeah, I forgot. If you din't tick the checkbox default number of threads will be used, so by default default number of threads is used :)

Thank you for packaging this patch :)

6 hours ago, stanislas69 said:

It went so well that I ran out of memory maybe the Leak @elexis is tracking down

I will take a look at this too.

8 minutes ago, stanislas69 said:

Do you get the out of memory error when pitting 4 AIs against each other @Kuba386 ? Might be a windows only bug. 

Actually my memory usage is 3.5 GB for 7 AIs on linux.
 

My memory usage exceeds 4GB (5.5GB at the moment), but I get no crash. I've heard that windows version is 32-bit so cannot allocate more than 4GB.

Edited by Kuba386
  • Like 1

Share this post


Link to post
Share on other sites
Guest

The only leak fix I have seen is the one on phab. Which would be irrelevant for a standard game (see the rev proposal). If something else, I don’t know. But most likely not windows specific.

Edited by Guest

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this  

×
×
  • Create New...