Progress reports on funded work

[k776]

Aside from fixing the algorithm to be consistent, how about when a player reconnects, have all players trigger a recompute of LoS, so they all get the freshest LoS possible?

[Pureon]

Great work. Thanks for explaining the problem encountered so clearly

[Ykkrosh]

Aside from fixing the algorithm to be consistent, how about when a player reconnects, have all players trigger a recompute of LoS, so they all get the freshest LoS possible?

That sounds kind of like an ugly hack, and if the bugs are fixed properly then it won't be necessary, so I think it's better to just fix the bugs properly . Currently the simulation code doesn't even know that a player has reconnected (that's handled purely by the network server code), and it would add non-trivial complexity to have the simulation depend on that information (since it'd be a new input channel that would have to be synchronised over the network, stored in replays, tested, etc), so I think that general kind of thing should be avoided if possible.

[Ykkrosh]

Day 4

I fixed the known problems with the LOS calculation and other saving-related bugs (player colours, unit animations, etc), and cleaned up the GUI when reconnecting (it now goes straight to the loading screen and tells other players that someone has joined), which is kind of tedious but necessary. Now it just needs people to test it to find out what else is broken

Perhaps more excitingly, I also added some code for saving and loading single-player games:

The GUI is pretty basic at the moment, and the saved games are a bit fragile (it doesn't work if you have AI players, and it'll likely get confused if you update the game's data files and then use an old saved game), but otherwise it seems to be working okay.

[Almin]

First I wanted to add, that your work is great, Ykkrosh, but I have a question:

What if one player wants to join a multiplayer-game of another player(the host), where the host has an other version of the game? For example, the host has alpha 8 and opens a game, and an other player, who himself is playing alpha 9, joins him. Are there any possibilities to stop this. There could be terrible problems, if for example, one player with a newer version of the game chooses a race, that isn't even implemented in the older release of the other player. Also what happens, if both play a multiplayer-game, save it, and one of them or both are updating on a newer release? Does it work, and if yes: How does it work?

Just for those, who don't know how that is handled in AoE3: If you want to join a game, a message appears, that you can't join because of different versions and you will automatically get back into the main menu.

But don't misundestand me, the work, that you've done is still awesome!

Edited October 30, 2011 by Almin

[Ykkrosh]

Currently it doesn't attempt to handle incompatible versions sensibly. If you're joining a new multiplayer game, you'll probably just get an out-of-sync error. If you're loading a saved game, you may get errors saying the file is corrupted (or actually I think it might just crash) or may get script errors or the game may just run a bit weirdly (e.g. if you changed a unit's max HP after saving, then load the earlier game with that unit, its max HP will have changed but its current HP won't have, so it'll look more injured than it used to).

For now, I think we ought to make it compute some kind of checksum of all the simulation data files and use that in saved games and in multiplayer sessions. If the data files have changed at all, it will notice and can warn you that the game probably won't work. (But it should allow you to override that warning since developers might be intentionally changing their data files to test stuff). In the future it would be nice if saved games were compatible across new releases of the game, but that's probably not worth implementing yet, so players will just have to restart if they update the game.

[Almin]

Currently it doesn't attempt to handle incompatible versions sensibly. If you're joining a new multiplayer game, you'll probably just get an out-of-sync error. If you're loading a saved game, you may get errors saying the file is corrupted (or actually I think it might just crash) or may get script errors or the game may just run a bit weirdly (e.g. if you changed a unit's max HP after saving, then load the earlier game with that unit, its max HP will have changed but its current HP won't have, so it'll look more injured than it used to).

For now, I think we ought to make it compute some kind of checksum of all the simulation data files and use that in saved games and in multiplayer sessions. If the data files have changed at all, it will notice and can warn you that the game probably won't work. (But it should allow you to override that warning since developers might be intentionally changing their data files to test stuff). In the future it would be nice if saved games were compatible across new releases of the game, but that's probably not worth implementing yet, so players will just have to restart if they update the game.

Thank you very much for that explanation. Now if you would use a checksum of the savegame on both computers or something similar, you could also prevent one of the players cheating by changing or editing the savegame? Is this right? This would be so cool!

[feneur]

Thank you very much for that explanation. Now if you would use a checksum of the savegame on both computers or something similar, you could also prevent one of the players cheating by changing or editing the savegame? Is this right? This would be so cool!

I'm not a programmer so I don't know the technical details, but as far as I know it doesn't so much prevent as make it a lot harder to cheat by editing the savegame/datafiles.

[quantumstate]

Thank you very much for that explanation. Now if you would use a checksum of the savegame on both computers or something similar, you could also prevent one of the players cheating by changing or editing the savegame? Is this right? This would be so cool!

I would expect that editing a saved game and then resuming a match would rapidly give an out of sync error.

[fabio]

Day 4

I fixed the known problems with the LOS calculation and other saving-related bugs (player colours, unit animations, etc), and cleaned up the GUI when reconnecting (it now goes straight to the loading screen and tells other players that someone has joined), which is kind of tedious but necessary. Now it just needs people to test it to find out what else is broken

Perhaps more excitingly, I also added some code for saving and loading single-player games:

The GUI is pretty basic at the moment, and the saved games are a bit fragile (it doesn't work if you have AI players, and it'll likely get confused if you update the game's data files and then use an old saved game), but otherwise it seems to be working okay.

Thanks for all the work

Loading/saving seems to work but I get these warnings when loading a saved game:

ERROR: JavaScript error: simulation/ai/jubot/gamestate.js line 26
TypeError: this.templates is undefined
  ("structures/hele_civil_centre")@simulation/ai/jubot/gamestate.js:26
  ([object Object],"structures/{civ}_civil_centre",1)@simulation/ai/jubot/plan-building.js:7
  ([object Object],[object Object])@simulation/ai/jubot/economy.js:225
  ([object Object],[object Object])@simulation/ai/jubot/economy.js:653
  ()@simulation/ai/jubot/jubot.js:115
  ([object Object])@simulation/ai/common-api/base.js:92
  @:0
ERROR: JavaScript error: simulation/ai/jubot/gamestate.js line 26
TypeError: this.templates is undefined
  ("structures/hele_civil_centre")@simulation/ai/jubot/gamestate.js:26
  ([object Object],"structures/{civ}_civil_centre",1)@simulation/ai/jubot/plan-building.js:7
  ([object Object],[object Object])@simulation/ai/jubot/economy.js:225
  ([object Object],[object Object])@simulation/ai/jubot/economy.js:653
  ()@simulation/ai/jubot/jubot.js:115
  ([object Object])@simulation/ai/common-api/base.js:92
  @:0

(and so on...)

This is with r10458.

[WhiteTreePaladin]

It doesn't work with AIs yet, so disable those before starting a match. He will add support for those later when he starts fixing the AI framework.

[Ykkrosh]

Day 5

Now working on improved support for performance profiling.

There are lots of parts of the code that need optimisation to make them faster, but I don't think our current tools are good enough for accurately measuring and visualising the important aspects of performance. In particular, the current in-game profiler (press F11 and digit keys when running the game) is designed to show an instantaneous view of the average performance over a number of frames. Similarly, typical third-party profiling tools (CodeAnalyst, Callgrind, etc) show an average over a long period time. That's fine for applications where you only care about overall throughput (e.g. how many seconds it takes to encode a video file), but for games we also care about consistency of framerate. If a game runs at 60fps but then pauses for 100msecs once per second, that's worse than running at a smooth 30fps. Similarly, it might pause for a second once every few minutes (perhaps when loading some new data from disk). We need a way to detect that kind of inconsistency and to understand what caused it.

One way to smooth out the framerate is to run slow tasks in a separate thread, so they don't interrupt the flow of the main rendering loop. As a bonus, that also improves throughput for users with multi-core CPUs. The current in-game profiler can't cope with multi-threading at all, so we need to fix that too.

I was pointed at the RAD Telemetry tool, which appears to be designed to solve exactly this problem. It's $6000 and not open source, so we can't use it for our game, but it's a good demonstration of the features that would be useful.

So I started writing a new profiler. It collects and stores timing data from the engine (without doing any averaging across frames), then you can visualise and analyse the data in a web browser.

(Technical details: Each thread has a ring buffer, into which it writes a series of events (frame starts, entering named region, leaving named region, etc). An embedded Mongoose server provides an HTTP interface: an HTML page requests the data from the engine, so the engine grabs a copy of all those ring buffers and converts them to JSON and returns them. The HTML page then processes the data and renders it onto canvases.)

The current status is that it looks like:

The top section shows the length of each frame. The blue box is expanded into the middle section, which shows the named regions. In this example it's clear that the occasional spikes are caused by the "simulation update" region (which doesn't happen on every frame). The blue box in the middle is expanded into the bottom, showing a lot of "Move" regions taking up some of the time.

I think this is almost useful now - I'm planning to spend the next day making it a bit more powerful and cleaning up the code, and then it should be ready to help with some optimisation work.

[Chakakhan]

Philip this is fantastic! This will really help facilitate algorithm changes and other incremental performance improvements. I have been hesitant to make some changes because there was no way to accurately measure the change in performance.

I am amazed at the quality of work you produce in such a short period of time.

[Sylvain Gadrat]

Wooh ! Impressive tool ! Great work !

Just as a reminder, please keep the "EngineProfiling" wiki page up to date when integrated. It is an awsome wikipage

While talking about optimization, on the first post you talk about optimizing the pathfinder. Do you think your changes will impact the long pathfinder ? I am still musing about implementing hpa* whereas I terribly lack time. Note that it is just a question, not a request, your work is awsome and I lack time

[vts]

Well, colour me thoroughly impressed. The amount of work you've done in this timespan is nothing short of extraordinary. Can't wait to see what the future brings

[Ykkrosh]

Day 6 plus a half

(This was Thursday, but I got a bit distracted and failed to post here earlier.)

Did some more work on the profiler, primarily fixing some bugs and cleaning up code enough to commit it (here/here/here).

If anyone wants to test it, you should recompile the game then launch it and press F11 to enable the new profiler's HTTP output (it's disabled by default for performance and security), then open the file source/tools/profiler2/profiler2.html in a web browser. (I've only tested Firefox 7; browser compatibility isn't a priority). Then it should work, with luck.

One extra feature is that the code can add text annotations to each profile region, which are displayed as tooltips:

Holding the mouse over the "convert texture [...]" region pops up the box saying which texture it's converting. The "compress" in this image is happening in a secondary thread, so it overlaps many frames. (In practice the compression is done before running the game and cached, so this isn't normally a real slowdown). I think this makes it particularly useful for improving the game's startup/load performance, since we can see what files are taking a long time when they're first loaded.

Day not-counted

I subsequently played around with a few more things (not committed yet), mostly for fun rather than to be useful for the game, so they don't count as proper work. In particular, the profiler can display the GPU like a separate thread (using the GL_ARB_timer_query extension, supported on most NVIDIA and ATI/AMD cards with the proprietary drivers):

Here the CPU is the top group of coloured boxes, the GPU is the bottom group. This gives a nice view of the fancy things that graphics drivers do. At the start of the timeline (on the left), the CPU is rendering frame 4824 while the GPU is rendering frame 4822. That means the CPU is submitting a load of rendering commands, which get queued up for 10-20msec before the GPU (which is busy rendering an earlier frame) gets around to processing those commands and doing the actual rendering. On frame 4827 I did something to delay the CPU a bit. The GPU carried on rendering the queued frames, but then had to pause until frame 4827 was ready to render. After that, the CPU and GPU times wobble around a bit (but the GPU maintains a fairly smooth framerate) until the GPU has synchronised back to 2 frames behind the CPU.

One annoyance is that Intel drivers don't support GL_ARB_timer_query, and Intel GPUs are where optimisation matters most (since they're so slow). It turns out that the most recent ones have an extension called GL_INTEL_performance_queries, which sounded interesting, but there's no specification and it seems nobody has ever written anything about it. So I fiddled with it a bit and wrote some documentation of the extension. Not sure it's hugely useful, but at least it can give a reasonable indication of the time the GPU spends on each part of the rendering. (It only gives relative times, not absolute, so unfortunately it messes up the timeline view a bit, but it's better than nothing.)

[fabio]

I subsequently played around with a few more things (not committed yet), mostly for fun rather than to be useful for the game, so they don't count as proper work. In particular, the profiler can display the GPU like a separate thread (using the GL_ARB_timer_query extension, supported on most NVIDIA and ATI/AMD cards with the proprietary drivers):

Here the CPU is the top group of coloured boxes, the GPU is the bottom group. This gives a nice view of the fancy things that graphics drivers do. At the start of the timeline (on the left), the CPU is rendering frame 4824 while the GPU is rendering frame 4822. That means the CPU is submitting a load of rendering commands, which get queued up for 10-20msec before the GPU (which is busy rendering an earlier frame) gets around to processing those commands and doing the actual rendering. On frame 4827 I did something to delay the CPU a bit. The GPU carried on rendering the queued frames, but then had to pause until frame 4827 was ready to render. After that, the CPU and GPU times wobble around a bit (but the GPU maintains a fairly smooth framerate) until the GPU has synchronised back to 2 frames behind the CPU.

One annoyance is that Intel drivers don't support GL_ARB_timer_query, and Intel GPUs are where optimisation matters most (since they're so slow). It turns out that the most recent ones have an extension called GL_INTEL_performance_queries, which sounded interesting, but there's no specification and it seems nobody has ever written anything about it. So I fiddled with it a bit and wrote some documentation of the extension. Not sure it's hugely useful, but at least it can give a reasonable indication of the time the GPU spends on each part of the rendering. (It only gives relative times, not absolute, so unfortunately it messes up the timeline view a bit, but it's better than nothing.)

Awesome work

Did you try it on Windows? On mesa drivers (intel included) there is some support for timer_query (EXT variant):

http://cgit.freedesktop.org/mesa/mesa/log/?qt=grep&q=timer_query

http://cgit.freedesktop.org/mesa/mesa/tree/docs/GL3.txt

[Ykkrosh]

Just as a reminder, please keep the "EngineProfiling" wiki page up to date when integrated. It is an awsome wikipage

Okay, I'll try to update that within the next few days (when the profiler design is a bit more stable) - please remind me if I forget then

While talking about optimization, on the first post you talk about optimizing the pathfinder. Do you think your changes will impact the long pathfinder ? I am still musing about implementing hpa* whereas I terribly lack time. Note that it is just a question, not a request, your work is awsome and I lack time

Currently I'm only thinking of the short pathfinder, so it shouldn't affect that.

Did you try it on Windows? On mesa drivers (intel included) there is some support for timer_query (EXT variant)

Yeah, I'm currently using Windows. I started with the ARB_timer_query because it's more powerful than EXT_timer_query - the EXT one can only measure relative times, not absolute times, so it can say how long each part of the rendering takes but can't say how far it's delayed behind the CPU, so you can't line the CPU and GPU up properly. But INTEL_performance_query has the same problem with only providing relative times, so I need to deal with that anyway, so I suppose I might as well add EXT_timer_query too. (Currently my GPU-timing code is hideous so I need to rewrite it all anyway )

[fabio]

OK, I tested it and it works as expected.

Just a couple of questions:

1) the profiler server starts when pressing F11 but can no longer be stopped?

2) the test fails with:

Running 255 tests....................................................................................................................................................................................................................................................Profiler2.cpp(187): Assertion failed: "m_Initialised"
Assertion failed: "m_Initialised"
Location: Profiler2.cpp:187 (RegisterCurrentThread)

Call stack:

(0x84a2f60) ./test() [0x84a2f60]
(0x8449fb4) ./test() [0x8449fb4]
(0x844a247) ./test() [0x844a247]
(0x844b184) ./test() [0x844b184]
(0x8286b0c) ./test() [0x8286b0c]
(0x834fc50) ./test() [0x834fc50]
(0xb6ec2e99) /lib/i386-linux-gnu/libpthread.so.0(+0x5e99) [0xb6ec2e99]
(0xb6e2c73e) /lib/i386-linux-gnu/libc.so.6(clone+0x5e) [0xb6e2c73e]

errno = 0 (No error reported here)
OS error = ?



In TestTextureConverter::test_convert_quality:
../../../source/graphics/tests/../../../source/graphics/tests/test_TextureConverter.h:78: Error: Expected ((m_VFS->LoadFile(dest, file, fileSize)) == INFO::OK), found ({ EB 51 FE FF FF FF FF FF  } != { 00 00 00 00 00 00 00 00  })
tex_codec.cpp(83): Function call failed: return value was -120101 (Unknown error (-120101, 0xFFFFFFFFFFFE2ADB))
Function call failed: return value was -120101 (Unknown error (-120101, 0xFFFFFFFFFFFE2ADB))
Location: tex_codec.cpp:83 (tex_codec_for_header)

Call stack:

(0x84a2f60) ./test() [0x84a2f60]
(0x8449fb4) ./test() [0x8449fb4]
(0x844a247) ./test() [0x844a247]
(0x844ae10) ./test() [0x844ae10]
(0x849632c) ./test() [0x849632c]
(0x8499b52) ./test() [0x8499b52]
(0x807166f) ./test() [0x807166f]
(0x805b4a5) ./test() [0x805b4a5]
(0x8064c8c) ./test() [0x8064c8c]
(0x805b774) ./test() [0x805b774]
(0xb6d72e37) /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0xb6d72e37]
(0x8056181) ./test() [0x8056181]

errno = 0 (No error reported here)
OS error = ?


Sleeping until debugger attaches.
Please wait.
GNU gdb (Ubuntu/Linaro 7.2-1ubuntu11) 7.2
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 10674
Reading symbols from /home/fabio/sorgenti/0ad/trunk/binaries/system/test...done.
Couldn't get registers: Nessun processo corrispondente.
(gdb) 

Also for some reason I am no longer able to get proper backtrace since (probably) when the message window was added (look at the previous output).

I usually prefer running testing app with something like:

gdb -return-child-result -batch -ex run -ex 'thread apply all bt full' -ex kill -ex quit --args ./pyrogenesis

[Inazuma]

I was just wondering... I know the engine has been in development for years, however I can't help but feel you might benefit from switching to a ready-made engine. It would be stable, perform well and have nice bells and whistles. Above all, it would mean you can start focusing more on the game aspect.

Has this been considered?

[Ykkrosh]

1) the profiler server starts when pressing F11 but can no longer be stopped?

Correct. This is probably all temporary though - I'll add some config variables and/or other ways to control the profiling.

the test fails

Fixed.

I was just wondering... I know the engine has been in development for years, however I can't help but feel you might benefit from switching to a ready-made engine.

We've not seen any engines that would be suitable (open source, capable of supporting all the features we want, better than what we've currently got, etc). Even if there was one, it would generally be far more expensive in time and effort to adopt a radical change than to continue incrementally improving what we've already got (assuming what we've already got is capable of being extended to what we want, which I believe it is ).

[howlingflute]

Some really great new feature you are working on. Keep up the good work . I really like the save game feature and hope that the profiling tool acts as an asset in getting 0ad running faster. Awesome work.

[k776]

Had a chance to use the new profiler. The biggest time consumers per frame were:

* dispatch events (beween 500ms and 2000ms)

* sim update (about 500ms)

* swap buffer (about 80% of frames where 'dispatch events' and 'sim update' weren't run)

[fabio]

Correct. This is probably all temporary though - I'll add some config variables and/or other ways to control the profiling.

OK, just a couple of suggestions: maybe you can also automatically start a browser (it's already done with the website button on welcome screen) when pressing the new profiler button (maybe different from the in game profiler, F12?) so one doesn't need to remember the path of the html.

Also the framerate (with SHIFT-F) varies too fast and is hardly readable. Maybe it should update less often, or eventually using a moving average.

[plumo]

The simupdate regularly freezes my game for a second or something. It jumps sometimes to 70 when i check with F11.

EDIT: when i try to open profiler2.html with google chrome dev version 17 AND internet explorer 9 it doesnt open. ""Failed to connect to server ("error") "". Should I click on profiler2.html while running 0 A.D. ? cause i ran it after shutting down the game.

Edited November 8, 2011 by plumo