Crash Handler

[madmax]

Hi,

I was trying out 0AD today after a gap of few months and with brand new upgraded hardware (Nvidia 630M with i7 2670QM , 6GB RAM : http://www.amazon.com/N53SM-ES72-15-6-Inch-Laptop-Silver-Aluminum/dp/B007BY57Y2/ref=cm_cr_pr_product_top) and I must say the game look awesome. Super work guys and I am looking forward to the next version.

I was trying to save a game today after playing for about 20 mins and I think the game saved ok. It showed a green line in the upper left corner saying the game was saved but then the game froze and I had to force close it. Then I restarted 0AD and tried to load the game. The game did load, I can see my city, buildings, walls villagers etc, but the window has frozen again. And this time I had to force hibernate the computer as I could not get back to the desktop.

'

So I was wondering if its something in the saved game and can I post the saved game somewhere so that the devs can try loading it to see whats up. Also if a crash handler can be put in the code that's invoked when the game crashes or becomes unresponsive for about a minute, then it can be used to report the hardware platform and the stack trace back to the devs.

Its a great effort guys and I want to be more involved in fixing bugs....will get the code today

I have attached the sim log for the 1st game (3548) & the reload attempt (7188) from \0ad\logs\sim_log

Also the saved game which refuses to load is attached.

sim_log.zip

quicksave-0003.zip

Edited April 22, 2012 by madmax

[feneur]

There should be a crash handler already, depending on the nature of the crash it might not be working though =)

Also, to help debug the present issue, are you using a release version of the game or the SVN version?

[madmax]

There should be a crash handler already, depending on the nature of the crash it might not be working though =)

Also, to help debug the present issue, are you using a release version of the game or the SVN version?

Hi feneur,

I am using a release version : Ides of March. But I can use a SVN version, compile it etc too in case you need a stacktrace. I got VC++ express 2010 and directx sdk for compiling. OpenGL should be working too.

[k776]

I've added a ticket for this bug at http://trac.wildfiregames.com/ticket/1354

[madmax]

So was anyone able to reproduce it with the saved game or is it something specific to my hardware platform ?

By the way, I was wondering since the log format is a command-by-command log, so can it be played back to reproduce the steps ?

Edited April 22, 2012 by madmax

[historic_bruno]

Are you familiar with using Visual Studio's debugger? Try running the game (release version is fine) and attach the VS debugger to the pyrogenesis.exe process. Then continue reproducing this freeze or crash, and once it's there, go back to VC2010 and break the process (Ctrl+Alt+Break). Then find the call stack for the main thread, so we can perhaps see what's going on.

Of course if you want to try building from SVN, that would determine if the bug has been fixed already, and it makes it slightly easier to debug.

[madmax]

This is the call stack while debugging the exe :

> ntdll.dll!77dd27a0()

[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]

ntdll.dll!77e0afbf()

ntdll.dll!77dd578e()

ntdll.dll!77e0b10d()

ntdll.dll!77dd578e()

ntdll.dll!77e0b10d()

ntdll.dll!77e0a71a()

msvcr90.dll!737f36c5()

msvcr90.dll!737f38b3()

msvcr90.dll!737f3600()

msvcr80.dll!736d532d()

msvcp90.dll!735b5e94()

pyrogenesis.exe!CSimulation2Impl::SerializationTestState::SerializationTestState() + 0x5b bytes C++

msvcr90.dll!7383a4f5()

msvcr90.dll!7383a536()

msvcr90.dll!73839d72()

msvcr90.dll!737ffea2()

msvcr90.dll!737f2e2b()

msvcr90.dll!737ff33b()

msvcr90.dll!73800051()

msvcr90.dll!737f38c5()

pyrogenesis.exe!_onexit(int (void)* func) Line 121 + 0x5 bytes C

pyrogenesis.exe!atexit(void (void)* func) Line 127 + 0x8 bytes C

pyrogenesis.exe!__tmainCRTStartup() Line 583 + 0x17 bytes C

pyrogenesis.exe!CallStartupWithinTryBlock() Line 397 C++

kernel32.dll!75d5339a()

ntdll.dll!77db9ef2()

ntdll.dll!77db9ec5()

Here is the list of threads :

Unflagged > 6876 0 Main Thread main CSimulation2Impl::SerializationTestState::SerializationTestState Normal

Unflagged 6464 0 Worker Thread whrt_UpdateThread UpdateThread Normal

Unflagged 7576 0 Worker Thread CUserReportWorker SDL_SemWait Normal

Unflagged 2912 0 Worker Thread Win32 Thread 77da013d Normal

Unflagged 3432 0 Worker Thread Win32 Thread 77da1f26 Normal

Unflagged 6244 0 Worker Thread CCmpObstructionManager::TestStaticShape CCmpObstructionManager::TestStaticShape Normal

Unflagged 7636 0 Worker Thread Win32 Thread 77da1f26 Normal

Unflagged 6072 0 RPC Thread RPC Callback Thread 77d9fd71 Normal

Unflagged 5768 0 Worker Thread TextureConverter SDL_SemWait Normal

Unflagged 4724 0 Worker Thread Win32 Thread 77da013d Time Critical

Unflagged 2120 0 Worker Thread Win32 Thread 77da013d Normal

Unflagged 7784 0 Worker Thread Win32 Thread 77da013d Time Critical

Unflagged 6736 0 Worker Thread Win32 Thread 77da1dfe Normal

Unflagged 7920 0 Worker Thread Win32 Thread 77da013d -3

Unflagged 7036 0 Worker Thread Win32 Thread 77d9fd71 Normal

Unflagged 6428 0 Worker Thread Win32 Thread 77d9f939 Above Normal

Meanwhile I'll try to grab the code from SVN

----------------------------------------------

ok so I have downloaded, built and am running the game in the vc++ debugger. The game starts ok with the music and all. The first few times when I selected single player game and then Matches, the game was stuck at the Loading Map data dialog and I had to stop the process. It loaded the next screen now, so I ll try and reproduce the error in a fresh game.

ok tried to reload the crashing game and got this assertion failure :

Assertion failed: "0 && (L"CCmpDecay must not be used on non-local (network-synchronised) entities")"

Location: CCmpDecay.cpp:95 (CCmpDecay::Init)

Call stack:

CCmpDecay::Init (ccmpdecay.cpp:95)

this = (unavailable)

paramNode = 0x00000000

CCmpOverlayRenderer::Deserialize (ccmpoverlayrenderer.cpp:74)

this = (unavailable)

paramNode = 0x0032F144 ->

m_Value = (unsupported basic_string<wchar_t,char_traits<wchar_t> >)

m_Childs = (unsupported map<basic_string<char,char_traits<char> >,CParamNode >)

m_IsOk = true

m_ScriptVal =

m_Val =

px = 0x00000000

pn =

pi_ = 0x00000000

__formal = 0x0032F10C -> (IDeserializer)

CComponentManager::DeserializeState (componentmanagerserialization.cpp:319)

this = (unavailable)

stream = 0x0032F1E4 -> (basic_istream<char,char_traits<char> >)

rng = (unsupported basic_string<char,char_traits<char> >)

deserializer =

(IDeserializer)

m_ScriptBackrefs = (unsupported map<size_t,JSObject * >)

m_ScriptInterface = 0x106F8440 ->

m = (auto_ptr<scriptInterface_impl>)

m_Stream = 0x0032F1E4 (see above)

noParam =

m_Value = (unsupported basic_string<wchar_t,char_traits<wchar_t> >)

m_Childs = (unsupported map<basic_string<char,char_traits<char> >,CParamNode >)

m_IsOk = true

m_ScriptVal =

m_Val =

px = 0x00000000

pn =

pi_ = 0x00000000

numComponentTypes = 59 (0x0000003B)

i = 3 (0x00000003)

ctname = (unsupported basic_string<char,char_traits<char> >)

numComponents = 307 (0x00000133)

j = 200 (0x000000C8)

ent = 3922 (0x00000F52)

e = 0x6D381598 ->

(exception)

m_msg = 0x00000000

CGame::LoadInitialState (game.cpp:176)

this = (unavailable)

state = (unsupported basic_string<char,char_traits<char> >)

stream = (basic_stringstream<char,char_traits<char> >)

MemFunThunk<CGame> (loaderthunks.h:60)

param = 0x16972150

__formal = 0.01 (0x3F847AE147AE147B)

LDR_ProgressiveLoad (loader.cpp:228)

time_budget = 0.01 (0x3F847AE147AE147B)

des(no more room in buffer)

errno = 34 (No error reported here)

OS error = 487 (Attempt to access invalid address.)

--------------------------------------------------

The call stack for the crash :

user32.dll!76f1f5be()

[Frames below may be incorrect and/or missing, no symbols loaded for user32.dll]

user32.dll!76f1f5be()

user32.dll!76f1cd46()

user32.dll!76f1cf5c()

user32.dll!76f1ce8a()

user32.dll!76f1d009()

> pyrogenesis.exe!sys_display_error(const wchar_t * text, unsigned int flags) Line 315 + 0x13 bytes C++

pyrogenesis.exe!debug_DisplayError(const wchar_t * description, unsigned int flags, void * context, const wchar_t * lastFuncToSkip, const wchar_t * pathname, int line, const char * func, volatile int * suppress) Line 469 + 0x18 bytes C++

pyrogenesis.exe!debug_OnAssertionFailure(const wchar_t * expr, volatile int * suppress, const wchar_t * file, int line, const char * func) Line 558 + 0xa7 bytes C++

pyrogenesis.exe!CCmpDecay::Init(const CParamNode & paramNode) Line 95 + 0x1b bytes C++

pyrogenesis.exe!CCmpOverlayRenderer::Deserialize(const CParamNode & paramNode, IDeserializer & __formal) Line 74 C++

pyrogenesis.exe!CComponentManager::DeserializeState(std::basic_istream<char,std::char_traits<char> > & stream) Line 319 C++

pyrogenesis.exe!CGame::LoadInitialState() Line 176 C++

pyrogenesis.exe!MemFunThunk<CGame>(void * param, double __formal) Line 60 + 0xa bytes C++

pyrogenesis.exe!LDR_ProgressiveLoad(double time_budget, wchar_t * description, unsigned int max_chars, int * progress_percent) Line 228 + 0x11 bytes C++

pyrogenesis.exe!ProgressiveLoad() Line 232 + 0x25 bytes C++

pyrogenesis.exe!Frame() Line 344 C++

pyrogenesis.exe!RunGameOrAtlas(int argc, const char * * argv) Line 526 + 0x5 bytes C++

pyrogenesis.exe!main(int argc, char * * argv) Line 569 + 0xc bytes C++

pyrogenesis.exe!wmain(int argc, wchar_t * * argv) Line 380 + 0xa bytes C++

pyrogenesis.exe!__tmainCRTStartup() Line 552 + 0x17 bytes C

pyrogenesis.exe!CallStartupWithinTryBlock() Line 397 C++

kernel32.dll!75d5339a()

ntdll.dll!77db9ef2()

ntdll.dll!77db9ec5()

Edited April 23, 2012 by madmax

[historic_bruno]

The last crash looks like the one when using a saved game from a different version of the engine, we don't handle that cleanly If you want to test that, you'll have to revert to the same revision as Alpha 9, r11339.

[madmax]

ok, then what I ll do is try with the current latest and greatest SVN code and try to play the game into a crash. By the way, is there any way to make the game run a bit faster while running under the visual c++ debugger ? It seems to be rather slow with loading icons, textures and unit movment when run from visual c++.

[historic_bruno]

It will always be a little slower, but make sure you're in a release build instead of a debug build (debug build disables optimizations which is good for debugging but bad for playing). Builds from SVN will be slower loading textures the first time compared to the release package because they aren't pre-cached.

[madmax]

Yeah they are ok now, speed is up .

[Ykkrosh]

The first load of a texture from SVN will be slow since the game has to compress it, but that's only a temporary issue. When running in Release mode from the debugger, you also have to add "_NO_DEBUG_HEAP=1" to the environment variables (somewhere in the project properties), otherwise memory allocations will be abnormally slow.

[madmax]

Cool, I ll try that. By the way I just have to ask, even though its been asked many times before ...about walls, so when will draggable wall building be in the game. qbot is just killing me

Edited April 23, 2012 by madmax

[Pureon]

By the way I just have to ask, even though its been asked many times before ...about walls, so when will draggable wall building be in the game.

Soon

[Sonarpulse]

YES!!!!

[madmax]

Ah no gaps I see...cool

So is there a possibility of the archers on the walls thing ? Wow the wall building is really fast....did you change the game speed to high ?

Edited April 25, 2012 by madmax

[Zeta1127]

Ah no gaps I see...cool

So is there a possibility of the archers on the walls thing ? Wow the wall building is really fast....did you change the game speed to high ?

I would imagine walls are building instantly for demonstration purposes only, in fact I am pretty sure vts was making them build instantly with a command menu.

[madmax]

So what was the reason for the game having occasional jerks in the motion of the villagers like it was getting overloaded by too many updates ? Was it because of threading issues ? I was getting a lag yesterday playing with qbot on belgian bog with my population at about 110.

I remember Ykkrosh doing a chart depicting where the game spends most of its time. maybe this profiling chart can be enabled for any build so that testers can report in with the chart whenever they see lags in the game.

If there are more cores present can the updates happen in parallel for different sets of villagers independently on diffferent threads.

Edited April 26, 2012 by madmax

[historic_bruno]

So what was the reason for the game having occasional jerks in the motion of the villagers like it was getting overloaded by too many updates ? Was it because of threading issues ? I was getting a lag yesterday playing with qbot on belgian bog with my population at about 110.

Typically the lag is due to pathfinding and AI. I would expect Belgian Bog to be especially hard on the pathfinder because it's full of narrow passages and complex terrain, and qBot would make it worse as its military formations recalculate the path from time to time.

I remember Ykkrosh doing a chart depicting where the game spends most of its time. maybe this profiling chart can be enabled for any build so that testers can report in with the chart whenever they see lags in the game.

You can access that with F11 and then view source/tools/profiler2/profiler2.html, if you're curious about what's causing the lag. There's been some vague discussion in the past about collecting more detailed profiling data from users but we haven't gotten to that point yet. I think we're still trying to manage the data we already have

If there are more cores present can the updates happen in parallel for different sets of villagers independently on diffferent threads.

I think AIs and possibly pathfinding will benefit the most from multithreading, but they still need to be optimized for single threading first

[madmax]

So are there 2 pathfinders ? One coarse long ranged one and another accurate short range one. So the long ranged one gets waypoints at larger distances and the short range one simply routes from way point to way point avoiding immediate obstacles ?

Isnt stuff like reusing a route taken by a close by villager to the same destination, possible ?

Edited April 27, 2012 by madmax

[FeXoR]

So are there 2 pathfinders ? One coarse long ranged one and another accurate short range one. So the long ranged one gets waypoints at larger distances and the short range one simply routes from way point to way point avoiding immediate obstacles ?

Isnt stuff like reusing a route taken by a close by villager to the same destination, possible ?

Check this out: http://www.moddb.com/games/0-ad/news/pathfinding-update-24-january-2012