Game crashes on startup

[LukeV1]

First of all: I'm new to this forum so I am probably posting things in the wrong forum or whatever. Apologize for any mistake

Originally I wanted to create a ticket on trac, but unfortunately I never recieved a trac account activation email so I am not able to do so...

 

Concerning my orignial problem: 0 A. D., Release 22 and the (currently) newest svn version 20096 crashes on startup. It displays a black screen and then a window of the error handler. I appended the generated crashlog of the svn version, the Release 22 crash is almost identical (can upload this as well if needed). The "interesting log" file is empty and the "mainlog" file shows just the configuration settings (maybe this is intended?).

I guess the error is caused by faulty hardware detection, but I am no expert.

You may be interested in my hardware (created with dxdiag):

Spoiler

------------------
System Information
------------------
      Time of this report: 9/2/2017, 12:12:21
             Machine name: //Should not be interesting ...
               Machine Id: //Should not be interesting ...
         Operating System: Windows 10 Home 64-bit (10.0, Build 15063) (15063.rs2_release.170317-1834)
                 Language: German (Regional Setting: German)
      System Manufacturer: System manufacturer
             System Model: System Product Name
                     BIOS: BIOS Date: 03/20/17 11:22:44 Ver: 05.0000C
                Processor: AMD Ryzen 5 1600 Six-Core Processor             (12 CPUs), ~3.2GHz
                   Memory: 8192MB RAM
      Available OS Memory: 8120MB RAM
                Page File: 2903MB used, 7711MB available
              Windows Dir: C:\Windows
          DirectX Version: DirectX 12
      DX Setup Parameters: Not found
         User DPI Setting: Using System DPI
       System DPI Setting: 96 DPI (100 percent)
          DWM DPI Scaling: Disabled
                 Miracast: Available, with HDCP
Microsoft Graphics Hybrid: Not Supported
           DxDiag Version: 10.00.15063.0000 64bit Unicode

Hope you may help

crashlog.txt

Edited September 9, 2017 by LukeV1
cleanup

[Lion.Kanzen]

@elexis 

[elexis]

Definitely never seen that stacktrace.

apic.cpp is part of the arch linux dependencies if I see this correctly. What struggles me is that your OS is windows though. That part was called from the Hardware Detection part according to the crashlog.

Machine name and ID might be interesting, but indeed not relevant

Can you try previous versions of 0AD and see if that breaks too?

We did change some Hardware Detection code in #4561, so maybe it doesn't work for your particular hardware configuration.

If a previous works, you can try to bisect which revision introduced the broken change (if X works and Y fails, try (X+Y)/2 next).

I'd test for r 19876. If that crashes too, could try Alpha 21 release (see Changelogs for the revision IDs).

(Notice that the pyrogenesis.exe had been updated only every now and then, so you won't be able to pin down the exact commit that broke it).

[leper]

source/lib/sysdep/ might have some secrets. No clue how you ended up at some of those "results" though.

[LukeV1]

Wow! Thanks for your fast response! I haven't thought that I would get an reply that fast.

12 hours ago, elexis said:

Can you try previous versions of 0AD and see if that breaks too?

I'd test for r 19876. If that crashes too, could try Alpha 21 release (see Changelogs for the revision IDs).

Tested r19876 and r18915 (Release of Alpha 21), both crashed as well - but quite differently to the actual version

Both old versions created slightly differrent log outputs (not counting different adresses), see attachments...

(Now you've got the machine name :D)

 

Anything else I should test / upload??

crashlog_r18915.txt

crashlog_r19876.txt

[elexis]

I've investigated the stacktrace further and now know less than before.

Looking at the revision log of apic.cpp, Janwas was probably the only one who knew what this code does exactly (and N/A).

The line calling from the windows code into the unix code was added by r8951 with the commit message " Add some more stuff to hwdetect, because there can never be too much". Maybe we have found an exception to the rule

So you could try to delete this one line from the stacktrace that induces the crash (i.e source/ps/GameSetup/HWDetect.cpp 313)  and compile the game with a free edition of Visual Studio 2013, see BuildInstructions.

scriptInterface.SetProperty(settings, "numa_numnodes", (u32)numa_NumNodes());

I don't see anything using that numa_numnodes property (perhaps the hardware database server won't like that property being missing, but that'd be unrelated).

Alternatively, the entire hardware detection code could be cut off by adding a "return;" as the first statement in RunHardwareDetection().

As far as I know, disabling the hardware detection code won't reduce your ability to play (you might have to test which graphics settings work with your graphics card and driver at worst).

[LukeV1]

21 hours ago, elexis said:

So you could try to delete this one line from the stacktrace that induces the crash (i.e source/ps/GameSetup/HWDetect.cpp 313)

 

Did so, recompiled everything and - it still crashes. Now at the next line in HWDetect (which has become line 313 since I deleted the orignal one). I've appended the crashlog.

BTW: you may update the wiki links for downloading VS 2013. The second link is totally broken and the first one points to the VS17 site. You may consider inserting this link: http://download.microsoft.com/download/7/2/E/72E0F986-D247-4289-B9DC-C4FB07374894/wdexpress_full.exe  (taken from here)

I'll probably try disabling the hardware detection completely, but at least to me this doesn't sound like a real solution

 

crashlog.txt

[LukeV1]

Pyrogenesis survived startup now!

I've deleted all three "numa_"-lines (313-315) and 0 A. D. is starting now (first couple of times it crashed while loading fonts and textures, but then it worked). It also created a nice system_info.txt - where I noticed that HWdedect seems to have a problem with DDR4 RAM - i've got only 1 8GB module but it lists four (see appendix).

For testing purposes I've skipped RunHardwareDetection() completely then (by adding a return statement as suggested) and 0 A. D. is behaving completely same. What means that those three lines are causing that error. Now I'm very interested in what this numa_-thing should actually do (starting with the question what numa should mean). Any idea?

 

Unfortunately 0 A. D. still crashes if I try to start a game. It shows the map select menu but as soon as I hit play everything freezes after a second with a half-loaded map loading screen and windows displays it's obligatory "this program has stopped working" message. Sadly no crashlog file gets generated and the mainlog shows 0 errors (last entries are loading messages of simulation scripts).

Thats pobably what you meant with

On 4.9.2017 at 4:57 PM, elexis said:

you might have to test which graphics settings work with your graphics card and driver at worst

Where can I change that (I guess its some kind of textfile)?

Thanks,

Luke

system_info.txt

[elexis]

Main Menu -> Tools & Options -> Settings (or was it called Options?). Or copying stuff from default.cfg to user.cfg at wiki/GameDataPaths if you're into that. But mostly try to disable GLSL. If that doesn't already solve it, I'd rather investigate where it crashes.

It should be possible to debug pyrogenesis.exe in Visual Studio (wiki/Debugging), so that if it crashes, it would show you the line where it does. From that we can decide what it probably wants. If that site is too lengthy and if debugging over there isn't straight forward, Stan might answer which buttons to push.

About numa, there is numa.h:

Spoiler

/* Copyright (C) 2010 Wildfire Games.
 *
 * Permission is hereby granted, free of charge, to any person obtaining
 * a copy of this software and associated documentation files (the
 * "Software"), to deal in the Software without restriction, including
 * without limitation the rights to use, copy, modify, merge, publish,
 * distribute, sublicense, and/or sell copies of the Software, and to
 * permit persons to whom the Software is furnished to do so, subject to
 * the following conditions:
 *
 * The above copyright notice and this permission notice shall be included
 * in all copies or substantial portions of the Software.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
 * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
 * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
 * IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
 * CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
 * TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
 * SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 */

#ifndef INCLUDED_NUMA
#define INCLUDED_NUMA

//-----------------------------------------------------------------------------
// node topology

/**
 * @return number of NUMA "nodes" (i.e. groups of CPUs with local memory).
 **/
LIB_API size_t numa_NumNodes();

/**
 * @param processor
 * @return node number (zero-based) to which \<processor\> belongs.
 **/
LIB_API size_t numa_NodeFromProcessor(size_t processor);

/**
 * @param node
 * @return bit-mask of all processors constituting \<node\>.
 **/
LIB_API uintptr_t numa_ProcessorMaskFromNode(size_t node);


//-----------------------------------------------------------------------------
// memory


/**
 * @param node
 * @return bytes of memory available for allocation on \<node\>.
 **/
LIB_API size_t numa_AvailableMemory(size_t node);

/**
 * @return the ratio between maximum and minimum times that one processor
 * from each node required to fill a globally allocated array.
 * in other words, this is the maximum slowdown for NUMA-oblivious
 * memory accesses. Microsoft guidelines require it to be <= 3.
 **/
LIB_API double numa_Factor();

/**
 * @return an indication of whether memory pages are node-interleaved.
 *
 * note: this requires ACPI access, which may not be available on
 * least-permission accounts. the default is to return false so as
 * not to cause callers to panic and trigger performance warnings.
 **/
LIB_API bool numa_IsMemoryInterleaved();

#endif	// #ifndef INCLUDED_NUMA

 

The more relevant question is how it's used. One could check using svn blame or git blame which commits introduced these lines to find some context or check the code.

I guess its only used for the performance database. It's only relevant if it has something to do with the crash, so just check the crash.

[elexis]

> I'll probably try disabling the hardware detection completely, but at least to me this doesn't sound like a real solution

You can write a bugreport linking to this topic at https://trac.wildfiregames.com/, but as long as you're the only one affected and noone cares about the performance database, killing bugged useless code  sounds perfectly fine to me.

[LukeV1]

23 hours ago, elexis said:

Main Menu -> Tools & Options -> Settings (or was it called Options?). Or copying stuff from default.cfg to user.cfg at wiki/GameDataPaths if you're into that. But mostly try to disable GLSL. If that doesn't already solve it, I'd rather investigate where it crashes.

Um, unfortunately disabling GLSL and/or setting everything to minimum/maximum/custom doesn't help (if sometimes GLSL is causing crashes, why does the tooltip recommend using it?). 0 A. D. seriously do not like that hardware. So I'll have to dive into debugging. Since the PC isn't actually mine but from a friend of mine, I'll have to convince him to install VS13 - this will probably take a few days

the numa.h crash just seems to be the first sign of the underlying problem (game crash at loading a game) - and it has to do with memory size/allocation. I'll probably check if the numa-thing is connected with the strange output in the system_info.txt regarding RAM properties...

On 6.9.2017 at 9:20 PM, elexis said:

You can write a bugreport linking to this topic at https://trac.wildfiregames.com/, but as long as you're the only one affected and noone cares about the performance database, killing bugged useless code  sounds perfectly fine to me.

Yeah, I will do this - but first I want a running 0 A. D. version. Maybe the problem isn't actually caused by this lines but by something in a lower layer (i. e. memory allocation)

Right now I'm really focused on that memory thing without knowing that much about it - but to me it seems to be the most probable reason...

[Stan`]

@LukeV1 Can you attach the crashlog.dmp ?

[LukeV1]

Where should this file be located? Checked "%localappdata%\0ad\", but there is no *.dmp file. Just crashlog.txt,  interestinglog.html, mainlog.html and (since I've commented out the _numa lines) system_info.txt.

[Stan`]

If you can crash the game again, it should appear there. Else that means there is something really strange going on.

[LukeV1]

Crashing the came is absolutely no problem

I've retested the svn (head version, as of now) and my custom build, both crashing each time and not producing any *.dmp file. I've attached a screenshot of "%localappdata%\0ad\"  as proof.

BUT: I found out something different. If I start the official Alpha 22 release version, I get the error as written in the first post with the crashlog uploaded in the 5th post. But if I hit continue then, the crash handler window disappears and then reappears showing exactly the same error. If I press continue again the game starts and works flawlessly (loading and simulating maps, just everthing)! Sorry that I  wasn''t coming up with the idea of ignoring errors earlier

EDIT: Anyway, if I try this procedure with r19876, r19921 the current version (r20096) or my build (custom r20096), the program crashes if I try to start a game (showing the windows message "this program has stopped working").

Now I really wonder what is going on...

 

Edited September 8, 2017 by LukeV1
added information, marked with 'edit'

[Stan`]

Any errors in interestinglog ?

[LukeV1]

wrote some wrong information in the last post, edited that. Sorry.

Official Alpha 22: one message in interestinglog (after closing the game): game exited succesfully with 0 errors (while still creating a crashlog.txt!)

svn/custom: the interestinglog is completely empty (0 kb size) - it even don't show the caption.

[LukeV1]

After several tests I would split the initial problem in two parts:

1. HWDetect error, which is somehow caused by "numa.h". This error can be ignored and the game works then. Workaround to avoid the error handler window: comment out line 313-315 of HWDetect.cpp.
2. Game crash when loading a map / starting a game. This error is more severe because it just breaks everything. No crashlog.txt or crashlog.dmp gets created

So I will irgnore the HWDetect problem for now since its basically ignorable

Regarding the starting game crash I found out another strange thing. As written in an above post the Alpha 22 Release works (besides of the HWDetect thing). But EXACTLY the same svn version (r19921) doesn't work. I've appended a screenshot. But this versions should be completely identical or I am wrong?

[LukeV1]

@stanislas69 I could generate a dump file of the crash manually with procdump. Would this help?

[Stan`]

Maybe I need to see what visual studio says.

[LukeV1]

Finally got permission to install VS2013 and WinDbg

I've updated my svn version to r20157 and followed the instructions on https://trac.wildfiregames.com/wiki/DebuggingOnWindows I've appended the file since the service mentioned on the wiki site is down.

As I am absolutely new in debugging i have no clue what to do next - so if I should test/upload/change anything let me know.

 

And: thanks to everybody who helped me so far! I know at least something about debugging - it's quite time consuming. So thanks for support!

WinDBG.txt

[Stan`]

@LukeV1 If you compiled the game with vs, you could launch it inside it. It would tell you exactly where it crashes. On the bottom right corner there will be a window called stack. Copy everything inside it.

[LukeV1]

Ok, did so. Updated to r20159 and used the debug version inside VS13

stack.txt

[Stan`]

Thanks, try the same with building release.

 

@leper
 

[LukeV1]

Here you go: call stack of  release version r20160

stack_r20160.txt