crash testing Internationalization

[Lion.Kanzen]

Much to our regret we must report the program has encountered an error.

Please let us know at http://trac.wildfiregames.com/ and attach the crashlog.txt and crashlog.dmp files.

Details: unhandled exception (Access violation reading 0x00000014)

Location: unknown:0 (?)

Call stack:

(error while dumping stack: No stack frames found)

errno = 0 (No error reported here)

OS error = 0 (no error code was set)

[Yves]

Update svn and try again please.

Some DLL files were missing.

[Lion.Kanzen]

Ok

[Lion.Kanzen]

its updated, the problem persist.

Edited April 21, 2014 by Lion.Kanzen

[Lion.Kanzen]

how I can revert the changes this happens after change to spanish (Chile).

[sanderd17]

If you wait a while, it should get fixed. Yves is occupying him with it.

[Yves]

Some output from WinDbg:

0:000> .ecxreax=00000000 ebx=0032deac ecx=26206e23 edx=62ee2134 esi=1117e070 edi=62f083f7eip=62f08406 esp=0032dd10 ebp=0032dd48 iopl=0         nv up ei pl nz na pe nccs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00210206msvcp90!std::operator<<<char,std::char_traits<char>,std::allocator<char> >+0xf:62f08406 8b4814          mov     ecx,dword ptr [eax+14h] ds:002b:00000014=????????*** ERROR: Symbol file could not be found.  Defaulted to export symbols for pyrogenesis.exe - 0:000> kc  *** Stack trace for last set context - .thread/.cxr resets itmsvcp90!std::operator<<<char,std::char_traits<char>,std::allocator<char> >WARNING: Stack unwind information not available. Following frames may be wrong.pyrogenesis!glooxwrapper::Client::discopyrogenesis!glooxwrapper::Client::discopyrogenesis!glooxwrapper::Client::discopyrogenesis!glooxwrapper::Client::discopyrogenesis!glooxwrapper::Client::discopyrogenesis!glooxwrapper::Client::discopyrogenesis!glooxwrapper::Client::discopyrogenesis!glooxwrapper::Client::discopyrogenesis!glooxwrapper::Client::discopyrogenesis!glooxwrapper::Client::discopyrogenesis!glooxwrapper::Client::discopyrogenesis!glooxwrapper::Client::discopyrogenesis!glooxwrapper::Client::discopyrogenesis!glooxwrapper::Client::discopyrogenesis!glooxwrapper::Client::discopyrogenesis!glooxwrapper::Client::discopyrogenesis!glooxwrapper::StanzaExtension::extensionTypepyrogenesis!glooxwrapper::StanzaExtension::extensionTypepyrogenesispyrogenesispyrogenesis!glooxwrapper::Client::discopyrogenesis!std::_Init_locks::operator=pyrogenesis!glooxwrapper::Client::discokernel32!BaseThreadInitThunkntdll!__RtlUserThreadStartntdll!_RtlUserThreadStart

Seems to be related to gloox somehow.

[Lion.Kanzen]

If you wait a while, it should get fixed. Yves is occupying him with it.

Thank you, is good to know.

[gameboy]

@Yves: My friend!

Using the latest Svn14962, start the game, the game does not respond. 0 AD (no response.)

<h2>0 A.D. Main log (warnings and errors only)</h2>
<p class="error">ERROR: [Localization] Exception while reading virtual PO file: bad allocation</p>
<p class="error">ERROR: [Localization] Exception while reading virtual PO file: bad allocation</p>

Edited April 21, 2014 by gameboy

[gameboy]

Assertion failed: "0 && ("Shader type doesn't support VertexAttribPointer")"

Location: ShaderProgram.cpp:775 (CShaderProgram::VertexAttribPointer)

Call stack:

CShaderProgram::VertexAttribPointer (shaderprogram.cpp:775)

this = (unavailable)

__formal =

m = 0x023ACEF8 ->

data = (unsupported basic_string<char,char_traits<char> >)

hash = 4132243198 (0xF64D06FE)

__formal = 4 (0x00000004)

__formal = 5126 (0x00001406)

__formal = 1 (0x01)

__formal = 64 (0x00000040)

<p class="error">ERROR: JavaScript error: gui/lobby/prelobby.js line 107
ReferenceError: feedback is not defined
onTick@gui/lobby/prelobby.js:107
__eventhandler28 (tick)@dialog tick:0</p>
<p class="error">ERROR: JavaScript error: gui/lobby/prelobby.js line 107
ReferenceError: feedback is not defined
onTick@gui/lobby/prelobby.js:107
__eventhandler28 (tick)@dialog tick:0</p>
<p class="error">ERROR: JavaScript error: gui/lobby/prelobby.js line 107
ReferenceError: feedback is not defined
onTick@gui/lobby/prelobby.js:107
__eventhandler28 (tick)@dialog tick:0</p>
<p class="error">ERROR: JavaScript error: gui/lobby/prelobby.js line 107
ReferenceError: feedback is not defined
onTick@gui/lobby/prelobby.js:107
__eventhandler28 (tick)@dialog tick:0</p>
<p class="error">ERROR: JavaScript error: gui/lobby/prelobby.js line 107
ReferenceError: feedback is not defined
onTick@gui/lobby/prelobby.js:107
__eventhandler28 (tick)@dialog tick:0</p>
<p class="error">ERROR: JavaScript error: gui/lobby/prelobby.js line 107
ReferenceError: feedback is not defined
onTick@gui/lobby/prelobby.js:107
__eventhandler28 (tick)@dialog tick:0</p>
<p class="error">ERROR: JavaScript error: gui/lobby/prelobby.js line 107

[Yves]

That's completely weird.

It's caused by tinygettext and is related to STL types. We knew about the problem before that different runtime libraries from Microsoft use different STL implementations and this was the reason why we moved tinygettext to a static library which is built as part of the project. Building it as part of the project instead of adding a binary to the repository should ensure that tinygettext uses the same CRT (runtime) version as the engine.

Somehow the STL types used in tinygettext still cause problems and I don't know why.

I've reduced the problem to a very simle test-case.

This code causes the crash if used from the engine. It only causes the crash if the constructor is part of the cpp file.

I assume otherwise the class is compiled in the engine static library completely.

Header:

class Dictionary1{public:	Dictionary1();	typedef std::map<std::string, std::vector<std::string> > Entries;	Entries entries;};

CPP file:

Dictionary1::Dictionary1(){};

Any ideas?

[sanderd17]

For anyone wanting to play the game, testing art, or whatever, you can still revert to revision 14949 if you want.

Looks like this issue well take longer to solve than first thought.

[gameboy]

Function call failed: return value was -100407 (Symbol nesting too deep or infinite recursion)
Location: wdbg_sym.cpp:1455 (udt_dump_normal)

Call stack:

While generating an error report, we encountered a second problem. Please be sure to report both this and the subsequent error messages.
errno = 34 (?)
OS error = 0 (no error code was set)

[Yves]

I've figured out that the crash only happens if NDEBUG is defined.

This causes the different behaviour in debug and release mode, but it doesn't help me much to figure out where the problem is exactly.

Using application verifier I get this callstack which seems to make more sense:

 	msvcp90.dll!69767290() 	 	[Frames below may be incorrect and/or missing, no symbols loaded for msvcp90.dll]	>	pyrogenesis.exe!tinygettext::Dictionary::Dictionary(const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & charset_="UTF-8")  Line 29 + 0x60 bytes	C++ 	pyrogenesis.exe!L10n::L10n()  Line 47 + 0x6f bytes	C++ 	pyrogenesis.exe!L10n::Instance()  Line 41 + 0x24 bytes	C++ 	pyrogenesis.exe!CGUI::Xeromyces_ReadObject(XMBElement Element={...}, CXeromyces * pFile=0x0043e5a0, IGUIObject * pParent=0x25450f60, const std::vector<std::pair<CStr8,CStr8>,std::allocator<std::pair<CStr8,CStr8> > > & NameSubst=[0](), boost::unordered::unordered_set<Path,boost::hash<Path>,std::equal_to<Path>,std::allocator<Path> > & Paths={...})  Line 1317 + 0x20 bytes	C++ 	pyrogenesis.exe!CGUI::Xeromyces_ReadObject(XMBElement Element={...}, CXeromyces * pFile=0x0043e5a0, IGUIObject * pParent=0x24bccf60, const std::vector<std::pair<CStr8,CStr8>,std::allocator<std::pair<CStr8,CStr8> > > & NameSubst=[0](), boost::unordered::unordered_set<Path,boost::hash<Path>,std::equal_to<Path>,std::allocator<Path> > & Paths={...})  Line 1227 + 0x1e bytes	C++ 	pyrogenesis.exe!CGUI::Xeromyces_ReadObject(XMBElement Element={...}, CXeromyces * pFile=0x0043e5a0, IGUIObject * pParent=0x2338ef60, const std::vector<std::pair<CStr8,CStr8>,std::allocator<std::pair<CStr8,CStr8> > > & NameSubst=[0](), boost::unordered::unordered_set<Path,boost::hash<Path>,std::equal_to<Path>,std::allocator<Path> > & Paths={...})  Line 1227 + 0x1e bytes	C++ 	pyrogenesis.exe!CGUI::Xeromyces_ReadRootObjects(XMBElement Element={...}, CXeromyces * pFile=0x1025222b, boost::unordered::unordered_set<Path,boost::hash<Path>,std::equal_to<Path>,std::allocator<Path> > & Paths={...})  Line 996 + 0x18 bytes	C++ 	pyrogenesis.exe!CGUI::LoadXmlFile(const Path & Filename={...}, boost::unordered::unordered_set<Path,boost::hash<Path>,std::equal_to<Path>,std::allocator<Path> > & Paths={...})  Line 945	C++ 	pyrogenesis.exe!CGUIManager::LoadPage(CGUIManager::SGUIPage & page={...})  Line 227	C++ 	pyrogenesis.exe!CGUIManager::PushPage(const CStrW & pageName={...}, boost::shared_ptr<ScriptInterface::StructuredClone> initData={...})  Line 86	C++ 	pyrogenesis.exe!CGUIManager::SwitchPage(const CStrW & pageName={...}, ScriptInterface * srcScriptInterface=0x0043e808, CScriptVal initData={...})  Line 78	C++ 	pyrogenesis.exe!InitPs(bool setup_gui=true, const CStrW & gui_page={...}, ScriptInterface * srcScriptInterface=0x00000000, CScriptVal initData={...})  Line 525	C++ 	pyrogenesis.exe!InitGraphics(const CmdLineArgs & args={...}, int flags=0)  Line 1031 + 0x65 bytes	C++ 	pyrogenesis.exe!RunGameOrAtlas(int argc=1, const char * * argv=0x09454ff8)  Line 470 + 0xb bytes	C++ 	pyrogenesis.exe!main(int argc=1, char * * argv=0x09454ff8)  Line 515 + 0xd bytes	C++ 	pyrogenesis.exe!wmain(int argc=1, wchar_t * * argv=0x08ed6f98)  Line 380 + 0xa bytes	C++ 	pyrogenesis.exe!__tmainCRTStartup()  Line 583 + 0x17 bytes	C 	pyrogenesis.exe!CallStartupWithinTryBlock()  Line 397	C++ 	kernel32.dll!7558338a() 	 	ntdll.dll!77109f72() 	 	ntdll.dll!77109f45()

Someties I got even a bit closer to the crash. It happens during the initialization of an std::map object.

[Yves]

I've removed tinygettext, iconv and ICU and replaced the localization functions that used them with dummy-functions.

The crash still happens in release mode, but now on different places again with completely different callstacks (but always in STL).

I've also replaced the singleton implementation of L10n because I suspected it could cause some rarely tested constelations that could be bugged in VS2008.

That also didn't help.

I guess I have to revert the patch piece by piece and implement more or less working intermediate steps to figure out which part causes these issues.

Debugging in VS only produces garbage as far as I've seen so far.

[Lion.Kanzen]

Thank you for your work guys.

[Yves]

It currently looks like a bug in the VS2008 linker.

When incremental linking is enabled, Link.exe crashes quite often, but I don't know if this is related.

There's KB948127, but it's part of SP1 and is already installed on my system.

Anyway, I think there are two similar but different issues with the crashes in 0 A.D..

The first is a crash that can happen at different places if the debug runtime is used (/MDd) and NDEBUG is specified. It always crashes in STL functions (usually constructors) but happens with many different callstacks.

It looks very similar to the other issue and might be related, but I'll focus on the other one because we normally don't need /MDd plus NDEBUG.

This issue happened when I removed all libraries as described in the previous post, so I didn't quite notice that the other issue was already gone.

The second issue is in the Dictionary constructor with the callstack posted above.

If we don't call that constructor (don't use tinygettext), the game also builds and runs properly in release mode with VS2008.

This means the issue is isolated to a single place in our code.

I've written a simple example-program that basically does the same as the problematic code. It has a dictionary class which is part of a static library and the main executable calls new/delete on that class (code here).

In the game this crashes in the constructor when the STL map gets initialized, in the example program it works fine.

Comparing the instruction that fails, I see this:

Game:

	_Nodeptr _Buynode()		{	// allocate a head/nil node		_Nodeptr _Wherenode = this->_Alnod.allocate(1);00205AA0  push        44h  00205AA2  call        CMessageVisionRangeChanged::ToJSVal+143h (1C47E3h) 00205AA7  add         esp,4 		int _Linkcnt = 0;

Example program:

	_Nodeptr _Buynode()		{	// allocate a head/nil node		_Nodeptr _Wherenode = this->_Alnod.allocate(1);013B21D0  push        44h  013B21D2  call        operator new (13B22E0h) 013B21D7  add         esp,4 		int _Linkcnt = 0;

Callstack:

>	pyrogenesis.exe!std::_Tree<std::_Tmap_traits<unsigned int,StaticShape,std::less<unsigned int>,std::allocator<std::pair<unsigned int const ,StaticShape> >,0> >::_Buynode()  Line 1384	C++ 	pyrogenesis.exe!std::_Tree<std::_Tmap_traits<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::vector<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::allocator<std::basic_string<char,std::char_traits<char>,std::allocator<char> > > >,std::less<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,std::vector<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::allocator<std::basic_string<char,std::char_traits<char>,std::allocator<char> > > > > >,0> >::_Tree<std::_Tmap_traits<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::vector<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::allocator<std::basic_string<char,std::char_traits<char>,std::allocator<char> > > >,std::less<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,std::vector<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::allocator<std::basic_string<char,std::char_traits<char>,std::allocator<char> > > > > >,0> >(const std::less<std::basic_string<char,std::char_traits<char>,std::allocator<char> > > & _Parg=less, const std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,std::vector<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::allocator<std::basic_string<char,std::char_traits<char>,std::allocator<char> > > > > > & _Al={...})  Line 510 + 0xf bytes	C++ 	pyrogenesis.exe!tinygettext::Dictionary1::Dictionary1()  Line 24 + 0x15 bytes	C++ 	pyrogenesis.exe!L10n::L10n()  Line 42 + 0x2c bytes	C++ 	pyrogenesis.exe!Init(const CmdLineArgs & args={...}, int flags=0)  Line 869 + 0x2a bytes	C++ 	pyrogenesis.exe!RunGameOrAtlas(int argc=1, const char * * argv=0x09600ff8)  Line 470	C++ 	pyrogenesis.exe!main(int argc=1, char * * argv=0x09600ff8)  Line 515 + 0xd bytes	C++ 	pyrogenesis.exe!wmain(int argc=1, wchar_t * * argv=0x08ff2f78)  Line 380 + 0xa bytes	C++ 	pyrogenesis.exe!__tmainCRTStartup()  Line 583 + 0x17 bytes	C 	pyrogenesis.exe!CallStartupWithinTryBlock()  Line 397	C++ 	kernel32.dll!754b338a() 	 	[Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]	 	ntdll.dll!77a29f72() 	 	ntdll.dll!77a29f45() 	

Apparently instead of calling new, it calls another random function.

Now the question is, at which stage this problem happens exactly. This assembler output is from visual studio directly (while debugging).

I assume this code is part of the CRT, so in which binary would it actually be? I'd like to figure out if the binary on the disk already looks that way or if something breaks during dynamic loading.

[Lion.Kanzen]

Ok I wasopen the game then I see option menu, then lenguage then I choose esp( Spanish ) chile.

And then the game crashes and even updated the SVN crashes, I can revert my lenguage to default? English in some file manually.

[Yves]

@Lion

You can delete the line that stats with "Locale = " from user.cfg or change the value to something else.

[Lion.Kanzen]

@Lion

You can delete the line that stats with "Locale = " from user.cfg or change the value to something else.

Thank you. A lot.

[sanderd17]

So it doesn't crash if you use the en_US locale? Michael might be please to hear this. As it would allow him to continue working on 0 A.D.

[Lion.Kanzen]

Local.cfg or other file I don't find the locate line or similar.

[Yves]

Finally! I've found the problem.

The autobuild is committed and the translations should now also work on Windows.

[Lion.Kanzen]

Fixed, not I can play again. Thanks a lot all you guys.

[Gallaecio]

Yves, you are a hero.