Testing pathfinder update

[Stan`]

20 minutes ago, Kuba386 said:

Yeah, I forgot. If you din't tick the checkbox default number of threads will be used, so by default default number of threads is used

Thank you for packaging this patch

I will take a look at this too.

Actually my memory usage is 3.5 GB for 7 AIs on linux.
 

My memory usage exceeds 4GB (5.5GB at the moment), but I get no crash. I've heard that windows version is 32-bit so cannot allocate more than 4GB.

Windows isn't per se but the application we build is 32 bits.

If you don't pass --large-address-aware in the update workspaces it can only use up to 2GB.

The map changes the consumption greatly.

The crash might only occur on Windows because we rely on undefined behavior somewhere that works on Unix distribution.

 

You're welcome for the packaging. Best way to get generic feedback is too make it easier for people to test.

[Kuba386]

I've set up a game with 6 AIs, one hard defensive and 5 very hard aggressive. I went to kitchen to eat something and after 30-40 minutes when I went back game was already using all of my RAM (8GB, game was using like 7.3) and 4 GB of swap.

Hard defensive AI has crushed opponents and number of units on map was ~700. When I exited to main menu game was still using all that RAM.

This memory is not used by pathfinder components. I don't know what is allocating it, maybe JS.

[Stan`]

Okay so I guess it's just a coincidence.

[OptimusShepard]

2 hours ago, Kuba386 said:

What if you use one thread or disable threading? (0 threads)

With 1 or 0 in settings the game is freezing or crashing.

crashlog.dmp

[Stan`]

Call stack is not really helpful sadly

 	msvcr100.dll!__VEC_memcpy()	Inconnu
 	ucrtbase.dll!try_get_function()	Inconnu
 	ucrtbase.dll!try_get_function()	Inconnu
>	kernel32.dll!75ec8484()	Inconnu

 

[Stan`]

Didn't occur on the random map I just tried but it occured on Corinthian Ismuth (4) 4AI very hard. Full Callstack

 	00000000()	Inconnu	Code non-utilisateur
 	[Les frames ci-dessous sont peut-être incorrects et/ou manquants]	Inconnu	Aucun symbole n'a été chargé.
>	pyrogenesis.exe!InitDbghelp() Ligne 80	C++	Les symboles ont été chargés.
 	pyrogenesis.exe!ModuleInit(volatile int * initState, __int64(*)() init=0x013badc0) Ligne 47	C++	Les symboles ont été chargés.
 	[Cadre en ligne] pyrogenesis.exe!sym_init() Ligne 110	C++	Les symboles ont été chargés.
 	pyrogenesis.exe!wdbg_sym_WalkStack(__int64(*)(const _tagSTACKFRAME64 *, unsigned int) cb=0x013bb470, unsigned int cbData=0, _CONTEXT & context={...}, const wchar_t * lastFuncToSkip=0x479b4018) Ligne 317	C++	Les symboles ont été chargés.
 	pyrogenesis.exe!debug_DumpStack(wchar_t * buf=0x009000de, unsigned int maxChars=262033, void * pcontext=0x00d3e700, const wchar_t * lastFuncToSkip=0x479b4018) Ligne 1742	C++	Les symboles ont été chargés.
 	pyrogenesis.exe!debug_BuildErrorMessage(const wchar_t * description=0x00d3ef18, const wchar_t * filename=0x0147e9d0, int line=139, const char * func=0x0147e948, void * context=0x00d3e700, const wchar_t * lastFuncToSkip=0x479b4018, ErrorMessageMem * emm=0x00d3e5f8) Ligne 304	C++	Les symboles ont été chargés.
 	pyrogenesis.exe!debug_DisplayError(const wchar_t * description=0x00d3ef18, unsigned int flags=4, void * context=0x00d3e700, const wchar_t * lastFuncToSkip=0x479b4018, const wchar_t * pathname=0x0147e978, int line=139, const char * func=0x0147e948, volatile int * suppress=0x014f2bbc) Ligne 471	C++	Les symboles ont été chargés.
 	pyrogenesis.exe!debug_OnAssertionFailure(const wchar_t * expr=0x0147e9e0, volatile int * suppress=0x014f2bbc, const wchar_t * file=0x0147e978, int line=139, const char * func=0x0147e948) Ligne 563	C++	Les symboles ont été chargés.
 	pyrogenesis.exe!Allocators::DynamicArena::AllocateNewChunk() Ligne 139	C++	Les symboles ont été chargés.
 	[Cadre en ligne] pyrogenesis.exe!Allocators::DynamicArena::{ctor}(unsigned int) Ligne 120	C++	Les symboles ont été chargés.
 	pyrogenesis.exe!CPatchRData::RenderBases(const std::vector<CPatchRData *,std::allocator<CPatchRData *> > & patches={...}, const CShaderDefines & context={...}, ShadowMap * shadow=0x0a3f7ad0, bool isDummyShader=false, const std::shared_ptr<CShaderProgram> & dummy={...}) Ligne 732	C++	Les symboles ont été chargés.
 	pyrogenesis.exe!TerrainRenderer::RenderTerrainShader(const CShaderDefines & context={...}, int cullGroup, ShadowMap * shadow=0x0a3f7ad0) Ligne 503	C++	Les symboles ont été chargés.
 	pyrogenesis.exe!CRenderer::RenderPatches(const CShaderDefines & context={...}, int cullGroup) Ligne 951	C++	Les symboles ont été chargés.
 	pyrogenesis.exe!CRenderer::RenderSubmissions(const CBoundingBoxAligned & waterScissor={...}) Ligne 1542	C++	Les symboles ont été chargés.
 	pyrogenesis.exe!CRenderer::RenderScene(Scene & scene={...}) Ligne 1897	C++	Les symboles ont été chargés.
 	pyrogenesis.exe!CGameView::Render() Ligne 489	C++	Les symboles ont été chargés.
 	pyrogenesis.exe!Render() Ligne 223	C++	Les symboles ont été chargés.
 	pyrogenesis.exe!Frame() Ligne 416	C++	Les symboles ont été chargés.
 	pyrogenesis.exe!RunGameOrAtlas(int argc=1, const char * * argv=0x010f0008) Ligne 634	C++	Les symboles ont été chargés.
 	pyrogenesis.exe!SDL_main(int argc=1, char * * argv=0x010f0008) Ligne 680	C++	Les symboles ont été chargés.
 	pyrogenesis.exe!main_utf8(int argc=1, char * * argv=0x010f0008) Ligne 126	C	Les symboles ont été chargés.
 	pyrogenesis.exe!wmain(int argc=1, unsigned short * * wargv=0x01002770, unsigned short * wenvp=0x010073a0) Ligne 151	C	Les symboles ont été chargés.
 	[Cadre en ligne] pyrogenesis.exe!invoke_main() Ligne 79	C++	Code non-utilisateur. Les symboles ont été chargés.
 	pyrogenesis.exe!__scrt_common_main_seh() Ligne 253	C++	Code non-utilisateur. Les symboles ont été chargés.
 	pyrogenesis.exe!CallStartupWithinTryBlock() Ligne 364	C++	Les symboles ont été chargés.
 	kernel32.dll!77698484()	Inconnu	Code non-utilisateur. Impossible de trouver ou d'ouvrir le fichier PDB.
 	ntdll.dll!779e3ab8()	Inconnu	Code non-utilisateur. Impossible de trouver ou d'ouvrir le fichier PDB.
 	ntdll.dll!779e3a88()	Inconnu	Code non-utilisateur. Impossible de trouver ou d'ouvrir le fichier PDB.

 Final line of the crash

static Status InitDbghelp()
{
	hProcess = GetCurrentProcess();

	dbghelp_ImportFunctions();

	// set options
	// notes:
	// - can be done before SymInitialize; we do so in case
	//   any of the options affect it.
	// - do not set directly - that would zero any existing flags.
	DWORD opts = pSymGetOptions();
	//opts |= SYMOPT_DEBUG;	// lots of debug spew in output window
	opts |= SYMOPT_DEFERRED_LOADS;	// the "fastest, most efficient way"
	opts |= SYMOPT_LOAD_LINES;
->	opts |= SYMOPT_UNDNAME;
	pSymSetOptions(opts);

The game reach Gigs fast and then crash. (14 error messages)

ERROR: JavaScript error: uncaught exception: out of memory
ERROR: JavaScript error: uncaught exception: out of memory
ERROR: JavaScript error: uncaught exception: out of memory
ERROR: JavaScript error: uncaught exception: out of memory
ERROR: JavaScript error: uncaught exception: out of memory
ERROR: JavaScript error: uncaught exception: out of memory
ERROR: JavaScript error: uncaught exception: out of memory
ERROR: JavaScript error: uncaught exception: out of memory
ERROR: JavaScript error: uncaught exception: out of memory
ERROR: JavaScript error: uncaught exception: out of memory
ERROR: JavaScript error: uncaught exception: out of memory
ERROR: JavaScript error: uncaught exception: out of memory
ERROR: JavaScript error: uncaught exception: out of memory
ERROR: JavaScript error: uncaught exception: out of memory

 12 Threads

[wowgetoffyourcellphone]

I get no performance improvement from 2 or 4. Perhaps I have not applied everything correctly, but it's pretty easy to apply thanks to stan. My laptop is 5 years old however, and grfx card is only a 730M.

[balduin]

Did anybody an analysis of the memory usage. Can you see what part of the game requires the most memory?

For the JavaScript parts, would it be possible to use WebAssembly to speed up some parts of the game?

[Guest]

Why write C++ code to compile it to that and run on an interpreter? That’s what WebAssembly is.

Script runtimes are initialised with the memory usage. It’s pretty small compared to “gigabytes”. So I doubt it’s a leak in JS. Considering the magnitude described.

[balduin]

@(-_-) You are right. I misunderstood the goal of WebAssembly. I thought there is a transpiler which allows you to compile JavaScript ahead-of-time to WebAssembly.

However, I my first question still remains:

Quote

Did anybody an analysis of the memory usage. Can you see what part of the game requires the most memory?

 

[Rolf Dew]

Would this patch be advisable to use on a mac book air? 2017 model 8 gb ram, 1.8 GHz processor Intel i5, 128 gb hd

Edited December 24, 2018 by Rolf Dew

[Stan`]

6 hours ago, wowgetoffyourcellphone said:

I get no performance improvement from 2 or 4. Perhaps I have not applied everything correctly, but it's pretty easy to apply thanks to stan. My laptop is 5 years old however, and grfx card is only a 730M.

Maybe it only occurs in late game ? Since the beginning is not that intensive.

 

3 hours ago, balduin said:

@(-_-) You are right. I misunderstood the goal of WebAssembly. I thought there is a transpiler which allows you to compile JavaScript ahead-of-time to WebAssembly.

However, I my first question still remains:

 

Diagnosing memory is a bit tough though it's easier on Linux when using Valgrind. The thing Valgrind does is trying to find leaks and issues with memory. But it slows drastically the game.

That's something Im willing to learn but I haven't had the time yet

also the debug build on Windows is broken because of Fcollada. I need the help of @vladislavbelov to figure out why it compile on vs2015 but makes and unusable dll. And why it crashes on vs2013 but only on the release build and on models exported by blender and containing animations. And why it does so too When packaging on Windows. According to leper an assert somewhere could be removed but that would only not make it crash and not fix the bug.

 

2 hours ago, Rolf Dew said:

Would this patch be advisable to use on a mac book air? 2017 model 8 gb ram, 1.8 GHz processor Intel i5, 128 gb hd

You might want to try since it's very likely it will be in upcoming versions of the game.  Just don't go too hard on the number of threads and avoid playing on your lap. Your hardware is supposed to be Designed for thermal protection so you should be fine.

[wowgetoffyourcellphone]

2 hours ago, stanislas69 said:

Maybe it only occurs in late game ? Since the beginning is not that intensive.

 

Beginning of a match is generally around 50-60 fps and it just gets worse from there, with or without the patch. I'll try it with 6 threads and see how it goes. 

[Stan`]

@Kuba386 Would you mind updating your patch ? At least removing those white spaces changes so that we can see the actual changes, and taking into account the comments. Also can you post a message here when you are online and willing to play ?

[Kuba386]

21 hours ago, stanislas69 said:

Would you mind updating your patch ? At least removing those white spaces changes so that we can see the actual changes, and taking into account the comments

Yeah, sure. I'm working on update but recently (because of christmas) I had no time, and development was frozen. Today I will probably start actual work again.

21 hours ago, stanislas69 said:

Also can you post a message here when you are online and willing to play ?

Yeah, sure.

On 12/24/2018 at 10:02 AM, stanislas69 said:

it's very likely it will be in upcoming versions of the game.

I hope it will . But first we need to analyze and solve bugs reported here, and maybe do what I was talking about on phabricator.

Memory leak is kind of a release blocker.

[Kuba386]

On 12/23/2018 at 11:57 PM, stanislas69 said:

Didn't occur on the random map I just tried but it occured on Corinthian Ismuth (4) 4AI very hard. Full Callstack

I don't have any problems with 0-1 threads... That's actually bad because it makes issue harder to find for me. Is it OS-specific?

[OptimusShepard]

15 minutes ago, Kuba386 said:

I don't have any problems with 0-1 threads... That's actually bad because it makes issue harder to find for me. Is it OS-specific?

Maybe you need some creater workload. I had nearly 2000 units on the map when my system crashes.

FYI I use Win10.

[Kuba386]

 

On 12/26/2018 at 12:07 AM, stanislas69 said:

Would you mind updating your patch ?

I've uploaded new patch to D14 that should fix memory leak issues.

On 12/26/2018 at 12:07 AM, stanislas69 said:

At least removing those white spaces changes so that we can see the actual changes, and taking into account the comments.

My code editor (CLion) messed the code by automatically reformatting source files... That's why there are whitespace differences. How can you tell arcanist to ignore them?

[Stan`]

34 minutes ago, Kuba386 said:

My code editor (CLion) messed the code by automatically reformatting source files... That's why there are whitespace differences. How can you tell arcanist to ignore them?

You can't. Can't you disable the auto reformat ?
https://secure.phabricator.com/T3498
 

34 minutes ago, Kuba386 said:

I've uploaded new patch to D14 that should fix memory leak issues.

Great gonna test it and maybe bundle it.

[Stan`]

Looks like you patch didn't apply correctly. What you can do to remove the whitespaces is to do the diff manually using svn diff or git diff if you are using git.

[Guest]

(Diff got an option to ignore whitespaces btw)

[sarcoma]

10 hours ago, Kuba386 said:

 

I've uploaded new patch to D14 that should fix memory leak issues.

My code editor (CLion) messed the code by automatically reformatting source files... That's why there are whitespace differences. How can you tell arcanist to ignore them? 

https://www.jetbrains.com/help/clion/code-style-c-c.html

 

[Kuba386]

I've updated my patch, fixed issues with whitespace differences and some other small changes. Now it builds successfully and passes tests.

[Stan`]

I tried the previous iteration and didn't get any leaks though it was a quick test so it's going in the right direction. Still getting errors in the menu. Maybe Missing a '+' to tell JavaScript to interpret the option as an integer

 

[Rolf Dew]

Does this work on mac yet? I do believe that my 0 ad app crashed when I applied the patch, I will try again as I recently updated my os from sierra to high sierra. Note I tried the patch when I uploaded my previous post, so the issue might be resolved now.

Edited December 30, 2018 by Rolf Dew