Bellaz89 Posted June 4, 2020 Report Share Posted June 4, 2020 (edited) README First: The official topic about 0ad spidermonkey update is here: ==================================================================================================================================== Hi there! DISCLAIMER: This is a work in progress. The game starts but might be subject to instabilities, crashes or other imperfections. Just for fun and to learn something new I started adding SM68 support to 0ad. Here you can find the link to my 0ad fork at https://github.com/bellaz89/0ad/tree/rework I don't know what kind of development milestones are planned for 0ad. This is a pure one side project that doesn't pretend to be merged (but of course, if it is useful to someone I am going to be happy) Also note that I do not assure that everything I did is reasonable since this is my first approach with 0ad source base and spidermonkey. Some code could easily be incorrect or redundant. To compile it a system version of mozjs68 has to be installed. use the --with-system-mozjs68 workspace generation option. Tested and developed only on linux (Debian Bullseye 11). Some changes I did: Switched to C++14 due to Spidermonkey 68 requirements JSRuntime direct access is deprecated in favor of JSContext. Therefore ScriptRuntime create a new JSContext. All the methods/functions that used JSRuntime now use JSContext. the Compartment is now JS::Realm Since only a single JSContext is allowed per-thread, ScriptInterface just generates a new global object and populates it. The environment separation is done changing the Realm rather than using a different JSContext. Now the Realm holds the ScriptInterface private data. A macro to switch the Realm was added (CX_IN_REALM) Removed JSAutoRequest Starting adding explicit JS error checks since JS_SetErrorReporter was removed. A lot of API fixes, some are just name changes See links below to get a partial (quite long) list of additional API changes. Increased runtime size to 512 MB to make the tests run without segfault (to optimize). Why increasing the runtime size helped? Support of ArrayBuffer in Serialization/Deserialization is explicitly added since ArrayBuffer is matched by JS_IsArrayBufferViewObject anymore. The used API can be found in js/ArrayBuffer.h Open questions: How to pass data between Contexts. The StructuredClone API was changed and there are some problems continuing using it https://groups.google.com/forum/#!topic/mozilla.dev.tech.js-engine/ViE6Yg8yCvY Script sources are passed with JS::SourceText. Doubts about the right conversions between string format to be done Are the optimizations on SM45 GC still valid in SM68? How passing data efficiently between Realms Update June 8 2020 Basic error tests fixed (updated to mozjs style error output) All tests run, 98% pass. The failing tests are on serialization/deserialization Update June 9 2020 ArrayBuffer tests fixed modifying the serializer/deserializer. Fixed UTF test removing U+D800 (high surrogate) unicode character. All tests run, 99% pass. The last failing test is on exception catching. For the moment it can be ignored. Starting to understand why the GUI doesn't show up when launching pyrogenesis Update 2 Fixed GUI with https://code.wildfiregames.com/D2768 Still crashing when changing page. Update 17 June Patched Spidermonkey 68 source code. the class SharedArrayRawBufferRefs in ./js/public/StructuredClone.hStructuredClone.h (firefox source directory) was tagged as JS_PUBLIC_API to export its symbols. Please note that since Spidermonkey 68 clang is required to compile the codebase (at least on linux). Use instructions in https://github.com/mozilla-spidermonkey/spidermonkey-embedding-examples/blob/esr68/docs/Building SpiderMonkey.md. PersistentRooted is initialized with .init() Applied https://code.wildfiregames.com/D2768 ObjectToIdMap is based on GCHashMap Update 18 June Corrected a cross-compartment bug in CGame::ReallyStartGame, line 329. Does this affect the current 0ad version too?? Added checks to consume error messages I was able to start a game! ...but almost nothing (except the gui!) works properly Update 22 June Fixed error handling issue. Now 100% tests are running It was found that non-ASCII characters in game schema resulted in a crash. Since the presence of these characters is non-essential and limited, these characters were temporarly removed (Identity.js). The problem is under investigation. Probably this is due Game is running with Spidermonkey 68 Todo: Clean up. (better naming.. remove failed attempts to correct issue in the code) Benchmarking/Profiling. See how much the performaces increase/decrease after the upgrade Try link the game with SM78 beta () A big thanks to the WFG team for helping me in this task even though it doesn't follow the 'normal' 0ad development process. I hope this will be helpful in porting the official branch to newer versions of Spidermonkey (New) Spidermonkey 78 Since the next firefox 78 ESR is going to be released this month. I also started to work on the SM78 port using the beta release. Here you can see the branch https://github.com/bellaz89/0ad/tree/rework78 I am able to run a game but I get a lot of failing tests. So expect even a more instabilities. Important: In SM 78 uneval() and .toSource JS methods were removed. Therefore a workaround has to be decided Changes done for SM 78: Lot cosmetic API changes (different method names, method moved to different workspaces..) Seems that it is not possible anymore to disable the dynamic VM grow (to confirm) Needs C++17 to compile Useful links (contain a more in-depth list of changes): https://github.com/mozilla-spidermonkey/spidermonkey-embedding-examples/blob/esr68/docs/Migration Guide.md http://mozilla.6506.n7.nabble.com/Success-with-embedding-ESR52-spidermonkey-td369034.html https://groups.google.com/forum/#!topic/mozilla.dev.tech.js-engine/Mu1arXbBhi8 https://github.com/mozilla-spidermonkey/spidermonkey-embedding-examples Happy Screenshots: Tests. all passing Finally the game runs! mainlog.html entity.zip interestinglog.html Edited July 31, 2020 by Bellaz89 3 2 Quote Link to comment Share on other sites More sharing options...
Lion.Kanzen Posted June 4, 2020 Report Share Posted June 4, 2020 @wraitii @Stan` Quote Link to comment Share on other sites More sharing options...
asterix Posted June 4, 2020 Report Share Posted June 4, 2020 @wraitii @Itms We thank you very much it one of the main blockers why we do not yet support C++14 and visual studio 2017 and vs 19 Quote Link to comment Share on other sites More sharing options...
Stan` Posted June 4, 2020 Report Share Posted June 4, 2020 2 minutes ago, Lion.Kanzen said: @wraitii @Stan` Thanks, he asked us on IRC first 1 Quote Link to comment Share on other sites More sharing options...
asterix Posted June 4, 2020 Report Share Posted June 4, 2020 2 minutes ago, Lion.Kanzen said: @wraitii @Stan` they already know about it since today http://irclogs.wildfiregames.com/2020-06/2020-06-04-QuakeNet-%230ad-dev.log 1 Quote Link to comment Share on other sites More sharing options...
Stan` Posted June 4, 2020 Report Share Posted June 4, 2020 Quote why we do not yet support C++14 and visual studio 2017 and vs 19 Why do you ask? We tend to support whatever the worst plateform support. For visual studio 2017 and 2019 extra work is needed. Quote Link to comment Share on other sites More sharing options...
Bellaz89 Posted June 8, 2020 Author Report Share Posted June 8, 2020 (edited) On 6/4/2020 at 7:08 PM, Stan` said: Why do you ask? We tend to support whatever the worst plateform support. For visual studio 2017 and 2019 extra work is needed. I think he meant that spidermonkey 68 requires at least c++ 14 to be used. This is unfortunately unavoidable.. I will add this change to the changelog. Edited June 8, 2020 by Bellaz89 1 Quote Link to comment Share on other sites More sharing options...
Bellaz89 Posted June 8, 2020 Author Report Share Posted June 8, 2020 Update All tests run, 98% pass. The failing tests are on serialization/deserialization (see below) 2 Quote Link to comment Share on other sites More sharing options...
Stan` Posted June 8, 2020 Report Share Posted June 8, 2020 Can you list the features that require C++14 in the main game source code? Quote Link to comment Share on other sites More sharing options...
Bellaz89 Posted June 8, 2020 Author Report Share Posted June 8, 2020 (edited) Hi! In fact none. It is just that SM 68 defines some values in its headers that usec++ 14 digit separators ( see https://riptutorial.com/cplusplus/example/31798/digit-separator ). Edited June 8, 2020 by Bellaz89 1 Quote Link to comment Share on other sites More sharing options...
Sturm Posted June 9, 2020 Report Share Posted June 9, 2020 Actually, what is the latest version of Spidermonkey nowadays? Does it follow the same numbering of Firefox? If so, why 68 and not 76/77? - https://en.wikipedia.org/wiki/SpiderMonkey - https://mozilla-spidermonkey.github.io/ Regards, Sturm 1 Quote Link to comment Share on other sites More sharing options...
Stan` Posted June 9, 2020 Report Share Posted June 9, 2020 4 hours ago, Sturm said: Actually, what is the latest version of Spidermonkey nowadays? Does it follow the same numbering of Firefox? If so, why 68 and not 76/77? - https://en.wikipedia.org/wiki/SpiderMonkey - https://mozilla-spidermonkey.github.io/ Regards, Sturm It does but we follow the ESR releases Which are more stable. The plan was to update to 52 then 62 then 68 but if @Bellaz89 manages to pull this off it's even better. The small incremental updates were to make sure we could track changes in an easy manner. 1 Quote Link to comment Share on other sites More sharing options...
wraitii Posted June 9, 2020 Report Share Posted June 9, 2020 50 minutes ago, Stan` said: The plan was to update to 52 then 62 then 68 but if @Bellaz89 manages to pull this off it's even better. The small incremental updates were to make sure we could track changes in an easy manner. I think @Itms will agree that we probably should move incrementally anyways, if only to make sure we aren't missing something that will come back and haunt us. The work will likely not be repeated too much regardless. 1 Quote Link to comment Share on other sites More sharing options...
Bellaz89 Posted June 9, 2020 Author Report Share Posted June 9, 2020 (edited) @Sturm Hi, As @Stan` pointed out, the main technical point is that SM 68 is the latest ESR version (SM 78 ESR is going to be released at the end of this month) There are some other points: There is at least some documentation for SM 68 (see links in the first post). This helps a lot in understanding the direction that has to be followed to port the code without getting crazy. SM 68 is the latest packaged version of spidermonkey on my linux distribution. Since SM stability depends a lot on compilation flags, I prefer to use something already tested by someone else. Also IMHO there won't be an huge performance difference between version 68 and 76. The main reason to switch, apart from support, is going to be some Javascript feature that could be beneficial to use. I think much more performances are going to be obtained taking advantages of the C++ and Javascript API additions since SM 45. Of course if someone will decide to port 0ad to a version of SM newer than 68 is going to be good and beneficial (like I decided to jump directly to SM 68). It is the beauty of open source . In this regards e.g. Webassembly could be something to be explored to improve performances of pyrogenesis without changing the runtime code (or even making the runtime lighter moving part of the non-IO code, like the IA, the mapgen and the pathfinder, to WASM). This would be very beneficial from the modding/hacking perspective. But this is going to be decided by someone else Edited June 9, 2020 by Bellaz89 1 Quote Link to comment Share on other sites More sharing options...
Bellaz89 Posted June 9, 2020 Author Report Share Posted June 9, 2020 (edited) Update ArrayBuffer tests fixed modifying the serializer/deserializer. Fixed UTF test removing U+D800 (high surrogate) unicode character. All tests run, 99% pass. The last failing test is on exception catching. For the moment it can be ignored. Starting to understand why the GUI doesn't show up when launching pyrogenesis Update 2 Fixed GUI with https://code.wildfiregames.com/D2768 Still crashing when changing page. (see below for test screenshots) (see below for gui screenshots) Edited June 9, 2020 by Bellaz89 1 Quote Link to comment Share on other sites More sharing options...
gameboy Posted June 9, 2020 Report Share Posted June 9, 2020 @Itms Please help. Quote Link to comment Share on other sites More sharing options...
Itms Posted June 15, 2020 Report Share Posted June 15, 2020 Hi @Bellaz89, nice to meet you! I have seen your work on SM in my emails, but as you've probably been told I was unavailable because of my PhD. I will be back to programming on 0 A.D. next week, and I will start with the SM 45->52 migration. I have already prepared this migration in a way that will probably conflict with yours, so I apologize in advance. When I'm back I propose that we schedule a chat together (IRC or elsewhere) to discuss how I plan to do the migration to 52 and whether you could help, and after that migration, I would be happy to help you include your work in the game by targeting SM 60, 68 and probably 78 at that point! To answer the question above, yes we plan to update incrementally every time. Thanks for your work and see you soon 3 1 2 Quote Link to comment Share on other sites More sharing options...
Bellaz89 Posted June 15, 2020 Author Report Share Posted June 15, 2020 3 hours ago, Itms said: Hi @Bellaz89, nice to meet you! I have seen your work on SM in my emails, but as you've probably been told I was unavailable because of my PhD. I will be back to programming on 0 A.D. next week, and I will start with the SM 45->52 migration. I have already prepared this migration in a way that will probably conflict with yours, so I apologize in advance. When I'm back I propose that we schedule a chat together (IRC or elsewhere) to discuss how I plan to do the migration to 52 and whether you could help, and after that migration, I would be happy to help you include your work in the game by targeting SM 60, 68 and probably 78 at that point! To answer the question above, yes we plan to update incrementally every time. Thanks for your work and see you soon Hi, It seems that you have my same problem.. (doing a PhD ). Don't worry. As I previously stated, I expected my effort not to be 100% compatible to the direction you want to follow. And also last days I found many issues to solve that popped out when I tried to link a debug version of SM68. I fear that my efforts will require quite more time to result in an usable version on 0ad. I think that going through SM 52 is the most reasonable way to have a stable code without wasting too much time. I will continue to try going directly to SM 68 and see what are the issues to face to port 0ad to that version of Spidermonkey. Of course I would be happy to discuss together and give as much help I can. Thanks for 0ad and see you soon! 1 1 Quote Link to comment Share on other sites More sharing options...
Bellaz89 Posted June 17, 2020 Author Report Share Posted June 17, 2020 (edited) I tried to link a debug version of Spidermokey. This undercovered a lot of nasty bugs Update Patched Spidermonkey 68 source code. the class SharedArrayRawBufferRefs in ./js/public/StructuredClone.hStructuredClone.h (firefox source directory) was tagged as JS_PUBLIC_API to export its symbols. Please note that since Spidermonkey 68 clang is required to compile the codebase (at least on linux). Use instructions in https://github.com/mozilla-spidermonkey/spidermonkey-embedding-examples/blob/esr68/docs/Building SpiderMonkey.md. PersistentRooted is initialized with .init() Applied https://code.wildfiregames.com/D2768 ObjectToIdMap is based on GCHashMap I am almost able to run a game ... Edited June 17, 2020 by Bellaz89 3 Quote Link to comment Share on other sites More sharing options...
Bellaz89 Posted June 18, 2020 Author Report Share Posted June 18, 2020 (edited) Update: Corrected a cross-compartment bug in CGame::ReallyStartGame, line 329. Does this affect the current 0ad version too?? Added checks to consume error messages I was able to start a game! ...but almost nothing (except the gui!) works properly Edited June 18, 2020 by Bellaz89 Quote Link to comment Share on other sites More sharing options...
Bellaz89 Posted June 22, 2020 Author Report Share Posted June 22, 2020 Update: Fixed error handling issue. Now 100% tests are running It was found that non-ASCII characters in game schema resulted in a crash. Since the presence of these characters is non-essential and limited, these characters were temporarly removed (Identity.js). The problem is under investigation. Probably this is due Game is running with Spidermonkey 68 Todo: Clean up. (better naming.. remove failed attempts to correct issue in the code) Benchmarking/Profiling. See how much the performaces increase/decrease after the upgrade Try link the game with SM78 beta () A big thanks to the WFG team for helping me in this task even though it doesn't follow the 'normal' 0ad development process. I hope this will be helpful in porting the official branch to newer versions of Spidermonkey Running game: 5 1 Quote Link to comment Share on other sites More sharing options...
Lion.Kanzen Posted June 22, 2020 Report Share Posted June 22, 2020 55 minutes ago, Bellaz89 said: Update: Fixed error handling issue. Now 100% tests are running It was found that non-ASCII characters in game schema resulted in a crash. Since the presence of these characters is non-essential and limited, these characters were temporarly removed (Identity.js). The problem is under investigation. Probably this is due Game is running with Spidermonkey 68 Todo: Clean up. (better naming.. remove failed attempts to correct issue in the code) Benchmarking/Profiling. See how much the performaces increase/decrease after the upgrade Try link the game with SM78 beta () A big thanks to the WFG team for helping me in this task even though it doesn't follow the 'normal' 0ad development process. I hope this will be helpful in porting the official branch to newer versions of Spidermonkey Running game: Increases the performance? 1 Quote Link to comment Share on other sites More sharing options...
Bellaz89 Posted June 22, 2020 Author Report Share Posted June 22, 2020 That's a good question. I didn't profile the game and for now I just did debug builds so far, therefore I cannot tell. However I expect some improvement since many changes were made on the underling VM since V45. Also, for the future, having the possibility to use WebAssebly (and WebWorkers as suggested on #0ad-dev) could have for sure an impact. Stay tuned for benchmarks! 3 Quote Link to comment Share on other sites More sharing options...
Bellaz89 Posted July 2, 2020 Author Report Share Posted July 2, 2020 (edited) Hi there! Probably you noticed that there were not updates for some time. Unfortunately I was quite busy with other stuff and in the future I don't think I will have more time to dedicate on this (at least for ~8months). Therefore I would say that this experiment finishes here and I am happy about the outcomes. Some WFG members are already working on porting the main branch of 0ad to more recent version of Spidermonkey, so don't worry about this. Last micro update. I tried to do a very rough and inaccurate performance test just looking at the framerate. It seems that my modifications reduced a bit (not dramatically) the FPS. In my opinion this is due to the fact that many optimizations of the GC parameters were done for SM45 and are not useful anymore or even counterproductive for most recent versions of Spidermonkey. I think that there will be the need to spend some time in profiling the VM to get the best achievable performance. Edited July 2, 2020 by Bellaz89 4 2 1 Quote Link to comment Share on other sites More sharing options...
happyconcepts Posted July 30, 2020 Report Share Posted July 30, 2020 On 6/18/2020 at 11:18 AM, Bellaz89 said: Update: Corrected a cross-compartment bug in CGame::ReallyStartGame, line 329. Does this affect the current 0ad version too?? Is there a cross-compartment bug in CGame::ReallyStartGame, line 329 affecting A23? Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.