Jump to content

Spidermonkey 68 (and 78) (WIP) (Running game!)


Recommended Posts

README First:

The official topic about 0ad spidermonkey update is here:

====================================================================================================================================

Hi there!

DISCLAIMER: This is a work in progress. The game starts but might be subject to instabilities, crashes or other imperfections.

Just for fun and to learn something new I started adding SM68 support to 0ad.

Here you can find the link to my 0ad fork at https://github.com/bellaz89/0ad/tree/rework

I don't know what kind of development milestones are planned for 0ad. This is a pure one side project that doesn't pretend to be merged (but of course, if it is useful to someone I am going to be happy)

Also note that I do not assure that everything I did is reasonable since this is my first approach with 0ad source base and spidermonkey. Some code could easily be incorrect or redundant.

To compile it a system version of mozjs68 has to be installed. use the --with-system-mozjs68 workspace generation option. Tested and developed only on linux (Debian Bullseye 11).

 

Some changes I did:

  • Switched to C++14 due to Spidermonkey 68 requirements
  • JSRuntime direct access is deprecated in favor of JSContext. Therefore ScriptRuntime create a new JSContext. All the methods/functions that used JSRuntime now use JSContext.
  • the Compartment is now JS::Realm
  • Since only a single JSContext is allowed per-thread, ScriptInterface just generates a new global object and populates it. The environment separation is done changing the Realm rather than using a different JSContext. Now the Realm holds the ScriptInterface private data. A macro to switch the Realm was added (CX_IN_REALM)
  • Removed JSAutoRequest
  • Starting adding explicit JS error checks since JS_SetErrorReporter was removed.
  • A lot of API fixes, some are just name changes
  • See links below to get a partial (quite long) list of additional API changes.
  • Increased runtime size to 512 MB to make the tests run without segfault (to optimize). Why increasing the runtime size helped?
  • Support of ArrayBuffer in Serialization/Deserialization is explicitly added since ArrayBuffer is matched by JS_IsArrayBufferViewObject anymore. The used API can be found in js/ArrayBuffer.h

Open questions:

  • How to pass data between Contexts. The StructuredClone API was changed and there are some problems continuing using it https://groups.google.com/forum/#!topic/mozilla.dev.tech.js-engine/ViE6Yg8yCvY 
  • Script sources are passed with JS::SourceText. Doubts about the right conversions between string format to be done
  • Are the  optimizations on SM45 GC still valid in SM68?
  • How passing data efficiently between Realms

 

Update June 8 2020

  • Basic error tests fixed (updated to mozjs style error output)
  • All tests run, 98% pass. The failing tests are on serialization/deserialization

Update June 9 2020

  • ArrayBuffer tests fixed modifying the serializer/deserializer.
  • Fixed UTF test removing U+D800 (high surrogate) unicode character.
  • All tests run, 99% pass. The last failing test is on exception catching. For the moment it can be ignored.
  • Starting to understand why the GUI doesn't show up when launching pyrogenesis

Update 2

Update 17 June

Update 18 June

  • Corrected a cross-compartment bug in CGame::ReallyStartGame, line 329. Does this affect the current 0ad version too??
  • Added checks to consume error messages
  • I was able to start a game!
  • ...but almost nothing (except the gui!) works properly

Update 22 June

  • Fixed error handling issue. Now 100% tests are running
  • It was found that non-ASCII characters in game schema resulted in a crash. Since the presence of these characters is non-essential and limited, these characters were temporarly removed (Identity.js). The problem is under investigation. Probably this is due
  • Game is running with Spidermonkey 68

 

Todo:

  • Clean up. (better naming.. remove failed attempts to correct issue in the code)
  • Benchmarking/Profiling. See how much the performaces increase/decrease after the upgrade
  • Try link the game with SM78 beta (B))

A big thanks to the WFG team for helping me in this task even though it doesn't follow the 'normal' 0ad development process. I hope this will be helpful in porting the official branch to newer versions of Spidermonkey :)

 

(New) Spidermonkey 78

Since the next firefox 78 ESR is going to be released this month. I also started to work on the SM78 port using the beta release. Here you can see the branch https://github.com/bellaz89/0ad/tree/rework78

I am able to run a game but I get a lot of failing tests. So expect even a more instabilities.

Important: In SM 78 uneval() and .toSource JS methods were removed. Therefore a workaround has to be decided

Changes done for SM 78:

  • Lot cosmetic API changes (different method names, method moved to different workspaces..)
  • Seems that it is not possible anymore to disable the dynamic VM grow (to confirm)
  • Needs C++17 to compile

 

Useful links (contain a more in-depth list of changes):

https://github.com/mozilla-spidermonkey/spidermonkey-embedding-examples/blob/esr68/docs/Migration Guide.md

http://mozilla.6506.n7.nabble.com/Success-with-embedding-ESR52-spidermonkey-td369034.html

https://groups.google.com/forum/#!topic/mozilla.dev.tech.js-engine/Mu1arXbBhi8

https://github.com/mozilla-spidermonkey/spidermonkey-embedding-examples

 

Happy Screenshots:

Tests. all passing

1991676977_Screenshotfrom2020-06-2209-42-05.png.2b15d52164e7fede224b8b38778be113.png

 

Screenshot from 2020-06-09 14-33-06.png

518596554_Screenshotfrom2020-06-1915-43-20.thumb.png.dd6343cdac870f1aca63517ba5cc3c9c.png

 

Screenshot from 2020-06-17 15-45-19.png

 

Finally the game runs!

 

1154595354_Screenshotfrom2020-06-2209-45-01.thumb.png.029cd40a098ee5015b44ddbd8eaa88ae.png

Screenshot from 2020-06-09 10-18-04.png

 

 

mainlog.html entity.zip interestinglog.html

Edited by Bellaz89
  • Like 3
  • Thanks 2
Link to post
Share on other sites

Quote

why we do not yet support C++14  and visual studio 2017 and vs 19

Why do you ask? We tend to support whatever the worst plateform support. For visual studio 2017 and 2019 extra work is needed.

Link to post
Share on other sites
Posted (edited)
On 6/4/2020 at 7:08 PM, Stan` said:

 

Why do you ask? We tend to support whatever the worst plateform support. For visual studio 2017 and 2019 extra work is needed.

I think he meant that spidermonkey 68 requires at least c++ 14 to be used. This is unfortunately unavoidable..

I will add this change to the changelog.

Edited by Bellaz89
  • Thanks 1
Link to post
Share on other sites
4 hours ago, Sturm said:

Actually, what is the latest version of Spidermonkey nowadays? Does it follow the same numbering of Firefox? If so, why 68 and not 76/77?

https://en.wikipedia.org/wiki/SpiderMonkey

https://mozilla-spidermonkey.github.io/

Regards,

Sturm

It does but we follow the ESR releases :) Which are more stable.

The plan was to update to 52 then 62 then 68 but if @Bellaz89 manages to pull this off it's even better.

The small incremental updates were to make sure we could track changes in an easy manner.

  • Like 1
Link to post
Share on other sites
50 minutes ago, Stan` said:

The plan was to update to 52 then 62 then 68 but if @Bellaz89 manages to pull this off it's even better.

The small incremental updates were to make sure we could track changes in an easy manner.

I think @Itms will agree that we probably should move incrementally anyways, if only to make sure we aren't missing something that will come back and haunt us. The work will likely not be repeated too much regardless.

  • Sad 1
Link to post
Share on other sites
Posted (edited)

@Sturm

Hi, :)

As @Stan` pointed out, the main technical point is that SM 68 is the latest ESR version (SM 78 ESR is going to be released at the end of this month)

There are some other points:

  • There is at least some documentation for SM 68 (see links in the first post). This helps a lot in understanding the direction that has to be followed to port the code without getting crazy.
  • SM 68 is the latest packaged version of spidermonkey on my linux distribution. Since SM stability depends a lot on compilation flags, I prefer to use something already tested by someone else.

Also IMHO there won't be an huge performance difference between version 68 and 76. The main reason to switch, apart from support, is going to be some Javascript feature that could be beneficial to use. I think much more performances are going to be obtained taking advantages of the C++ and Javascript API additions since SM 45.

Of course if someone will decide to port 0ad to a version of SM newer than 68 is going to be good and beneficial (like I decided to jump directly to SM 68). It is the beauty of open source ;) .

In this regards e.g. Webassembly could be something to be explored to improve performances of pyrogenesis without changing the runtime code (or even making the runtime lighter moving part of the non-IO code, like the IA, the mapgen and the pathfinder, to WASM). This would be very beneficial from the modding/hacking perspective. But this is going to be decided by someone else :)

Edited by Bellaz89
  • Thanks 1
Link to post
Share on other sites
Posted (edited)

Update

  • ArrayBuffer tests fixed modifying the serializer/deserializer.
  • Fixed UTF test removing U+D800 (high surrogate) unicode character.
  • All tests run, 99% pass. The last failing test is on exception catching. For the moment it can be ignored.
  • Starting to understand why the GUI doesn't show up when launching pyrogenesis

Update 2

(see below for test screenshots)

(see below for gui screenshots)

Edited by Bellaz89
  • Like 1
Link to post
Share on other sites

Hi @Bellaz89, nice to meet you! I have seen your work on SM in my emails, but as you've probably been told I was unavailable because of my PhD. I will be back to programming on 0 A.D. next week, and I will start with the SM 45->52 migration. I have already prepared this migration in a way that will probably conflict with yours, so I apologize in advance.

When I'm back I propose that we schedule a chat together (IRC or elsewhere) to discuss how I plan to do the migration to 52 and whether you could help, and after that migration, I would be happy to help you include your work in the game by targeting SM 60, 68 and probably 78 at that point! To answer the question above, yes we plan to update incrementally every time.

Thanks for your work and see you soon (y)

  • Like 3
  • Thanks 1
  • Sad 2
Link to post
Share on other sites
3 hours ago, Itms said:

Hi @Bellaz89, nice to meet you! I have seen your work on SM in my emails, but as you've probably been told I was unavailable because of my PhD. I will be back to programming on 0 A.D. next week, and I will start with the SM 45->52 migration. I have already prepared this migration in a way that will probably conflict with yours, so I apologize in advance.

When I'm back I propose that we schedule a chat together (IRC or elsewhere) to discuss how I plan to do the migration to 52 and whether you could help, and after that migration, I would be happy to help you include your work in the game by targeting SM 60, 68 and probably 78 at that point! To answer the question above, yes we plan to update incrementally every time.

Thanks for your work and see you soon (y)

Hi,

It seems that you have my same problem.. (doing a PhD :D).

Don't worry. As I previously stated, I expected my effort not to be 100% compatible to the direction you want to follow. And also last days I found many issues to solve that popped out when I tried to link a debug version of SM68. I fear that my efforts will require quite more time to result in an usable version on 0ad. I think that going through SM 52 is the most reasonable way to have a stable code without wasting too much time. I will continue to try going directly to SM 68 and see what are the issues to face to port 0ad to that version of Spidermonkey. Of course I would be happy to discuss together and give as much help I can.

Thanks for 0ad and see you soon!

  • Like 1
  • Thanks 1
Link to post
Share on other sites
Posted (edited)

I tried to link a debug version of Spidermokey. This undercovered a lot of nasty bugs

Update

 

I am almost able to run a game ...

Edited by Bellaz89
  • Like 3
Link to post
Share on other sites
Posted (edited)

Update:

  • Corrected a cross-compartment bug in CGame::ReallyStartGame, line 329. Does this affect the current 0ad version too??
  • Added checks to consume error messages
  • I was able to start a game!
  • ...but almost nothing (except the gui!) works properly

 

Screenshot from 2020-06-18 20-14-43.png

Edited by Bellaz89
Link to post
Share on other sites

Update:

  • Fixed error handling issue. Now 100% tests are running
  • It was found that non-ASCII characters in game schema resulted in a crash. Since the presence of these characters is non-essential and limited, these characters were temporarly removed (Identity.js). The problem is under investigation. Probably this is due
  • Game is running with Spidermonkey 68

 

Todo:

  • Clean up. (better naming.. remove failed attempts to correct issue in the code)
  • Benchmarking/Profiling. See how much the performaces increase/decrease after the upgrade
  • Try link the game with SM78 beta (B))

 

A big thanks to the WFG team for helping me in this task even though it doesn't follow the 'normal' 0ad development process. I hope this will be helpful in porting the official branch to newer versions of Spidermonkey :)

 

Running game:

1263764680_Screenshotfrom2020-06-2209-45-01.thumb.png.8c3a2264a8d785299a6ef2a67f38d105.png

  • Like 5
  • Thanks 1
Link to post
Share on other sites
55 minutes ago, Bellaz89 said:

Update:

  • Fixed error handling issue. Now 100% tests are running
  • It was found that non-ASCII characters in game schema resulted in a crash. Since the presence of these characters is non-essential and limited, these characters were temporarly removed (Identity.js). The problem is under investigation. Probably this is due
  • Game is running with Spidermonkey 68

 

Todo:

  • Clean up. (better naming.. remove failed attempts to correct issue in the code)
  • Benchmarking/Profiling. See how much the performaces increase/decrease after the upgrade
  • Try link the game with SM78 beta (B))

 

A big thanks to the WFG team for helping me in this task even though it doesn't follow the 'normal' 0ad development process. I hope this will be helpful in porting the official branch to newer versions of Spidermonkey :)

 

Running game:

1263764680_Screenshotfrom2020-06-2209-45-01.thumb.png.8c3a2264a8d785299a6ef2a67f38d105.png

Increases the performance?

  • Like 1
Link to post
Share on other sites

That's a good question.

I didn't profile the game and for now I just did debug builds so far, therefore I cannot tell. However I expect some improvement since many changes were made on the underling VM since V45. Also, for the future, having the possibility to use WebAssebly (and WebWorkers as suggested on #0ad-dev) could have for sure an impact.

Stay tuned for benchmarks!

  • Like 3
Link to post
Share on other sites
  • 2 weeks later...
Posted (edited)

Hi there!

Probably you noticed that there were not updates for some time. Unfortunately I was quite busy with other stuff and in the future I don't think I will have more time to dedicate on this (at least for ~8months). Therefore I would say that this experiment finishes here and I am happy about the outcomes. Some WFG members are already working on porting the main branch of 0ad to more recent version of Spidermonkey, so don't worry about this.

 

Last micro update. I tried to do a very rough and inaccurate performance test just looking at the framerate. It seems that my modifications reduced a bit (not dramatically) the FPS. In my opinion this is due to the fact that many optimizations of the GC parameters were done for SM45 and are not useful anymore or even counterproductive for most recent versions of Spidermonkey. I think that there will be the need to spend some time in profiling the VM to get the best achievable performance.

Edited by Bellaz89
  • Like 4
  • Thanks 2
  • Sad 1
Link to post
Share on other sites
  • 4 weeks later...
On 6/18/2020 at 11:18 AM, Bellaz89 said:

Update:

  • Corrected a cross-compartment bug in CGame::ReallyStartGame, line 329. Does this affect the current 0ad version too??

 

Is there a cross-compartment bug in CGame::ReallyStartGame, line 329 affecting A23? 

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...