Itms Posted July 29, 2020 Report Share Posted July 29, 2020 Hi everyone In this thread, I am going to present the SpiderMonkey upgrade work I have dedicated myself to since around a month. The aim of the thread is to explain what this upgrade is, why it is such a big change, and how I organized it. Please don't hesitate if you have comments or questions on anything! What is SpiderMonkey? SpiderMonkey is Mozilla's JavaScript engine, used first and foremost in Firefox. As you probably know, 0 A.D. is written in C++ but a large part of the gameplay is programmed in JavaScript, in order to be easily modifiable by mods. In order to execute the JavaScript code of the game, we include SpiderMonkey inside 0 A.D. (this is also called embedding). We are not the only ones to embed SpiderMonkey: for instance, GNOME (a well-known desktop environment) also embeds SpiderMonkey. I will often use the acronym SM to designate SpiderMonkey. No kinks here, just laziness to type! All about upgrading The advantage of embedding SM is that we don't have to take care of bugs or complicated JavaScript interpretation quirks: the folks at Mozilla have us covered. The downside, however, is that we are often out of the loop on the evolution of SpiderMonkey. In order to get the improvements of the engine, we have to upgrade it to recent versions, and this is very complicated. A big part of 0 A.D. is built upon SpiderMonkey, so when the latter changes, we have a lot of work to do to adapt. On top of this, SM upgrades are big changes. SM evolves along with Firefox (which has a new release every 8 weeks), but a stand-alone SM version is only released for embedders approximately once a year, when Firefox releases an Extended Support Release (a kind of Firefox LTS, which is, for instance, the one available on Debian). See the Firefox release calendar. The difficulty of upgrading is that, between two ESR releases, SM accumulates a lot of changes, and, whereas Firefox is always on top of them, other front-ends like 0 A.D. are faced with numerous breakages which could create bugs. Additionally, at 0 A.D. we haven't upgraded SM for a long time due to the difficulty of upgrading. Thus, it is also difficult to find information to upgrade to SM releases that are already outdated. It is not much easier to jump to the current SM release: so many changes have happened that we would be unable to make sense of them all at once. Current status I upgraded SM to version 38 in the ends of 2016. At that point, we were mostly up-to-date: version 45 was already out but it was still getting some fixes, and the stand-alone SM45 was not officially released. Nowadays, stand-alone SM versions are almost never officially released, however the SM development team has started to put out more and more news (see their brand new website) so things are certainly getting better. I couldn't work on the SM45 upgrade until the summer of 2019, and even then I left a few bugs in, that we will discover in the next posts I am releasing today the upgrade to SM52, which is a big one (hence the forum thread). That brings us to "only" two years late, as SM52 stopped receiving fixes in June of 2018. It is now necessary to work on the upgrade to SM60, SM68, and SM78 (the latter is still getting fixes until the beginning of 2021). However, I am going to work on other urgent things in August. I need to improve a lot of things with the CI, Jenkins and the handling of patches, and I would like to allow contributors to use git instead of SVN. I hope these exciting news will make you forgive me for delaying the SM upgrades In any case, I want to get back to further SM work as soon as possible. In this other thread, @Bellaz89 did the daunting work of making SM68 work with no previous knowledge of 0 A.D.! Many of you probably saw it. Bellaz was then very helpful and answered all my questions concerning the upgrade. His spadework, which cannot be committed just as-is, made the actual upgrade a breeze; and apparently there will be no big work to perform on 0 A.D. between SM52 and SM68, apart from an encoding change. We will see... Overview of the upgrade The upgrade can be found at this git branch. It is very big. Not counting the changes to SpiderMonkey itself, in the engine of 0 A.D., it consists of 145 files changed, with 2667 insertions(+) and 3243 deletions(-)! The objective is to split the massive change into independent parts, which can be understood, tested and committed one by one. I tried to make a majority of changes before the actual upgrade. That way, the upgrade itself is leaner. There are a few changes that I will be able to do just after the upgrade. Right now they are not uploaded, but I will announce them here later. I will also keep the posts below updated whenever I update or commit a part of the upgrade. You are probably wondering about the performance. From my few tests, the performance is identical in SM45 and SM52. If I'm not mistaken, the structural changes in SM52 pave the way for performance improvements in the next SM versions. On top of this, I have written a patch for a huge performance bottleneck (D2919) that was very apparent under SM52, and I have identified a couple of easy improvements that I will list in a ticket soon. Finally, Bellaz thinks that we have a few JIT optimizations (i.e. whether JavaScript code should be compiled, which costs some time, but makes it faster) which have become counterproductive. We could try and tweak our JIT options and see if we can improve the performance that way. You can already checkout the branch and test it! Now, to the details for the curious programmers! 5 4 Quote Link to comment Share on other sites More sharing options...
Itms Posted July 29, 2020 Author Report Share Posted July 29, 2020 (edited) Removal of ObjectToIDMap In source/scriptinterface/third_party/ObjectToIDMap.h, we have some code, taken from Firefox, allowing us to keep a map where JavaScript objects are the keys, and the values can be anything (integers or strings in our case). This code was upgraded in Firefox 45 and the old version, which I had forgotten to update, stops working with SpiderMonkey 52. However, I discovered recently that the new code doesn't work either! SpiderMonkey developers tell me it is never actually used in Firefox. In fact, this code is not needed. If we have an object to which we want to attach an ID, we can use Symbol properties. JS symbols act as unique property keys that allow us to safely assign an ID to an object. Additionally, they are ignored when iterating over an object's properties, so the rest of the code won't know about them. f1fa7a1 (up for review at D2897 in Phabricator) In the AI manager, we used object maps in some code about "serializable prototypes" which is completely dead: no code calls into it. So I removed it. @wraitii has written D2746 which actually implements prototype serialization, but he tells me he would rather rebase his work after the upgrade. e9f2161 : I replaced the other use of object maps, in the serialization code, with symbol properties. 727f26c : This allowed me to remove that code entirely. This means less external code to maintain! Sometimes what we actually need is a map with custom keys, and JS objects as values. Currently, we do that using std::map<KeyType,JS::Heap<JSObject*>>, but I believe we should use JS::GCHashMap. I'll look into that in the future. Other cleanups 43bb449 : Previously, in SM, objects rooted on the stack did not have a default constructor, so we had created our own class wrapping them with such a default constructor. Nowadays, this wrapper is not needed anymore, so I removed its remaining use and deleted the class. Also less code to maintain. 58ba02e : I performed an optional cleanup that would make the refactoring below easier. c148184 : We have a ScriptInterface::IsExceptionPending function that just calls a SpiderMonkey function with an unneeded GC request, so I removed it. TODO: I did that under SM52 and I don't know if it works under SM45. If it does, it should be committed before the upgrade. Refactoring of runtimes, contexts, and compartments This is the main part of the upgrade Until SM45, there were three levels of containers in the SpiderMonkey API. Each thread of the engine needed to have a runtime to execute some JavaScript. In each runtime, there could be several isolated contexts. In our case, we had one context for the simulation, another one for each page of the GUI, etc. The isolation of contexts allows us to separate properly the different parts of our code. Inside contexts, there could be several isolated compartments, but we didn't use that. Each context had its own and only compartment. In our code, JSContext is abstracted by our ScriptInterface class. We also have ScriptRuntime for JSRuntime. In SM52, the runtime and context levels are merged. Runtimes are removed from the API. There must be one context per thread, and isolation of the different parts of a thread must be done using different compartments. This is summarized in the following picture made by @Bellaz89: The difficulty here is that calls to the SpiderMonkey API still use a context. Previously, it was enough to get the context from the script interface, and call SM: the target code would be in the correct compartment. Now, getting the context is not enough: we first need to make the context enter the compartment of the script interface, and only then, we can call the SM API on the context. In order to prevent mistakes, I decided to perform a big change. I think it's better to prevent getting the context from a script interface directly. I wrote a Request struct which allows one to get the context from a script interface, and which automatically enters and leave the associated compartment. In the past, we had many issues with JSAutoRequest, which is needed to properly handle the memory of the JS runtimes. In a lot of cases, we forgot to add JSAutoRequest, which created bugs. The new Request object allows us to prevent these mistakes, because it automatically opens a memory request as well. The changes go like this: c65093e and 25dc831 : Prevent getting the context, except through a Request that acts just like JSAutoRequest. I made two commits for readability but they should be committed together. This change removes quite some boilerplate code previously caused by JSAutoRequest. 5d4e4c4 : Rename "context private" to "compartment private" (this commit is huge but changes nothing, just the name of classes and variables). 8142e29 : Here I make the central change: each ScriptRuntime now has one single context, whereas each ScriptInterface is associated with a compartment. Getting the context through a Request automatically enters the correct compartment. The Request also provides the global object associated with the compartment. It is still possible to get the general context but this is now advertised as unsafe. 521fd88 : There is a check made on the creation/destruction of runtimes, I change it to be made on the creation/destruction of contexts. 17ce48e : Rename ScriptRuntime to ScriptContext to match its new role (this commit is big but changes nothing, just the name of classes and variables). 0 A.D. is now ready to use SM52! Actual SM upgrade be5b0ed and 3013473 : In two commits, I upgrade the SM52 files, including a prebuilt version for Windows, and I adapt the build and premake scripts to use the new version. Simple API changes A lot of commits are made to match the API changes. They are very simple. I will merge them as a single commit in the future, along with links to SpiderMonkey's bugtracker. Right now I keep them separate so that I have a list. Some of them deserve a note: c4fc364 : In this one I remove all uses of runtimes, which become "unsafe contexts" with no compartment barrier. 8c6476a : In this one I use the new API for structured clones. Those take a new argument which is a scope, depending on whether the clone will be used in the same thread, or even in the same run of the application. My choices of scopes have to be reviewed. Changes in the error reporting API cc31596: The heaviest API change is due to the error reporting made differently. Previously, both errors and warnings in JS code would make SM call the error/warning reporter, which we implemented ourselves. Now, only the warning reporter can be supplied. Errors are always handled by the exception system, in which all exceptions have to be caught. Sometimes, they are caught by the JS code itself, but sometimes they are not. In this case, the engine has to catch them and display them. If they stay uncaught, the engine can crash when a new exception is raised, or when performing a core operation such as creating a new compartment. Thus, I had to refactor a big part of our JS error handling. I think I have improved the design which was a bit hacky, but my code is not foolproof, as uncaught exceptions are maybe still possible. Edited August 1, 2020 by Itms 3 4 Quote Link to comment Share on other sites More sharing options...
Bellaz89 Posted July 30, 2020 Report Share Posted July 30, 2020 (edited) Good job! I really like your solution of using the context through Request . I wonder if in Spidermonkey V52 there is already the bug in StructuredClone API I encountered in SM68 and SM78. Back then I had to patch Spidermonkey itself to make it possible to use the StructuredClone API (a symbol was not exported). Is this the case also for SM52? Edited July 30, 2020 by Bellaz89 1 Quote Link to comment Share on other sites More sharing options...
Itms Posted July 30, 2020 Author Report Share Posted July 30, 2020 Thanks! No, I didn't need to patch SM to use the clone API. It probably breaks in a more recent version. 1 Quote Link to comment Share on other sites More sharing options...
badosu Posted July 31, 2020 Report Share Posted July 31, 2020 Awesome, great work! I was surprised with the amount of javascript 0ad relies on, hopefully this paves the way for massive performance improvements in the subsequent versions even though the bottleneck seems to be somewhere else. Quote Link to comment Share on other sites More sharing options...
Sturm Posted July 31, 2020 Report Share Posted July 31, 2020 (edited) Hi folks! Considering that the Wildfire team prefers to be more cautious in Spidermonkey updates, the idea is to update to V52 in Alpha 24 and then to Spidermonkey V62 in Alpha 25? Is that right? Regards, Sturm. Edited July 31, 2020 by Sturm Quote Link to comment Share on other sites More sharing options...
Itms Posted July 31, 2020 Author Report Share Posted July 31, 2020 Not necessarily. I am going to try to update SpiderMonkey as far as possible in A24. I don't want to unnecessarily delay the release of A24, but the less obsolete SM is, the fewer problems we will have. 1 Quote Link to comment Share on other sites More sharing options...
Nescio Posted August 1, 2020 Report Share Posted August 1, 2020 On 7/29/2020 at 7:12 PM, Itms said: Please don't hesitate if you have comments or questions on anything! What is SpiderMonkey? SpiderMonkey is Mozilla's JavaScript engine, used first and foremost in Firefox. As you probably know, 0 A.D. is written in C++ but a large part of the gameplay is programmed in JavaScript, in order to be easily modifiable by mods. In order to execute the JavaScript code of the game, we include SpiderMonkey inside 0 A.D. (this is also called embedding). We are not the only ones to embed SpiderMonkey: for instance, GNOME (a well-known desktop environment) also embeds SpiderMonkey. What is the advantage of embedding? Why not simply use e.g. Firefox's? Also, I noticed the libraries/source/spidermonkey/ folder in the svn development version is already 3.3 GB. For comparison, the public mod is 4.2 GB and the .svn folder, i.e. the entire revision history since the start, is 5.8 GB. Why is the SM folder so large? Firefox incorporates the most up-to-date version of SM (doesn't it?), but is only c. 64 MB to download and c. 250 MB installed size. On 7/31/2020 at 5:07 PM, Itms said: Not necessarily. I am going to try to update SpiderMonkey as far as possible in A24. I don't want to unnecessarily delay the release of A24, but the less obsolete SM is, the fewer problems we will have. Great! I like your ambition! The more up to date software is, the better! However, I read SM68 requires C++14 and SM78 uses C++17, so I guess that'll require a lot more work? Quote Link to comment Share on other sites More sharing options...
smiley Posted August 1, 2020 Report Share Posted August 1, 2020 20 minutes ago, Nescio said: Why is the SM folder so large? Maybe build artifacts? The SM binary distributed with 0AD is not that large as well. Somewhat matches Firefox sizes. Quote Link to comment Share on other sites More sharing options...
Itms Posted August 2, 2020 Author Report Share Posted August 2, 2020 11 hours ago, Nescio said: What is the advantage of embedding? Why not simply use e.g. Firefox's? There is a confusion here: we do use the same engine as Firefox. However, we need to build the game with it, so we need the development files of SpiderMonkey. When one installs Firefox, they only download the compiled Firefox which includes (or comes alongside) the compiled SpiderMonkey. Similarly, when we release 0 A.D., we distribute the compiled SpiderMonkey which is quite small. 0 A.D. end-users can download the compiled SpiderMonkey to run 0 A.D.; on the other hand, they cannot build 0 A.D. For that, one needs the development files which are heavy. If you want to take a look, in some package managers, the former is called mozjs52, the latter is called libmozjs52-dev. 12 hours ago, Nescio said: Also, I noticed the libraries/source/spidermonkey/ folder in the svn development version is already 3.3 GB. That is because you built SpiderMonkey. If you checkout a fresh SVN copy, that folder is 98 MB (which is already heavy: it's the development files). It is worth mentioning that SpiderMonkey devs improved the size of the source SM tarball so in SM52 that folder is now only 35 MB. When you build a big project such as SpiderMonkey, the compiler outputs a huge lot of intermediate build binary files that take up a lot of space. If you want to reclaim that space after building, you can delete the folder libraries/source/spidermonkey/mozjs-45.0.2/. The built files, needed by 0 A.D., are copied elsewhere at the end of a build. 12 hours ago, Nescio said: However, I read SM68 requires C++14 and SM78 uses C++17, so I guess that'll require a lot more work? Not really. It is SpiderMonkey that constrains which C++ version we support. Right now, it's a lot of work to migrate to C++14/17: that work is upgrading SM Once SM is upgraded, supporting new versions consists of a three-line change in premake and possibly some small adjustments. Upgrading SM also means dropping some old compilers, which is work, but it's communication work with package managers rather that programming work. Quote Link to comment Share on other sites More sharing options...
Nescio Posted August 2, 2020 Report Share Posted August 2, 2020 Great, many thanks for the clarification! Quote Link to comment Share on other sites More sharing options...
asterix Posted August 2, 2020 Report Share Posted August 2, 2020 3 hours ago, Nescio said: Great, many thanks for the clarification! 4 hours ago, Itms said: There is a confusion here: we do use the same engine as Firefox. However, we need to build the game with it, so we need the development files of SpiderMonkey. When one installs Firefox, they only download the compiled Firefox which includes (or comes alongside) the compiled SpiderMonkey. Similarly, when we release 0 A.D., we distribute the compiled SpiderMonkey which is quite small. 0 A.D. end-users can download the compiled SpiderMonkey to run 0 A.D.; on the other hand, they cannot build 0 A.D. For that, one needs the development files which are heavy. If you want to take a look, in some package managers, the former is called mozjs52, the latter is called libmozjs52-dev. That is because you built SpiderMonkey. If you checkout a fresh SVN copy, that folder is 98 MB (which is already heavy: it's the development files). It is worth mentioning that SpiderMonkey devs improved the size of the source SM tarball so in SM52 that folder is now only 35 MB. When you build a big project such as SpiderMonkey, the compiler outputs a huge lot of intermediate build binary files that take up a lot of space. If you want to reclaim that space after building, you can delete the folder libraries/source/spidermonkey/mozjs-45.0.2/. The built files, needed by 0 A.D., are copied elsewhere at the end of a build. Not really. It is SpiderMonkey that constrains which C++ version we support. Right now, it's a lot of work to migrate to C++14/17: that work is upgrading SM Once SM is upgraded, supporting new versions consists of a three-line change in premake and possibly some small adjustments. Upgrading SM also means dropping some old compilers, which is work, but it's communication work with package managers rather that programming work. Btw according to this https://www.mozilla.org/en-US/firefox/78.0/releasenotes/ this latest esr version supports old MacOS this what you call lts. 1 Quote Link to comment Share on other sites More sharing options...
Nescio Posted September 3, 2020 Report Share Posted September 3, 2020 On 7/29/2020 at 7:12 PM, Itms said: I am releasing today the upgrade to SM52, which is a big one (hence the forum thread). It's not merged into the svn development version yet, is it? Quote Link to comment Share on other sites More sharing options...
bb_ Posted September 3, 2020 Report Share Posted September 3, 2020 It is not indeed Quote Link to comment Share on other sites More sharing options...
wraitii Posted November 12, 2020 Report Share Posted November 12, 2020 Following @Itms lack of time recently, I have taken up the mantle of the spidermonkey migration. For SM52, most of what he had done is directly usable and I plan to commit the upgrade in the upcoming days. I will be uploading patches to Phabricator, but for those using git, you can access the whole thing here: https://github.com/wraitii/0ad/tree/sm52-clean My plan is to try and move rather quickly up the ladder, since we need SM78 for Python3 support (which we need to not be dropped by linux distros), and that will take 3 more rounds of upgrading. 6 3 Quote Link to comment Share on other sites More sharing options...
wraitii Posted November 18, 2020 Report Share Posted November 18, 2020 Hello everyone, as promised above, SM52 is now merged in SVN. If you experience issues with building the game, please drop by #0ad-dev on IRC or create a ticket on trac (or, if all else fail, write about it here ). Work will now continue towards SM60. 2 4 Quote Link to comment Share on other sites More sharing options...
wowgetoffyourcellphone Posted November 18, 2020 Report Share Posted November 18, 2020 Someone post to media? @Sundiata Quote Link to comment Share on other sites More sharing options...
gameboy Posted November 18, 2020 Report Share Posted November 18, 2020 @wraitii Work will now continue towards SM60. Are you implementing it? Quote Link to comment Share on other sites More sharing options...
wraitii Posted November 24, 2020 Report Share Posted November 24, 2020 Hello everyone, I have just committed SM60. I'll be around for any bug report should I have missed something. The most interesting thing about this upgrade was that we now compile the game using the C++14 standard. Now, I will work towards using Visual Studio 2017 (I'll make another forum post about that), and once that's done, we can start looking at SM68. 3 1 Quote Link to comment Share on other sites More sharing options...
wowgetoffyourcellphone Posted November 25, 2020 Report Share Posted November 25, 2020 15 hours ago, wraitii said: Hello everyone, I have just committed SM60. I'll be around for any bug report should I have missed something. The most interesting thing about this upgrade was that we now compile the game using the C++14 standard. Now, I will work towards using Visual Studio 2017 (I'll make another forum post about that), and once that's done, we can start looking at SM68. For stupid people like me, can you list some pros and cons to these improvements? Quote Link to comment Share on other sites More sharing options...
Silier Posted November 25, 2020 Report Share Posted November 25, 2020 1.) We need to get rid of python2 because debian removed us already, they stopped supporting it. 2.) In order to do so we need sm78 3.) To have 0ad buildable with sm78 we need c++17 what means vs17 4.) And there is speed improvements in sm Cons: We drop support for older compilers (that, do not support c++17) Quote Link to comment Share on other sites More sharing options...
wraitii Posted November 25, 2020 Report Share Posted November 25, 2020 2 hours ago, wowgetoffyourcellphone said: For stupid people like me, can you list some pros and cons to these improvements? I think I'll try to recap the changes after C++17 too. I expect to be a little shamed by the amount of detail @Itms provided in this initial post, though That being said, what Angen said is correct. Quote Link to comment Share on other sites More sharing options...
nani Posted November 25, 2020 Report Share Posted November 25, 2020 3 hours ago, Angen said: 1.) We need to get rid of python2 because debian removed us already, they stopped supporting it. Where is python 2 used in pyrogeneis? just for the build process? Quote Link to comment Share on other sites More sharing options...
Stan` Posted November 25, 2020 Report Share Posted November 25, 2020 19 minutes ago, nani said: Where is python 2 used in pyrogeneis? just for the build process? Yes, python2 is used only for the build process. 1 Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.