Leaderboard

Hi everyone In this thread, I am going to present the SpiderMonkey upgrade work I have dedicated myself to since around a month. The aim of the thread is to explain what this upgrade is, why it is such a big change, and how I organized it. Please don't hesitate if you have comments or questions on anything! What is SpiderMonkey? SpiderMonkey is Mozilla's JavaScript engine, used first and foremost in Firefox. As you probably know, 0 A.D. is written in C++ but a large part of the gameplay is programmed in JavaScript, in order to be easily modifiable by mods. In order to execute the JavaScript code of the game, we include SpiderMonkey inside 0 A.D. (this is also called embedding). We are not the only ones to embed SpiderMonkey: for instance, GNOME (a well-known desktop environment) also embeds SpiderMonkey. I will often use the acronym SM to designate SpiderMonkey. No kinks here, just laziness to type! All about upgrading The advantage of embedding SM is that we don't have to take care of bugs or complicated JavaScript interpretation quirks: the folks at Mozilla have us covered. The downside, however, is that we are often out of the loop on the evolution of SpiderMonkey. In order to get the improvements of the engine, we have to upgrade it to recent versions, and this is very complicated. A big part of 0 A.D. is built upon SpiderMonkey, so when the latter changes, we have a lot of work to do to adapt. On top of this, SM upgrades are big changes. SM evolves along with Firefox (which has a new release every 8 weeks), but a stand-alone SM version is only released for embedders approximately once a year, when Firefox releases an Extended Support Release (a kind of Firefox LTS, which is, for instance, the one available on Debian). See the Firefox release calendar. The difficulty of upgrading is that, between two ESR releases, SM accumulates a lot of changes, and, whereas Firefox is always on top of them, other front-ends like 0 A.D. are faced with numerous breakages which could create bugs. Additionally, at 0 A.D. we haven't upgraded SM for a long time due to the difficulty of upgrading. Thus, it is also difficult to find information to upgrade to SM releases that are already outdated. It is not much easier to jump to the current SM release: so many changes have happened that we would be unable to make sense of them all at once. Current status I upgraded SM to version 38 in the ends of 2016. At that point, we were mostly up-to-date: version 45 was already out but it was still getting some fixes, and the stand-alone SM45 was not officially released. Nowadays, stand-alone SM versions are almost never officially released, however the SM development team has started to put out more and more news (see their brand new website) so things are certainly getting better. I couldn't work on the SM45 upgrade until the summer of 2019, and even then I left a few bugs in, that we will discover in the next posts I am releasing today the upgrade to SM52, which is a big one (hence the forum thread). That brings us to "only" two years late, as SM52 stopped receiving fixes in June of 2018. It is now necessary to work on the upgrade to SM60, SM68, and SM78 (the latter is still getting fixes until the beginning of 2021). However, I am going to work on other urgent things in August. I need to improve a lot of things with the CI, Jenkins and the handling of patches, and I would like to allow contributors to use git instead of SVN. I hope these exciting news will make you forgive me for delaying the SM upgrades In any case, I want to get back to further SM work as soon as possible. In this other thread, @Bellaz89 did the daunting work of making SM68 work with no previous knowledge of 0 A.D.! Many of you probably saw it. Bellaz was then very helpful and answered all my questions concerning the upgrade. His spadework, which cannot be committed just as-is, made the actual upgrade a breeze; and apparently there will be no big work to perform on 0 A.D. between SM52 and SM68, apart from an encoding change. We will see... Overview of the upgrade The upgrade can be found at this git branch. It is very big. Not counting the changes to SpiderMonkey itself, in the engine of 0 A.D., it consists of 145 files changed, with 2667 insertions(+) and 3243 deletions(-)! The objective is to split the massive change into independent parts, which can be understood, tested and committed one by one. I tried to make a majority of changes before the actual upgrade. That way, the upgrade itself is leaner. There are a few changes that I will be able to do just after the upgrade. Right now they are not uploaded, but I will announce them here later. I will also keep the posts below updated whenever I update or commit a part of the upgrade. You are probably wondering about the performance. From my few tests, the performance is identical in SM45 and SM52. If I'm not mistaken, the structural changes in SM52 pave the way for performance improvements in the next SM versions. On top of this, I have written a patch for a huge performance bottleneck (D2919) that was very apparent under SM52, and I have identified a couple of easy improvements that I will list in a ticket soon. Finally, Bellaz thinks that we have a few JIT optimizations (i.e. whether JavaScript code should be compiled, which costs some time, but makes it faster) which have become counterproductive. We could try and tweak our JIT options and see if we can improve the performance that way. You can already checkout the branch and test it! Now, to the details for the curious programmers!

Removal of ObjectToIDMap In source/scriptinterface/third_party/ObjectToIDMap.h, we have some code, taken from Firefox, allowing us to keep a map where JavaScript objects are the keys, and the values can be anything (integers or strings in our case). This code was upgraded in Firefox 45 and the old version, which I had forgotten to update, stops working with SpiderMonkey 52. However, I discovered recently that the new code doesn't work either! SpiderMonkey developers tell me it is never actually used in Firefox. In fact, this code is not needed. If we have an object to which we want to attach an ID, we can use Symbol properties. JS symbols act as unique property keys that allow us to safely assign an ID to an object. Additionally, they are ignored when iterating over an object's properties, so the rest of the code won't know about them. f1fa7a1 (up for review at D2897 in Phabricator) In the AI manager, we used object maps in some code about "serializable prototypes" which is completely dead: no code calls into it. So I removed it. @wraitii has written D2746 which actually implements prototype serialization, but he tells me he would rather rebase his work after the upgrade. e9f2161 : I replaced the other use of object maps, in the serialization code, with symbol properties. 727f26c : This allowed me to remove that code entirely. This means less external code to maintain! Sometimes what we actually need is a map with custom keys, and JS objects as values. Currently, we do that using std::map<KeyType,JS::Heap<JSObject*>>, but I believe we should use JS::GCHashMap. I'll look into that in the future. Other cleanups 43bb449 : Previously, in SM, objects rooted on the stack did not have a default constructor, so we had created our own class wrapping them with such a default constructor. Nowadays, this wrapper is not needed anymore, so I removed its remaining use and deleted the class. Also less code to maintain. 58ba02e : I performed an optional cleanup that would make the refactoring below easier. c148184 : We have a ScriptInterface::IsExceptionPending function that just calls a SpiderMonkey function with an unneeded GC request, so I removed it. TODO: I did that under SM52 and I don't know if it works under SM45. If it does, it should be committed before the upgrade. Refactoring of runtimes, contexts, and compartments This is the main part of the upgrade Until SM45, there were three levels of containers in the SpiderMonkey API. Each thread of the engine needed to have a runtime to execute some JavaScript. In each runtime, there could be several isolated contexts. In our case, we had one context for the simulation, another one for each page of the GUI, etc. The isolation of contexts allows us to separate properly the different parts of our code. Inside contexts, there could be several isolated compartments, but we didn't use that. Each context had its own and only compartment. In our code, JSContext is abstracted by our ScriptInterface class. We also have ScriptRuntime for JSRuntime. In SM52, the runtime and context levels are merged. Runtimes are removed from the API. There must be one context per thread, and isolation of the different parts of a thread must be done using different compartments. This is summarized in the following picture made by @Bellaz89: The difficulty here is that calls to the SpiderMonkey API still use a context. Previously, it was enough to get the context from the script interface, and call SM: the target code would be in the correct compartment. Now, getting the context is not enough: we first need to make the context enter the compartment of the script interface, and only then, we can call the SM API on the context. In order to prevent mistakes, I decided to perform a big change. I think it's better to prevent getting the context from a script interface directly. I wrote a Request struct which allows one to get the context from a script interface, and which automatically enters and leave the associated compartment. In the past, we had many issues with JSAutoRequest, which is needed to properly handle the memory of the JS runtimes. In a lot of cases, we forgot to add JSAutoRequest, which created bugs. The new Request object allows us to prevent these mistakes, because it automatically opens a memory request as well. The changes go like this: c65093e and 25dc831 : Prevent getting the context, except through a Request that acts just like JSAutoRequest. I made two commits for readability but they should be committed together. This change removes quite some boilerplate code previously caused by JSAutoRequest. 5d4e4c4 : Rename "context private" to "compartment private" (this commit is huge but changes nothing, just the name of classes and variables). 8142e29 : Here I make the central change: each ScriptRuntime now has one single context, whereas each ScriptInterface is associated with a compartment. Getting the context through a Request automatically enters the correct compartment. The Request also provides the global object associated with the compartment. It is still possible to get the general context but this is now advertised as unsafe. 521fd88 : There is a check made on the creation/destruction of runtimes, I change it to be made on the creation/destruction of contexts. 17ce48e : Rename ScriptRuntime to ScriptContext to match its new role (this commit is big but changes nothing, just the name of classes and variables). 0 A.D. is now ready to use SM52! Actual SM upgrade be5b0ed and 3013473 : In two commits, I upgrade the SM52 files, including a prebuilt version for Windows, and I adapt the build and premake scripts to use the new version. Simple API changes A lot of commits are made to match the API changes. They are very simple. I will merge them as a single commit in the future, along with links to SpiderMonkey's bugtracker. Right now I keep them separate so that I have a list. Some of them deserve a note: c4fc364 : In this one I remove all uses of runtimes, which become "unsafe contexts" with no compartment barrier. 8c6476a : In this one I use the new API for structured clones. Those take a new argument which is a scope, depending on whether the clone will be used in the same thread, or even in the same run of the application. My choices of scopes have to be reviewed. Changes in the error reporting API cc31596: The heaviest API change is due to the error reporting made differently. Previously, both errors and warnings in JS code would make SM call the error/warning reporter, which we implemented ourselves. Now, only the warning reporter can be supplied. Errors are always handled by the exception system, in which all exceptions have to be caught. Sometimes, they are caught by the JS code itself, but sometimes they are not. In this case, the engine has to catch them and display them. If they stay uncaught, the engine can crash when a new exception is raised, or when performing a core operation such as creating a new compartment. Thus, I had to refactor a big part of our JS error handling. I think I have improved the design which was a bit hacky, but my code is not foolproof, as uncaught exceptions are maybe still possible.

Hey folks, There is the contribution for today. Way harder than I though it would be. The result in low poly up close is really funky. But at distance I seems fine. Added a proof of concept animation as I had that idea of using span pose and use a deform bones to make the swimming/walking/chilling pose. Happy that it seems to work, maybe add a third wing 'layer' that would scale during closed wings poses so that it would be invisible the rest of the time. Kinda of out of the box idea, don't know if it'll work. What a odd little bird. The animation is bad mind you, a lot of the weight painting is to be tweaked. I was mostly interested about the idea of using that technique to have the same model walking and flying or if it's not necessary because it will never happen in game? Also I learn that the males change their colors during something called eclipse, and look a lot like the female (texture is next by the way). Soooo... I saw that there will a season system that could be implemented, I got some ideas to add texture season specific to the animals that have low variation in general. Using the alpha as a mask for snow for example (there were a lot of bisons like that). I digress. As always feedback and critique are welcome. https://streamable.com/jmuluh The files: Note: It's late and i saw that the texture has the artefacts of before topology cleanup in the low poly version. It doesn't affect the texture because it's out of the uv zone but I'll see to bake something more clean.

https://code.wildfiregames.com/D1323

Hey folks, To get the hang of things here is the model and texture for the european bison. I worked from a 3000 tri model and a 2k complete texture and for the game the I tried to follow the bovidae specs found in the game file. I more than welcome feedback on any aspect. I got the textures from wiki commons and can provide the originals. I think I the back legs should be higher for the european bison but really after tweaking I could't tell. Engine compliant (i hope) version. 574 triangles, a slight more than the bovidae Eye candy hp and 2k texture version The files themselves Source Texture, used a mix of those.

Hey all, Posting this as I'm not sure who specifically to talk to but to preface, I've been playing the game for a few weeks and I think its already amazing and still has so much potential. The one thing I feel that is lacking is the interface. There's some parts of it that aren't very intuitive and also visually could do with an overhaul. I have a lot of time on my hands at the moment due to lockdown and I also have 10 years in the business doing User Experience Research, User Experience Design and User Interface Design. I'd love to offer some help and expertise to the project but not sure who to reach out to about it. GG's

Nice job. It is a shame we didn't have proper Celtic heroes yet as a reference for you.

Come on do not scare the new kid

\public\simulation\components\ResourceTrickle.js - Resources.BuildSchema("nonNegativeDecimal") + + Resources.BuildSchema("decimal") + If one were so inclined, one could post a patch on code.wildfiregames.com

There was this entreprise on my feed and I think it might give ideas for the portraits. pretty sure it's not shared licence but I found the colorisation of old statues a good idea. And why not use that as a reference for drawing that would be legal.

How I make the resource trickle be negative?

I like them. I'm not so sure about the red highlight on Harsi tho. For Craterus, I think it's a bit hard to see his face. Pretty strong shadow. Also, historically Greeks have been depicted with a very curly beard so it would be awesome to show that.

those portraits look great

I offer Survival Team Game Tournament, various settings tg 4v4 3v3 2v2, ffa various buckets players. You like? Enter teams here

As @GunChleoc said the full history is not needed by those who just want the code or art only those actively involved in development really need it to prevent conflicts and resolve them which is what the review process is created to do and currently 0AD uses Phabricator for this not GitHub's interface at all though the mods do use GitHub there review interface is fairly transparent as I'm learning from my experience as vega-strike's lead and going to our first release in 6 years. Enjoy the Choice

Try git clone --depth=1 That will only clone the current commit and save a lot of time. You can always get more history later if you need it with e.g. git fetch --depth=1000

This was also an interesting thread to check out: