Leaderboard

:3

Hi everyone In this thread, I am going to present the SpiderMonkey upgrade work I have dedicated myself to since around a month. The aim of the thread is to explain what this upgrade is, why it is such a big change, and how I organized it. Please don't hesitate if you have comments or questions on anything! What is SpiderMonkey? SpiderMonkey is Mozilla's JavaScript engine, used first and foremost in Firefox. As you probably know, 0 A.D. is written in C++ but a large part of the gameplay is programmed in JavaScript, in order to be easily modifiable by mods. In order to execute the JavaScript code of the game, we include SpiderMonkey inside 0 A.D. (this is also called embedding). We are not the only ones to embed SpiderMonkey: for instance, GNOME (a well-known desktop environment) also embeds SpiderMonkey. I will often use the acronym SM to designate SpiderMonkey. No kinks here, just laziness to type! All about upgrading The advantage of embedding SM is that we don't have to take care of bugs or complicated JavaScript interpretation quirks: the folks at Mozilla have us covered. The downside, however, is that we are often out of the loop on the evolution of SpiderMonkey. In order to get the improvements of the engine, we have to upgrade it to recent versions, and this is very complicated. A big part of 0 A.D. is built upon SpiderMonkey, so when the latter changes, we have a lot of work to do to adapt. On top of this, SM upgrades are big changes. SM evolves along with Firefox (which has a new release every 8 weeks), but a stand-alone SM version is only released for embedders approximately once a year, when Firefox releases an Extended Support Release (a kind of Firefox LTS, which is, for instance, the one available on Debian). See the Firefox release calendar. The difficulty of upgrading is that, between two ESR releases, SM accumulates a lot of changes, and, whereas Firefox is always on top of them, other front-ends like 0 A.D. are faced with numerous breakages which could create bugs. Additionally, at 0 A.D. we haven't upgraded SM for a long time due to the difficulty of upgrading. Thus, it is also difficult to find information to upgrade to SM releases that are already outdated. It is not much easier to jump to the current SM release: so many changes have happened that we would be unable to make sense of them all at once. Current status I upgraded SM to version 38 in the ends of 2016. At that point, we were mostly up-to-date: version 45 was already out but it was still getting some fixes, and the stand-alone SM45 was not officially released. Nowadays, stand-alone SM versions are almost never officially released, however the SM development team has started to put out more and more news (see their brand new website) so things are certainly getting better. I couldn't work on the SM45 upgrade until the summer of 2019, and even then I left a few bugs in, that we will discover in the next posts I am releasing today the upgrade to SM52, which is a big one (hence the forum thread). That brings us to "only" two years late, as SM52 stopped receiving fixes in June of 2018. It is now necessary to work on the upgrade to SM60, SM68, and SM78 (the latter is still getting fixes until the beginning of 2021). However, I am going to work on other urgent things in August. I need to improve a lot of things with the CI, Jenkins and the handling of patches, and I would like to allow contributors to use git instead of SVN. I hope these exciting news will make you forgive me for delaying the SM upgrades In any case, I want to get back to further SM work as soon as possible. In this other thread, @Bellaz89 did the daunting work of making SM68 work with no previous knowledge of 0 A.D.! Many of you probably saw it. Bellaz was then very helpful and answered all my questions concerning the upgrade. His spadework, which cannot be committed just as-is, made the actual upgrade a breeze; and apparently there will be no big work to perform on 0 A.D. between SM52 and SM68, apart from an encoding change. We will see... Overview of the upgrade The upgrade can be found at this git branch. It is very big. Not counting the changes to SpiderMonkey itself, in the engine of 0 A.D., it consists of 145 files changed, with 2667 insertions(+) and 3243 deletions(-)! The objective is to split the massive change into independent parts, which can be understood, tested and committed one by one. I tried to make a majority of changes before the actual upgrade. That way, the upgrade itself is leaner. There are a few changes that I will be able to do just after the upgrade. Right now they are not uploaded, but I will announce them here later. I will also keep the posts below updated whenever I update or commit a part of the upgrade. You are probably wondering about the performance. From my few tests, the performance is identical in SM45 and SM52. If I'm not mistaken, the structural changes in SM52 pave the way for performance improvements in the next SM versions. On top of this, I have written a patch for a huge performance bottleneck (D2919) that was very apparent under SM52, and I have identified a couple of easy improvements that I will list in a ticket soon. Finally, Bellaz thinks that we have a few JIT optimizations (i.e. whether JavaScript code should be compiled, which costs some time, but makes it faster) which have become counterproductive. We could try and tweak our JIT options and see if we can improve the performance that way. You can already checkout the branch and test it! Now, to the details for the curious programmers!

Eso sería genial para cierto mapa!

Thanks @andy5995

Here's a mod for it (GitHub repo): Show Available Housing

@ItmsCartographyMode please.

Indeed. I think both 'mountainous' and 'watery' maps should bump up to the next size up (behind the scenes). This would help to balance out buildable land area among the maps.

Please this. Easy change.

As random maps aren't nearly as good looking as skirmish maps, I don't hardly think that is a solution for @Sundiata's issue.

Alright, you guys do that. lol.

Making random maps the default is my preferred solution: https://code.wildfiregames.com/D2926

These are a bit glaring though. Maybe increase the saturation from EA just enough to make them pop out, but no more. Sort of find the balance between gameplay needs and aesthetic needs.

The colors are for gameplay purposes. EA's colors are too muted. I agree, you're probably right. I could just remove the solid color variations.

Yes, the left selection panel needed to display five columns for formations and stances, therefore there is no reason limit barter resources to four when there is space reserved for five, hence why a fifth resource is supported. I didn't touch the top panel, though (that's beyond the purpose of the patch). The distance between the individual icons remains the same. I have a 3840×2160 screen and play with gui.scale = "2", because otherwise texts are hard to read, but even with gui.scale = "1" I have no difficulty distinguishing icons. That said, I'm biased, of course.

I can edit the scenario maps now. Very nice. But there is one more thing missing: how do I delete already-placed actors? The destroyed Roman camp in the first campaign has to go, and the grass models need to move.

Try making a new folder here: C:\Users\[username]\AppData\Local\0 A.D. alpha\binaries\data\mods\public\maps\scenarios And put the scenario files there and see if they load.

(I was looking for a mind blown one, but I couldn't find it)

During the Theban–Spartan War of 378–362 BC: Diodorus Siculus, 15, 70, 1: From Sicily, Celts and Iberians to the number of two thousand sailed to Corinth, for they had been sent by the tyrant Dionysius to fight in alliance with the Lacedaemonians, and had received pay for five months. The Greeks, in order to make trial of them, led them forth; and they proved their worth in hand-to-hand fighting and in battles and many both of the Boeotians and of their allies were slain by them. Accordingly, having won repute for superior dexterity and courage and rendered many kinds of service, they were given awards by the Lacedaemonians and sent back home at the close of the summer to Sicily. Xen. Hell. 7.1.20: Just after these events had happened, the expedition sent by Dionysius to aid the Lacedaemonians sailed in, numbering more than twenty triremes. And they brought Celts, Iberians, and about fifty horsemen. On the following day the Thebans and the rest, their allies, after forming themselves in detached bodies and filling the plain as far as the sea and as far as the hills adjoining the city, destroyed whatever of value there was in the plain. And the horsemen of the Athenians and of the Corinthians did not approach very near their army, seeing that the enemy were strong and numerous. During the Peloponnesian war, about the Sicilian Expedition in 415-413 BC: Thuc. 6.90: "Thus stands the matter touching my own accusation. And concerning what we are to consult of, both you and I, if I know anything which you yourselves do not, hear it now. [2] We made this voyage into Sicily, first (if we could) to subdue the Sicilians, after them the Italians, after them, to assay the dominion of Carthage, and Carthage itself. [3] If these or most of these enterprises succeeded, then next we should have undertaken Peloponnesus, with the accession both of the Greek forces there and with many mercenary barbarians, Iberians and others of those parts, confessed to be the most warlike of the barbarians that are now. we should also have built many galleys besides these which we have already (there being plenty of timber in Italy); with the which besieging Peloponnesus round, and also taking the cities thereof with our land forces, upon such occasions as should arise from the land, some by assault and some by siege, we hoped easily to have debelled it and afterwards to have gotten the dominion of all Greece. [4] As for money and corn to facilitate some points of this, the places we should have conquered there, besides what here we should have found, would sufficiently have furnished us. Also Italian mercenaries on the side of Athenes against Sicily: Thuc. 7.33: About the same time came unto them also the aid of the Camarinaeans, five hundred men of arms, three hundred darters, and three hundred archers. Also the Geloans sent them men for five galleys, besides four hundred darters and two hundred horsemen. [2] For now all Sicily, except the Agrigentines, who were neutral, but all the rest, who before stood looking on, came in to the Syracusian side against the Athenians. [3] [Nevertheless], the Syracusians, after this blow received amongst the Siculi, held their hands and assaulted not the Athenians for a while. Demosthenes and Eurymedon, having their army now ready, crossed over from Corcyra and the continent with the whole army to the promontory of Iapygia. From thence they went to the Choerades, islands of Iapygia, and here took in certain Iapygian darters to the number of two hundred and fifty, of the Messapian nation. [4] And having renewed a certain ancient alliance with Artas, who reigned there and granted them those darters, they went thence to Metapontum, a city of Italy. There, by virtue of a league, they got two galleys and three hundred darters, which taken aboard, they kept along the shore till they came to the territory of Thurii. [5] Here they found the adverse faction to the Athenians to have been lately driven out in a sedition. [6] And because they desired to muster their army here, that they might see if any were left behind, and persuade the Thurians to join with them freely in the war, and, as things stood, to have for friends and enemies the same that were so to the Athenians; they stayed about that in the territory of the Thurians.

Wait, does this mean the Seleucids can't build any more CCs without Seleucus?

Okay, got it. Took me an hour but I managed to get the model in game after modifying with the feedback. So the back is more flat and the legs a bit higher (minor tweaks). At first I took the scale of the bull but then I read that speciemens tend to be 1.8m to 2.2m hight so here is a ingame screenshot from the Atlas. Added the horse actor and iberian hero swordsman as a scale. Also there are the files in the respectives folders. I just used copy paste on the main art in mods/public for my convienience. I don't know how you want to receive it. About the naming convention, I went for animal_latin-name_variantion . Should it be otherwise? Lastly, I have two questions: - I got that I can rotate entities in the atlas but I don't know how to make them, any help, any use? - Animations are already listed even if the model doesn't have those. Is it because they are limited, and should I fit mine into those already describe (like th hawk that 'walks' by flying) ? art.zip

Removal of ObjectToIDMap In source/scriptinterface/third_party/ObjectToIDMap.h, we have some code, taken from Firefox, allowing us to keep a map where JavaScript objects are the keys, and the values can be anything (integers or strings in our case). This code was upgraded in Firefox 45 and the old version, which I had forgotten to update, stops working with SpiderMonkey 52. However, I discovered recently that the new code doesn't work either! SpiderMonkey developers tell me it is never actually used in Firefox. In fact, this code is not needed. If we have an object to which we want to attach an ID, we can use Symbol properties. JS symbols act as unique property keys that allow us to safely assign an ID to an object. Additionally, they are ignored when iterating over an object's properties, so the rest of the code won't know about them. f1fa7a1 (up for review at D2897 in Phabricator) In the AI manager, we used object maps in some code about "serializable prototypes" which is completely dead: no code calls into it. So I removed it. @wraitii has written D2746 which actually implements prototype serialization, but he tells me he would rather rebase his work after the upgrade. e9f2161 : I replaced the other use of object maps, in the serialization code, with symbol properties. 727f26c : This allowed me to remove that code entirely. This means less external code to maintain! Sometimes what we actually need is a map with custom keys, and JS objects as values. Currently, we do that using std::map<KeyType,JS::Heap<JSObject*>>, but I believe we should use JS::GCHashMap. I'll look into that in the future. Other cleanups 43bb449 : Previously, in SM, objects rooted on the stack did not have a default constructor, so we had created our own class wrapping them with such a default constructor. Nowadays, this wrapper is not needed anymore, so I removed its remaining use and deleted the class. Also less code to maintain. 58ba02e : I performed an optional cleanup that would make the refactoring below easier. c148184 : We have a ScriptInterface::IsExceptionPending function that just calls a SpiderMonkey function with an unneeded GC request, so I removed it. TODO: I did that under SM52 and I don't know if it works under SM45. If it does, it should be committed before the upgrade. Refactoring of runtimes, contexts, and compartments This is the main part of the upgrade Until SM45, there were three levels of containers in the SpiderMonkey API. Each thread of the engine needed to have a runtime to execute some JavaScript. In each runtime, there could be several isolated contexts. In our case, we had one context for the simulation, another one for each page of the GUI, etc. The isolation of contexts allows us to separate properly the different parts of our code. Inside contexts, there could be several isolated compartments, but we didn't use that. Each context had its own and only compartment. In our code, JSContext is abstracted by our ScriptInterface class. We also have ScriptRuntime for JSRuntime. In SM52, the runtime and context levels are merged. Runtimes are removed from the API. There must be one context per thread, and isolation of the different parts of a thread must be done using different compartments. This is summarized in the following picture made by @Bellaz89: The difficulty here is that calls to the SpiderMonkey API still use a context. Previously, it was enough to get the context from the script interface, and call SM: the target code would be in the correct compartment. Now, getting the context is not enough: we first need to make the context enter the compartment of the script interface, and only then, we can call the SM API on the context. In order to prevent mistakes, I decided to perform a big change. I think it's better to prevent getting the context from a script interface directly. I wrote a Request struct which allows one to get the context from a script interface, and which automatically enters and leave the associated compartment. In the past, we had many issues with JSAutoRequest, which is needed to properly handle the memory of the JS runtimes. In a lot of cases, we forgot to add JSAutoRequest, which created bugs. The new Request object allows us to prevent these mistakes, because it automatically opens a memory request as well. The changes go like this: c65093e and 25dc831 : Prevent getting the context, except through a Request that acts just like JSAutoRequest. I made two commits for readability but they should be committed together. This change removes quite some boilerplate code previously caused by JSAutoRequest. 5d4e4c4 : Rename "context private" to "compartment private" (this commit is huge but changes nothing, just the name of classes and variables). 8142e29 : Here I make the central change: each ScriptRuntime now has one single context, whereas each ScriptInterface is associated with a compartment. Getting the context through a Request automatically enters the correct compartment. The Request also provides the global object associated with the compartment. It is still possible to get the general context but this is now advertised as unsafe. 521fd88 : There is a check made on the creation/destruction of runtimes, I change it to be made on the creation/destruction of contexts. 17ce48e : Rename ScriptRuntime to ScriptContext to match its new role (this commit is big but changes nothing, just the name of classes and variables). 0 A.D. is now ready to use SM52! Actual SM upgrade be5b0ed and 3013473 : In two commits, I upgrade the SM52 files, including a prebuilt version for Windows, and I adapt the build and premake scripts to use the new version. Simple API changes A lot of commits are made to match the API changes. They are very simple. I will merge them as a single commit in the future, along with links to SpiderMonkey's bugtracker. Right now I keep them separate so that I have a list. Some of them deserve a note: c4fc364 : In this one I remove all uses of runtimes, which become "unsafe contexts" with no compartment barrier. 8c6476a : In this one I use the new API for structured clones. Those take a new argument which is a scope, depending on whether the clone will be used in the same thread, or even in the same run of the application. My choices of scopes have to be reviewed. Changes in the error reporting API cc31596: The heaviest API change is due to the error reporting made differently. Previously, both errors and warnings in JS code would make SM call the error/warning reporter, which we implemented ourselves. Now, only the warning reporter can be supplied. Errors are always handled by the exception system, in which all exceptions have to be caught. Sometimes, they are caught by the JS code itself, but sometimes they are not. In this case, the engine has to catch them and display them. If they stay uncaught, the engine can crash when a new exception is raised, or when performing a core operation such as creating a new compartment. Thus, I had to refactor a big part of our JS error handling. I think I have improved the design which was a bit hacky, but my code is not foolproof, as uncaught exceptions are maybe still possible.

But when you select "random map", then you want to be surprised whether the map comes with 30% map being water, 20% hills, or 100% land? For this map-aera problem, one could make it so that the map area relates to the passable map area, not the total map aera, but not sure if that makes it really more expected. If map authors are good, every map is unique, so the problem can occur with any setting. For example on a map with many hills or water, Regicide is a different deal. That we have a "random map" item to begin with implies the design principle that random maps adapt to the settings given by the user, not the other way around. It actually has been wished that random maps can restrict the settings that the user can chose, for example requiring a specific victory condition or mapsize. Then "random" won't work anymore without reinventing that (random selection from the maps that support the chosen settings. That has the disadvantage that the user can't predict which maps are subset thereof). It's not uncommon in multiplayer games to remain for half an hour in the gamesetup stage to make 8 players (and the other 6 players that wanted settings and left again) happy with every possible setting. So anything that will shorten that period without reducing the setting expectance will be benefitial. Yes, that was bad. I had already removed 20-30% of the forests and there is still too few space. The map author actually said it's the intention of the map that players have to use docks and chop wood. So I still think there is more deforestation necessary before I play that map. They aren't fake. I guess you're asking for tooltip that tells you that you need a merchant to pick it up. Which is just a random bug, that doesn't break the gameplay experience in such a way that it would have to be removed from the "acceptable map" filters. That always happens when there is enough space for a dock but too few space + a dock + a ship passing it. That's the case on about every map that comes with a small river or passages. Might be fixed by making docks not placeable in places where there is only space for a ship. But then players report that as a bug that they can't build a dock there. So eh, not sure what we're supposed to do. Players at least can expect from the size of the ship and the dock that this will be blocking in some areas. For this specific maps the thames can become larger. which I suspect is part of the reason why the "Land" filter is the "Default" one. But all of this feedback is feedback for ship and map development, not for the gamesetup filtering, no? English Channel in particular is one of the better than the worse maps IMO.