Leaderboard

Introduction This article describes different aspects of the scripting integration in our game engine. It starts explaining the basics like the difference between a scripting language and a native programming language which should be comprehensible for everyone with a basic knowledge of computers. In the last paragraph about the ongoing Spidermonkey upgrade, the article goes into more detail and is targeted more at programmers and scripters who are already a bit more familiar with the topic. What is scripting? Basically scripting uses another layer between the operating system (Windows, Mac OS X, Linux) and the code being executed. This layer is a program that reads the script code and executes the commands and is called "interpreter", "script engine" or "script runtime". You don't have to compile scripts before they can be executed by an interpreter. ScriptsNative code Native code on the other hand needs to be compiled before it can be executed. Compiling generates binary machine code that is specific to an operating system and even specific to a certain type of processor. For example you can't run the machine code generated for Linux or Mac OS X on a Windows system, but script code is the same on all three platforms and can run on all of them if there's an interpreter program available. An internet browser like Firefox contains a Javascript interpreter for example. You can display websites containing Javascript on all three operating systems, because Firefox contains the interpreter for those scripts. If you copy the .exe file of Firefox from Windows to Linux you won't be able to run it because it's binary machine code created specifically for Windows systems. Why scripting? With scripting languages you generally have less control over the low level parts of the program like memory access. First that might sound bad, but it also avoids a lot of potential errors the programmer could make. Bugs in code that accesses the memory directly usually can cause crashes or buffer overflows which are very severe problems and often also security issues. Interpreted scripts have an additional level of security because the scripts can only do what the interpreter allows them. Usually if scripts contain errors, an error message is printed to the screen and something in the application might not work as expected. It will be much less likely that an attacker can compromise the user's system if he finds an error in a script compared to finding an error in native code. From the security point of view, it would be very bad if not completely insane to download and execute untrusted native code. In addition to that it would also be quite difficult because the operating system and processor architecture need to match. Some scripting languages on the other hand are designed for that. Javascript can be used for websites and every time you access an unknown website containing Javascript, you execute untrusted script code in your browser. So let's sum it up. Advantages of scriptingMore or less "untrusted" code can be executed with little security concerns. In our case this can be used for automatic downloads of random map scripts, user defined mission scripts, user defined AIs or even GUI mods.Less low-level functionality makes scripts less vulnerable to security issues.Scripts generally allow faster iteration in the development process.Scripts are easier for modders because they only need a text editor instead of a lot of development toolsIn our case we attracted some developers with Javascript knowledge who probably wouldn't have joined us if we only used C++. This is probably different for other projects and not a general advantage of scripting though.Disadvantages of scripting The additional layer added by the interpreter and other characteristics of scripts make them run slower than native code.Maintaining a script interface and including a script interpreter in an application adds additional complexity and takes time.Where do we use scripting? Basically the idea is to use scripting wherever it's possible and makes sense performance wise. It doesn't really matter if it takes a few nanoseconds more until the game decides which piece of music to put next into the playlist. It can even do that before the previous piece of music stops playing, so it won't make a difference at all. On the other hand, it makes a difference for the player if pathfinding of hundreds of units on a map with a million of terrain tiles (little squares of terrain) runs in native code or in a slower script. If you order a unit to move somewhere and it takes half a second until it reacts, that's bad. The difficulty here is to find the right balance and to design sensible interfaces between native code an scripts. GUI (Graphical User Interface) Basically the whole GUI can be modified by changing XML files and Javascript scripts that are linked to actions in the GUI. The GUI system itself is written in C++ code, but XML files and scripts define how graphics are arranged and what happens when you click on a button. The Javascript code behind buttons might call native C++ function that are exposed to scripts though. Random map scripts Random map scripts are written in Javascript which makes them easy to share because they are meant to be shared and will probably even be downloaded automatically from other players in the future. Random map scripts aren't very time critical. The player shouldn't wait five minutes until a game starts, but it doesn't matter if it takes a few seconds longer. There was an idea recently that changed our view about that a bit though. It would be nice to generate previews of random maps before starting the game. This changes the situation a bit because the player might want to generate multiple maps before he's happy with the result. Now there are ongoing discussions about providing some commonly used functionality through native functions exposed to scripts. AI (Artifical intelligence) The AI is where using scripts is currently most controversial. The AI does a lot of calculations and works with a lot of data which runs too slow with the current scripts. Some people want to switch it to native C++ code completely, but that alone won't solve the problem completely. The better approach is probably switching the most performance critical AI functions to C++ and thinking about how they could be optimized at the same time. The less performance critical code can stay in the scripts. Of course this description of the solution is simplified and there are a lot of other aspects to consider which makes it more difficult than it might sound first. Gameplay logic Gameplay logic includes everything that defines how the game plays like that there are units and these units have proerties such as "health" or "walk speed". Other examples of gameplay logic are technology research, Battle dection (for playing battle music), the implementation of trade, or how building works (using more builders makes building faster, only some units are builders etc.). We use a system called the "simulation" for implementing gameplay logic and associated functionality. The simulation consists of components ("Health" or "Builder" for example). Components can be written either in C++ or in Javascript. In this area we are quite flexible to switch between a scripted or a native code implementation. Scripting integration Scripting language I've already mentioned that we have chosen Javascript as our scripting language. There were a few other candidates like Lua and I can't really tell why we have favoured Javascript. It's a very feature rich and flexible scripting language that is broadly supported and known by many people. While flexibility is its strength it's probably also its weakness. This flexibility makes it a bit slower and more difficult to be optimized by the scripting engine. It has also some quirks that modders and programmers need to know about in order to avoid trouble. In practice it's hard to compare performance to other scrip languages because benchmarks usually differ a lot from the code that is actually used in real world applications. Scripting engine There are multiple different scripting engines available for Javascript. These are all designed for maximum performance and there's an ongoing competition between companies like Apple, Google, Microsoft and Mozilla about who has the fastest web browser and the fastest Javascript engine. These engines go way beyond running Javascript in a script interpreter as described earlier in this article. They gather information about the script code and optimize it while it's running. They even create platform specific assembly code, which is quite similar to what a C++ compiler generates. This is a very complex task and all these companies put a lot of man-years and money into their scripting engines. Those engines we could use are mainly Google's V8 or Mozilla's Spidermonkey. Other engines are not compatible with our licenses, only work on a specific platform or aren't available as standalone libraries. We have chosen Spidermonkey years back and it would be a lot of work to change to another library today if we wanted that for some reason. Here the issue with benchmarks is basically the same. There are benchmarks available but we had to learn the hard way that those benchmarks don't really reflect the performance in our game and we have to conclude that there's no easy way of telling if V8 would be faster or slower than Spidermonkey in our case. More on that later. Spidermonkey upgrade Javascript is based on the ECMA standard, but the API (Application programming interface) isn't standardized accross different scripting engines and not even from one version of a scripting engine to the other. This means that changing to another script engine or updating to newer versions is a very time consuming task. We are currently using Spidermonkey v1.8.5 which is the version used in Firefox 4. We have seen benchmarks showing major improvements since Firefox 4 and red a lot of technical articles describing new features solely designed to improve this performance even further. There was the addition of type inference in Firefox 9, the new JIT compiler called Ion Monkey was added in Firefox 18 and a new baseline compiler was added in Firefox 23. We somehow expected that these benchmarking results would be a bit exagerated and that the results would be less spectacular in real world applications like 0 A.D.. Still I was very disappointed after spending dozens of hours adapting to the new version just to notice that the performance in our case is even below the level of version 1.8.5. I've spent even more time and got help from a Mozilla developer but even though we were able to fix some performance related issues in Spidermonkey, we haven't yet managed to get back to the performance before the upgrade. It might be possible that the big changes made recently in Spidermonkey still need some more time until they are polished enough to improve the performance for normal applications and scripts which are not as well tested by the Mozilla developers as the benchmarks. On the other hand we have measured quite a lot of different code parts which more or less all showed the same result as v1.8.5, so I don't expect a huge speedup anyway. The following graph shows the duration of simulation turns in milliseconds. A simulation turn happens all 200ms by default and updates all the simulation components and does AI calculations. Basically it shows the performance as the game progresses going from the left side of the graph to the right. Lower values are better. Check the detailed description below the graph for details! Y-Axis: Duration in milliseconds X-Axis: Turn number divided by 20 This was measured using a non-visual replay which only runs the simulation code and executes AI calculations. It's described in our wiki if you want to do such measurements yourself. Rendering overhead or network performance is not involved because no visuals are displayed and everything runs locally. The game here was a 2vs2 AI game with Aegis bots on the map Oasis 4. You can use this command to watch the AIs fight in the setup that was used for the measurements (but the real measurements were done using the non-visual replay). ./pyrogenesis -quickstart -autostart="Oasis 04" -autostart-ai=1:aegis -autostart-ai=2:aegis -autostart-ai=3:aegis -autostart-ai=4:aegisNot everything on this graph runs in JS. There's the pathfinding and Simulation components that are written in C++, but it runs on the same version of our code and the only difference between the two graphs is the Spidermonkey version. Measuring is a difficult topic and also this measurement has a factor that makes it not 100% accurate. The new Spidermonkey changes the iteration order of some types of loops under some circumstances which causes the AI to behave slightly different than with v1.8.5. For example if it looks for an idle worker, it might pick another one and send him/her to gather wood. There seems to be no easy way to change this back to the old behavior for the measurements without affecting the performance negatively. You notice four things when looking at that graph, The longer the game runs, the slower it gets. This is mainly due to more units being produced which means it takes longer to filter the units by specific criteria, more units move and need to find paths to somewhere, more units are involved in range calculations etc. It's definitely too slow. Later the simulation update takes close to or even more than 200ms which means it will run in each frame. If something that takes more than 200ms runs in each frame, you won't get more than 5 FPS (1000[ms] / 200[ms] = 5 [FPS]). As I already said, the new Spidermonkey is slower. Somewhere after 6000 turns (300 * 20) the performance drastically reduces with the new Spidermonkey. I still have to figure out why this happens. Until now I've looked more at the peaks that happen between turn 0 and 6000. One positive exception regarding performance is the random map loading performance which is now much faster with the new Spidermonkey for those configurations I tested! The exact command used for this measurement was (and I added some code to print the duration): ./pyrogenesis -autostart=alpine_lakes -autostart-random=123 -autostart-size=256 Conclusion regarding performance The conclusion for 0 A.D. is, that we have to redesign our current AI interface a bit. Long running performance intensive tasks need to run in native C++ code because only this way we have the fine grained control we need for optimization and only this way we get the performance we need. The challenge we face will be how the commonly used performance intensive tasks can be separated and moved to a C++ API without loosing the flexibility of scripting in our AI. Even though the performance aspect of the upgrade was a failure, it allowed us to see how far Javascript can get performance-wise and we don't have the expection anymore that it could suddenly run twice as fast with a new version of the script engine. We hope that the competition for better bench-marketing results between the major browser manufacturers will stop better sooner than later. For us a stable API would be much better than some new features aimed for performance improvements. Why the update is still required: We know that Javascript performance most likely won't improve a lot and can design our code accordingly.The Javascript library needs to stay up to date for security reason. This is not a real issue at the moment, but will become more important once we release the first stable version of 0 A.D..Linux distributions don't like to bundle thirth-party libraries with applications and they won't provide Spidermonkey v1.8.5 packages forever.It's important to know about several multi-threading related aspects that changed since v1.8.5 in order to design multi-threading sensibly for the parts that use scripting.The new version fixes bugs, and adds new features.A cleanup of our scripting related code was necessary. We used way to many different approaches which makes adapting to API changes harder, results in duplicated code and makes bugs more likely to occur. These issues were mostly grown historically.The new Spidermonkey is more strict about Javascript syntax and prints more warnings or errors that point developers or modders to potentially buggy code. We have already discovered some issues due to the new warnings.If we need any features or bug fixes in Spidermonkey, we need to have an updated version because Mozilla won't do any changes on v1.8.5.The next steps There's a work in progress patch for integrating the current Spidermonkey 27 development version into 0 A.D. This patch still needs a lot of work before it's ready and I'm going to commit independent parts of it step by step as I already did with some patches. I'm not yet sure which version we will exactly use. We should aim for an "ESR" release which is supported longer and for which a standalone library will be packaged by most Linux distributions. The problem ist that the next ESR version will be released somewhere around July 2014 which is too late. On the other hand it would be a waste to revert the API adaptions from ESR24 to v27 that are already included in the current patch. Links / Resources Trac ticket for the Spidermonkey upgrade with a list of related tickets: #1886Forum thread: Spidermonkey upgradeBugzilla meta-bug for performance fixes in Spidermonkey: Bug 897962

i explain it: we should put as rome three ages: * Early Germania, we put Cheruschi, the tribe of Arminius, and this tribe was after arminius victory a regional superpower, during 22 years, and after a big power on germania. * Medium Germania, we put the Marcomanni, this tribe was a true germanic empire!, duruing the marcomannian wars was a true challenge, here we don't use spearmen, we use swordsmen! * Late Germania, we put Goths this tribes were a true superpower!

I don't like the wooden rails and the rounded merlons... But the general layout and direction is sound. Thoughts, Enrique? I don't like the wooden rails because we are likely to not see them in any other building and their design is kind of odd. Though, the rounded merlons don't look bad here: I think they are newer in style (Middle Ages) than the rectangular and triangular merlons of antiquity.

I agree Ykkrosh, it's tough working on large scale changes on such a large project - it often looks like operating cancer, really. I'd say my focus would be on reviewing and bugfixing megapatch so that all the problems get fixed. And in the future, we should definitely avoid megapatch situations, since they become increasingly difficult to maintain the longer they remain off the main svn trunk.

Sure, just mentioning it, so I hope we don't forget

Maybe we should wait until their translations are included in the game before crediting them? I would imagine that is how most projects handle it.

This is why most people are misinformed. I will stop posting now. In the 1970s the continuity model was taken to an extreme, popularized by Colin Burgess in his book The Age of Stonehenge which theorized that Celtic culture in Great Britain "emerged" rather than resulted from invasion and that the Celts were not invading aliens, but the descendants of the people of Stonehenge. The existence of Celtic language elsewhere in Europe, however, and the dating of the Proto-Celtic culture and language to the Bronze Age, makes the most extreme claims of continuity impossible New World Enclclopedia: http://www.newworldencyclopedia.org/entry/Celts

sry...I can't always get my ideas through...I have social anxiety disorder and ocd. This would be a general indicator of when they were there. "The earliest Gallo-Belgic coins that have been found in Britain date to before 100 BC, perhaps as early as 150 BC." wikipedia Julius Caesar first landed in Britain on August 26th, 55 BC, but it was almost another hundred years before the Romans actually conquered Britain in AD 43. source:http://www.historytoday.com/richard-cavendish/julius-caesar%E2%80%99s-first-landing-britain Caesar's Gallic Wars essays chronicle the history of his military engagements during the years 58-51 B.C. in Gaul, Germany, and Britain. source:http://www.cliffsnotes.com/literature/g/gallic-wars/book-summary

The obvious fix would be to make animals "dumber," meaning they only run away when struck, and then only a small distance and don't continue to run away without being struck. Basically, they wait around to be hit again after running a small distance. Makes sense.

i'd recommend a fourth German faction to be included: the Anglo-Saxons also, i'm a tad confused: is the 1-500ad game going to be a completely separate sequel, or will it be a sequel expansion to teh first one? if nothing else, it would make for interesting scenario design, with the 500-1bc Athenians being used specifically for a map set in Athens during the Byzantine period

Another random map. Somehow similar to "Gold Rush" random maps in AoE-2. Players start in mountains around the remains of a dried lake full of stone and metal mines. Link to Download And of course an image: