SpiderMonkey build changes

[Ykkrosh]

Short version: Please complain if the game no longer compiles properly.

Long version: I've updated the code and build system so that it uses a new version of SpiderMonkey (the JavaScript engine used by Firefox). We used to use version 1.6, from 2006; now it's a version that's about a day old (picked at random from the mozilla-central repository, plus a patch that's not been committed there yet). SpiderMonkey's build system has been redesigned since 1.6, and I've rewritten the way we compile and link it, so it's all somewhat different.

Hopefully things will work automatically. On Windows, there's nothing to do. On Linux, when you first run update-workspaces.sh it will try to build SpiderMonkey and ought to work and then carry on as normal, but errors are possible. OS X theoretically should work like Linux but I haven't tested it at all.

One simplification is that we no longer use a threadsafe version of SpiderMonkey (we do use it in multiple threads but each individual JS runtime is only used in one, and the patch mentioned earlier makes it safe to do that), so there's no need for NSPR when compiling. The other main benefits are that the interpreter is faster, it has a JIT to make things much faster (though it's disabled for now), it has better debugging tools, and it supports new language features (these and these). It'd probably be good to update to a newer version at some point in the future, particularly when the JägerMonkey JIT is integrated and stable, but this should do for now.

[WhiteTreePaladin]

fatal error C1083: Cannot open include file: 'js/jspubtd.h': No such file or directory c:\0ad\source\scriptinterface\ScriptTypes.h

I'm getting a bunch of these errors.

[Ykkrosh]

Have you run update-workspaces.bat again and done a Rebuild in VS? Might be necessary to do that before it'll work. (It looks like it compiled successfully in the autobuilder so I believe I didn't just forget to add all the files, though that's always possible too )

[cgTobi]

Hi all,

I am following the progress of this project for quite a while and finally found the time to check out the sources to give it a try and build it myself.

I just wanted to share my thoughts and findings.

Here is the diff for libraries/spidermonkey-tip/build.sh to get update-workspaces.sh to run fine on my mac.

I hope this will help someone else, so here it is.

--- libraries/spidermonkey-tip/build.sh	(revision 7370)
+++ libraries/spidermonkey-tip/build.sh	(working copy)
@@ -61,6 +61,12 @@
   DLL_EXTN=dll
   LIB_EXTN=lib
   LIB_PREFIX=
+elif [ `uname` = "Darwin" ]
+then
+  INCLUDE_DIR=include-unix
+  DLL_EXTN=dylib
+  LIB_EXTN=dylib
+  LIB_PREFIX=lib
 else
   INCLUDE_DIR=include-unix
   DLL_EXTN=so
@@ -72,8 +78,8 @@
 mkdir -p ${INCLUDE_DIR}/debug/js
 mkdir -p ${INCLUDE_DIR}/release/js
 mkdir -p lib/
-cp -uL src/build-debug/dist/include/* ${INCLUDE_DIR}/debug/js/
-cp -uL src/build-release/dist/include/* ${INCLUDE_DIR}/release/js/
+cp -L src/build-debug/dist/include/* ${INCLUDE_DIR}/debug/js/
+cp -L src/build-release/dist/include/* ${INCLUDE_DIR}/release/js/
 cp -L src/build-debug/dist/lib/${LIB_PREFIX}mozjs-ps-debug.${LIB_EXTN} lib/
 cp -L src/build-release/dist/lib/${LIB_PREFIX}mozjs-ps-release.${LIB_EXTN} lib/
 cp -L src/build-debug/dist/bin/${LIB_PREFIX}mozjs-ps-debug.${DLL_EXTN} ../../binaries/system/

Cheers,

ToBe

[WhiteTreePaladin]

Is there anyway to automate that? Perhaps some script that reminds about updating workspaces when I try to compile? Or do I just need to change my desktop wallpaper with update workspaces in big letters?

[edit]

I suppose I should add here that I got it to compile on vc2005 after updating workspaces. Also, it runs alright.

Edited March 20, 2010 by WhiteTreePaladin

[janwas]

update-workspaces is "only" required when files are added/removed to the project. Since it does take a little while (VS isn't great about reloading solutions), it's best not to run this before every build.

I guess hooking into SVN would be possible for this purpose, but definitely overkill. The most economical solution is to get into the habit of checking for relevant SVN messages (whether a file was added/removed) and/or to try updating the workspace whenever build failures are encountered

If there's a nicer solution, I'd love to know about it.

Philip: build indeed succeeded on Windows here (VC2005)

[fabio]

Hi, is there a way to configure 0ad to link to the system SpiderMonkey library rather than the one in the SVN?

[Ykkrosh]

No.

SpiderMonkey isn't designed around regular stable releases - the versions that have been released are very old, the version that some people are planning to release soon is also very old, and the developers aren't interested in changing that (since they're mainly focused on Firefox; the recommended process for embedding SpiderMonkey is to just pick a recent snapshot and bundle it with your own code, like we do).

The behaviour between versions often changes in incompatible ways, so using a different version to what's in SVN is quite likely to introduce test failures and subtle bugs and network out-of-sync errors. The particular version we're using also has a patch to avoid some thread-safety issues - using a different version means the game will randomly crash or corrupt memory in rare situations, which is not good. So you shouldn't do that

Why do you want it to use the system library?

[fabio]

No.

SpiderMonkey isn't designed around regular stable releases - the versions that have been released are very old, the version that some people are planning to release soon is also very old, and the developers aren't interested in changing that (since they're mainly focused on Firefox; the recommended process for embedding SpiderMonkey is to just pick a recent snapshot and bundle it with your own code, like we do).

The behaviour between versions often changes in incompatible ways, so using a different version to what's in SVN is quite likely to introduce test failures and subtle bugs and network out-of-sync errors. The particular version we're using also has a patch to avoid some thread-safety issues - using a different version means the game will randomly crash or corrupt memory in rare situations, which is not good. So you shouldn't do that

Why do you want it to use the system library?

Packages provided by linux distributions usually prefer to depends on a shared library from another package rather than providing the same library in different packages (e.g. when there is a security problem the distribution only need to update 1 package; also save space on drive).

Debian provides spidermonkey inside xulrunner packages that are already used by other packages. This is what I get on an Ubuntu 10.04:

$ apt-cache rdepends xulrunner-1.9.1
xulrunner-1.9.1
Reverse Depends:
  zekr
  xulrunner-1.9.1-testsuite
  xulrunner-1.9.1-gnome-support
  xul-ext-flashgot
 |xul-ext-flashgot
  xul-ext-firecookie
 |xul-ext-firecookie
  xul-ext-firebug
 |xul-ext-firebug
  videolink
  python-hulahop
 |mozzemberek
  mozilla-nukeimage
  miro
  libgtk2-mozembed-perl
  libgtk-mozembed-ruby1.8
  libgjs0
  libgjs0
  kazehakase-gecko
  galeon
  xulrunner-1.9.1-dev
  xulrunner-1.9.1-dbg
$ apt-cache rdepends xulrunner-1.9.2
xulrunner-1.9.2
Reverse Depends:
  xulrunner-1.9.2-testsuite
  xulrunner-1.9.2-gnome-support
  xiphos
  tuxguitar
  prism
  mozzemberek
  moonlight-plugin-mozilla
  google-gadgets-xul
  gnome-web-photo
  fennec
  eclipse-rcp
  conkeror
  chmsee
  yelp
  xulrunner-1.9.2-dev
  python-gtkmozembed
  libswt-mozilla-gtk-3.5-jni
  couchdb-bin

IIRC some time ago 0ad was able to link to the system spidermonkey.

[Ykkrosh]

Packages provided by linux distributions usually prefer to depends on a shared library from another package rather than providing the same library in different packages (e.g. when there is a security problem the distribution only need to update 1 package; also save space on drive).

Okay - I agree that's usually good, but I'm not convinced it's the best tradeoff in this specific case. In its current state the game is very insecure, and bugs in SpiderMonkey are the least of its worries - it'd be better to focus on reviewing and fixing the game's code first, so it never uses null pointers or overflows buffers etc. The disk space impact is minimal - it's well under 1% of the size of the game's data files. In most cases we have no reason to prefer a specific library version, so we use the standard system packages as much as possible, but I think the drawbacks for SpiderMonkey (the risk of different bugs and features and incompatibilities between versions, given that the game relies heavily on its precise behaviour and performance, and the time wasted on debugging those problems) outweigh the benefits now.

Using xulrunner sounds like a better idea than using the standalone JS releases, since it's much more up-to-date, but it's got sub-optimal build settings (we want non-JS_THREADSAFE builds now to avoid complexity and performance overhead) and we'd still have compatibility problems whenever a new major version is released, and it would delay us adopting new or experimental SpiderMonkey features (e.g. the new JIT).

IIRC some time ago 0ad was able to link to the system spidermonkey.

I consider that to be a bug

[fabio]

OK, thanks for the response.

In this case the page:

http://trac.wildfiregames.com/wiki/BuildInstructions

for Fedora should be upfated to remove the packages:

js js-devel

EDIT: and maybe also *nspr* for all the OS?

Edited March 26, 2010 by fabio

[Ykkrosh]

Thanks, I forgot about updating that - removed the js/js-devel and all references to NSPR now.