Suggestions: Linux Build System, repo

[daniel.santos]

Hello guys, very cool game! This is my first posting and I'll bet you've heard this suggestion before. First off, I know the common response (certainly from me on my projects) when somebody has a suggestion like this is "Sounds great! When will you have it done? " Unfortunately, I don't have the time to work on 0ad, sorry.

None the less, there are certain traditional mechanisms used in *nix (Linux / UNIX) build systems that are generally followed

1.) Allow your build system to manage your configuration, never use shell scripts.

While it's occasionally hard to avoid custom scripts, the modern build systems (autotools, CMake, etc.) can do almost everything that needs to be done on their own, given proper input files. On almost all projects, I can do my configure & build from a directory other than the one I checked out, leaving it's contents completely unmodified. But the major advantage is that I can do it in a tempfs filesystem -- this means no disk writes are needed for the build and it's much faster. In fact, by the time I build it a second time (say, I've made a change somewhere), all of my source files will be in the Kernel's page cache, so there wont even be any disk reads!

2.) Don't include binaries in your source tree.

Linux users (especially Gentoo-ers) are especially wary of pre-built binaries. While I know that it's often useful and quite handy to have binaries included, it's fairly untraditional. I'm not sure what to propose for this because I'm very new to your system, but some optional mechanism would probably nice. Also, most game projects keep their game data in it's own directory. Often, this is what you'll see:

data
docs
src
(README.txt, configure.ac, CMake.txt, Makefile.in, etc.)

3.) Allow the user to build against their own libraries.

Maybe this goes with the above item. In fact, on my OS (Gentoo Linux) the only packages (ebuilds) that I don't have access to is nvtt and fcollada and I already had all of the remaining libraries installed except for spidermonkey and enet. Plus (back to item 1) standard build system will allow the person configuring to choose where their libraries are located, in case they wanted to link against yours, or even against a special modified version of them. So again, probably better if checking out this tree was optional as well, especially at a whopping 850MB!

4.) Build system should allow the choice between static & dynamic linking.

While operating system kernels now all memmap executables, thereby not using up memory that's unneeded, it's still a nice thing to allow. The area where this is (oddly) begging to become antiquated is with link-time optimizations. MSVC has had this for a little while now and gcc is finally getting them with 4.7 (you can get them in gcc-4.6 if you enable them in gcc's configure, but they are still very experimental). But projects like LLVM again makes this static linking unnecessary for obtaining these optimizations.

Of course, it's even better if you can adhere to one platform's standards using a mechanism or idiom that meshes nicely with both your project's and the standards of other platforms as well.

Anyway, I gotta run so I'll check back later. I know there was a few other items that I've forgotten to mention.

[Ykkrosh]

1.) Allow your build system to manage your configuration, never use shell scripts.

While it's occasionally hard to avoid custom scripts, the modern build systems (autotools, CMake, etc.) can do almost everything that needs to be done on their own, given proper input files.

The main thing we use shell scripts for is to automatically build some third-party libraries that aren't ubiquitous enough to assume the user will already have them - the rest is just done with Premake (which runs some Lua to generate Makefiles). I don't particularly like the current system, but I'm not really sure what would work better (without e.g. entirely replacing Premake with CMake, which I played with a while ago but seemed like probably too much effort to be worthwhile) - do you have any specific suggestions that would likely work well? (I'm happy to implement people's suggestions if they sound worthwhile and they don't want to work on it themselves )

On almost all projects, I can do my configure & build from a directory other than the one I checked out, leaving it's contents completely unmodified. But the major advantage is that I can do it in a tempfs filesystem -- this means no disk writes are needed for the build and it's much faster. In fact, by the time I build it a second time (say, I've made a change somewhere), all of my source files will be in the Kernel's page cache, so there wont even be any disk reads!

With our current system, you could symlink build/workspaces/gcc/ onto a tmpfs and I think that should basically work. Not ideal, though.

2.) Don't include binaries in your source tree.

Linux users (especially Gentoo-ers) are especially wary of pre-built binaries. While I know that it's often useful and quite handy to have binaries included, it's fairly untraditional. I'm not sure what to propose for this because I'm very new to your system, but some optional mechanism would probably nice.

We only have pre-built Windows binaries - the ones in SVN's libraries/*/lib/ are so developers on Windows can compile the game without having to separately download a libraries package, and the ones in binaries/system/ are for non-programmer developers (e.g. artists) on Windows so they can get the latest version of the game without compiling themselves, and also so programmers can use the .pdb files to debug crashes from non-programmers running those pre-built executables. It's all in SVN so that the binaries stay in sync with the code and data files, and people won't have to worry about running special update scripts instead of standard TortoiseSVN. Again, I don't particularly like the setup but I wouldn't want to change it to something that's less convenient for Windows developers, and don't know what would work better

Also, most game projects keep their game data in it's own directory.

We use binaries/data/ for that, so it's still separated but a bit more nested (for historical reasons). Moving binaries/data/ to data/ and moving binaries/system/ to bin/ (or bin/win32/ or something) could work, but I'm not sure what the benefit would be.

3.) Allow the user to build against their own libraries.

Maybe this goes with the above item. In fact, on my OS (Gentoo Linux) the only packages (ebuilds) that I don't have access to is nvtt and fcollada and I already had all of the remaining libraries installed except for spidermonkey and enet.

"./update-workspaces.sh --with-system-enet" uses the non-bundled copy, and there's a patch to add "--with-system-spidermonkey". We use a quite substantially patched version of FCollada so there's no non-bundled support. Non-bundled NVTT could work except the latest release has some bugs and no distro seems to ship a sufficiently patched version, so it's been safer to stick with bundled, but that could be changed.

None of the other things in libraries/ are used on Linux (they're only for Windows).

4.) Build system should allow the choice between static & dynamic linking.

Do you mean linking the different modules of the game, or linking to third-party libraries? For the latter, I thought that just depends on how those libraries are compiled (which isn't our responsibility). For the former, I'm not sure what the point of dynamic linking would be - the only reason we split the code up at all is to allow parallel builds in MSVC on Windows, and it all gets statically linked together at the end since all we want is an executable. So I'm unsure what you're suggesting

[historic_bruno]

We only have pre-built Windows binaries - the ones in SVN's libraries/*/lib/ are so developers on Windows can compile the game without having to separately download a libraries package, and the ones in binaries/system/ are for non-programmer developers (e.g. artists) on Windows so they can get the latest version of the game without compiling themselves, and also so programmers can use the .pdb files to debug crashes from non-programmers running those pre-built executables. It's all in SVN so that the binaries stay in sync with the code and data files, and people won't have to worry about running special update scripts instead of standard TortoiseSVN. Again, I don't particularly like the setup but I wouldn't want to change it to something that's less convenient for Windows developers, and don't know what would work better

I was thinking about this earlier and would like to do the same thing for OS X. As long as we test the prebuilt libraries on different versions of OS X (specifically 10.5-10.7) and confirm they work, it seems very worthwhile to avoid needing the likes of MacPorts. We could possibly even go so far as handling 32- and 64-bit builds, but if not stick with 64-bit.

As far as Windows is concerned, I don't see any problems with our setup, it's very nice to have all the libraries (except wxWidgets) worked out in advance by a developer, because they only rarely change in practice. I would like the OS X build process to become more like the Windows build process, not vice versa.

[infyquest]

I also want to have this patch included in the linux build, http://trac.wildfiregames.com/ticket/990

This one makes you to use the distro's libmozjs185 (js-devel package needed for building)

[arielkempf]

We only have pre-built Windows binaries - the ones in SVN's libraries/*/lib/ are so developers on Windows can compile the game without having to separately download a libraries package, and the ones in binaries/system/ are for non-programmer developers (e.g. artists) on Windows so they can get the latest version of the game without compiling themselves, and also so programmers can use the .pdb files to debug crashes from non-programmers running those pre-built executables. It's all in SVN so that the binaries stay in sync with the code and data files, and people won't have to worry about running special update scripts instead of standard TortoiseSVN. Again, I don't particularly like the setup but I wouldn't want to change it to something that's less convenient for Windows developers, and don't know what would work better

Sorry for reopening this old thread.

You could provide the binaries for Windows and Mac users as a separated download from your main site, so that those users could just unpack it in the root folder and get the same result.

Edited June 4, 2012 by arielkempf