Premake overview

[janwas]

Here's a quick tour of our Premake build system (for use by lyfestile and anyone else interested). We're mostly reading backwards in premake.lua:

First, the solution is split into several projects (rationale: VC2005 is only able to build concurrently on the project level, so it helps to split the engine into static libraries. Static is easier than dynamic (no dllimport/export annotation needed).)

Mostly, it's the Atlas editor stuff (which consists of multiple standalone executables, hence projects, though the very long term plan is to integrate them all into the main app), a self-test, those static libs, then the main game executable.

Duplication is avoided by moving the project (which Premake calls package) setup stuff into functions like setup_atlas_package.

The main EXE setup requires a bit of platform-specific junk (linker settings, extra dependencies), and there is a bit of magic concerning external libraries. The lowlevel static library (OS-dependent stuff) brings in dependencies like dbghelp / OpenGL etc, but they aren't linked into the static library and must instead also be injected into the main EXE project, hence the used_extern_libs table.

setup_static_lib_package is where those static libs (major modules of the game, like simulation and graphics) are created. Basically it receives a list of directories, which a helper function traverses and searches for all source files. extra_params is a general idiom for passing flags/special-case stuff like disabling PCH where not desired.

The package_* helper functions will need to be modified, since that seems to be the main focus of changes from Premake3 to 4. Fortunately, most of the changes should be contained there.

At the beginning of the file, we have code to detect OS and options (of chief interest is probably adding Atlas to the solution via --atlas, because it requires wxWidgets to build and that's not included in SVN by default)

Perhaps the trickiest part is the handling of external libraries. Since they all differ in terms of names/paths etc., but we don't want to write separate add-with-premake-syntax for each of them, we have some logic in extern_libs.lua that handles all the cases that came up, and is driven by a big table of external libraries and their respective properties. In the simplest case (something like an 'average' or ideal layout of a library and its include paths / debug suffix etc.), very few table entries are needed. It gets slightly longer when libraries like ffmpeg include several DLLs, or the debug/release versions are identical as in OpenAL, etc.

wxWidgets is probably the worst, its layout is unlike most other libraries, so we pretty much special-case it by calling add_func instead of using the other data-driven stuff in the table.

Some other miscellany:

- when adding source files, premake creates folders within the project for each component in the path, which means that the files are very deeply nested in VC. To avoid this, we added a premake extension trimprefix. If this isn't yet possible with Premake4, we might want to add that - and get the patch upstream.

- our other premake customizations involved assembling .asm files via custom build step and precompiled headers; both are now supported by Premake4, but require different syntax.

One final point: while we're updating the build system, we were hoping to implement Unity builds. In a nutshell, that involves new C++ files that just #include other source files, which can be compiled much more quickly because the compiler needs less IO. We discussed this and possible implementations last week; see http://zaynar.co.uk/0ad/0ad.log (search for unity build).

Whew, all done! Hope the overview helps. It's definitely nice to have a capable build system that provides a good infrastructure for the project - premake has served us well so far, and I hope things can be improved still further

[Ykkrosh]

Sounds good to me

our other premake customizations

There's also some customisations for running CxxTest as a custom build step, which might be doable with more generic functionality in new Premakes. There's also various tweaks to compiler options, Intel C++ Compiler support on Linux, linker command line orders, and others I forget, which are required for certain build environments (but not all, so it's hard to test). Some of these might already be fixed upstream, others ought to be fixed upstream, others might stay local patches. In any case, it'll be important to compare the generated Makefiles in various modes (Linux, Linux+ICC, OS X) with the old ones to make sure we don't introduce any unintentional changes.

Unity builds

Simon (olsner) has been working on this a bit ("now everything compiles! but it only almost links...") so it'll be worth checking with him before doing any work on this.

see http://zaynar.co.uk/0ad/0ad.log

That requires an SVN account so most people can't see it. (It's not secret (anyone can join the IRC channel) but I don't really want it publicly indexed.)

[lyfestile]

The link to the discussion log you posted requires a login.

One of the engines I worked with used Unity files, 0ad has a large codebase, so a good way to approach it may be to group up the .cpp files based on the system they belong to. Instead of having one giant unity file, you would have a handful, this helps keep things a bit more organized.

Edited February 16, 2010 by lyfestile

[Jeru]

I recommend the relevant part should be copy-pasted... Hard to find in that long transcript anyway.

[janwas]

One of the engines I worked with used Unity files, 0ad has a large codebase, so a good way to approach it may be to group up the .cpp files based on the system they belong to. Instead of having one giant unity file, you would have a handful, this helps keep things a bit more organized.

Yeah, sounds good, I think this is what Simon/olsner was doing.

I recommend the relevant part should be copy-pasted... Hard to find in that long transcript anyway.

Righto, copy-pasta coming up! Gentlemen, start your forks:

17:43 < olsner> someone<tm> should implement that "unity build" (or whatever someone called it) system for 0ad
17:43 <@Philip-> I think I prefer incremental builds that aren't as slow as full rebuilds
17:45 < olsner> I think you can bundle up quite a bit of code until you reach the point where your own code is larger than the amount of headers
17:45 <@Philip-> The headers are all precompiled :-)
17:46 <@Philip-> ...I suppose they could be precompiled in a unity build too
17:47 <@Philip-> but incremental builds currently take a fraction of a second per file, and I can't imagine that compiling n concatenated source files wouldn't be significantly slower for large values of n
17:48 < olsner> unless the time is spent doing complicated stuff like template instantiation, I think we should be able to speed the build up by 10x or something, based on the number of lines of code 
17:49 < janwas> olsner: unity builds are a great idea, we've been wanting to do that for a while.
17:50 < janwas> Philip-: i think it's common practice to enable incremental unity builds by disabling the unity step for whatever subproject you're working on
17:50 < janwas> that way, you'd get faster full builds while still allowing incremental
17:50 < olsner> but yes, there's a tradeoff in the size of compilation units... I think once each unit takes more than half a second you're starting to slow down incremental builds
17:50 < olsner> but until then it'll just be faster the more you concatenate
17:51 <@Philip-> How much code can you have before compilers run out of RAM?
17:52 < olsner> hmm, I have some data on that, but it's not readily available right now
17:53 <@Philip-> If I remember correctly, the source files that compile the slowest are the old simulation entity ones that have lots of headers full of lots of giant templates
17:53 < olsner> for some category of files, 700 files was too much (they required about 2GB of ram for gcc, 5 minutes to compile with optimization)
17:53 <@Philip-> but hopefully they'll be obsoleted and removed soon anyway
17:54 <@Philip-> Link times are the biggest pain in incremental builds now, and gold seemed to help with that
17:54 < janwas> hm, CMake seems to make unity builds fairly simple (http://cheind.wordpress.com/2009/12/10/reducing-compilation-time-unity-builds/)
17:54 < janwas> Philip-: "gold"?
17:54 < olsner> unity builds improve link times too 
17:55 < olsner> less debug info and less I/O overall
17:55 <@Philip-> janwas: [url="http://en.wikipedia.org/wiki/Gold_(linker)"]http://en.wikipedia.org/wiki/Gold_(linker)[/url] etc
17:58 < janwas> Philip-: ooh, hadn't heard of gold. nice!
18:01  * Philip- imagines it shouldn't be that hard to add unity builds in Premake - when the script collects the list of source files, just write a single new file that #includes them all and put that into the project instead of the individual ones
18:02 < olsner> yeah, shouldn't be too hard... one unit per subproject
18:02 < olsner> and then the fun work of making the code actually compile and work in that configuration 
18:02 < janwas> sure, it's doable. probably less unpleasant than in cmake because we have an actual language (not some cobbled together crap) behind it, even if i am not really fluent in lua
18:03 <@Philip-> Hmm, it wouldn't generate useful VS project files though
18:03 < janwas> olsner: oh yeah  i bet there will be tons of fun because we haven't namespaced each cpp file etc. do you have any experience with that, how painful it is?
18:04 <@Philip-> Are the only likely problems with static / anonymous-namespace functions that have conflicting names?
18:04 < olsner> janwas: basically, don't overdo per-file macros, undef if you do define local macros, either put static functions in headers or give them unique names
18:04 < olsner> "good" code shouldn't be doing any of that anyway, so it's easy to motivate the required refactoring
18:05 < olsner> it may be useful to provide some mechanism for compiling "difficult" files separately
18:06 < olsner> so that you can postpone fixing them for a little while
18:06 < janwas> i agree it'd be good to fix things eventually, am just wondering if breakage is rampant in practice
18:06 < janwas> AFAIK we usually #undef macros and static function names oughtnt collide that often
18:09 < janwas> ok, doesn't look like we can think of excessive breakage  but what about the project issue that philip raises? would we create another new project for the unity builds and have the actual source files reside in another project whose build is a nop?
18:10 < janwas> ISTR that adding files (for browsing convenience in the IDE) also causes those files to be built
18:10 < olsner> if we move the functionality into premake proper, we can add all the individual units but set "exclude from build" to true
18:11 < janwas> i suppose we could add all source files and set a custom (no-op) build step, and then add the unity files
18:12 < janwas> olsner: right.. that'd work
18:12 < janwas> (but then again, it's preferable to avoid premake customization, especially since it does seem to be under active development  )
18:15 < janwas> anyway, it seems doable and i've made a note to try that too whenever i get around to making our build script compatible with premake4 
18:32 < olsner> at least hte network/ subproject compiled out-of-the box as a single concatenated unit, not so lucky for simulation2, MESSAGE and INTERFACE get redefined
18:32 <@Philip-> That should just need a handful of #undefs
18:32 <@Philip-> in the files that include TypeList.h
18:38 < olsner> typical error from unity builds: missing, misspelled or non-unique header guards 
18:42 < janwas> olsner: ooh, interesting. i've made an attempt to be consistent in source/lib, using INCLUDED_PATH_BASENAME (the rationale was allowing external header guards in the cpp file). probably not much of the codebase bothers, though
18:42  * Philip- does INCLUDED_BASENAME except in one case where BASENAME conflicted with a header in a different project
18:42 <@Philip-> (and expect in any cases of bugs)
18:44 < janwas> ah, so basenames haven't conflicted much? at work, it happened often enough that i now prefer to add at least projectName, if not path
18:45 <@Philip-> I think you use shorter filenames than I tend to use :-)
18:45 <@Philip-> (The conflict I had was Entity.h)
18:46 < janwas> olsner: crikey, i see we often don't even have include guards at all  probably going to need a script to fix things
18:47 < olsner> ProfileViewer.cpp:40: error: conflicting declaration ‘int g_xres’
18:47 < olsner> Interact.cpp:61: error: ‘g_xres’ has a previous declaration as ‘float g_xres’
18:49 <@Philip-> Neither of those files actually use g_xres, it seems
18:50 < olsner> looks like it'll require a load of really trivial fixes, and a few non-trivial ones, like how Errors.cpp redefines a bunch of classes that are also defined in various headers all over the place (where the ERROR_GROUP and ERROR_TYPE macros are invoked)
18:50 < janwas> olsner: wow, wonder how that happened. int is definitely the correct one
18:50 <@Philip-> nor do the two uses in simulation/
18:52  * Philip- isn't sure how to do the Errors.cpp thing differently
18:53 < olsner> maybe by having Errors.cpp include all the headers instead of redefining the classes
18:53 < olsner> the perl script does know which headers contain error definitions
18:54 <@Philip-> Hmm, perhaps
18:54  * Philip- doesn't dislike the error system enough to want to redesign it entirely
20:34 < olsner> so I played with unity builds this saturday and/or sunday ... now everything compiles! but it only almost links...
20:35 < olsner> it's these network message macros I built 5-6 years ago that aren't properly generated when they have already been included in header form in a different source file
21:24 <@Philip-> Sounds good
21:24 <@Philip-> except for the part that doesn't work
21:25 < olsner> one way around is to compile "difficult" files outside the unity based on either hardcoded exclusions or something more sofisticated
21:25 < olsner> ... but just until a better solution to fix the source can be made
21:25 <@Philip-> Is there no way to fix the network macros that isn't really complex/tedious?
21:28 < olsner> it should be trivial really, just need generate declarations and definitions separately (which the code already does), but make sure that generating the declarations doesn't magically break the other stuff
21:29 < olsner> debugging macros just isn't entirely easy
21:35  * Philip- ought to try to look at the network code some time and see what how it works, because he'll need to integrate it with the new simulation system eventually

[olsner]

Attached is a patch that along with my recently committed header guards and misc fixes makes unity builds work as well as normal builds. The reason for not just committing it is that some stuff had to be hacked around a bit so I'd like to get these non-obvious fixes reviewed before simply committing them... Oh, and this is untested in Visual Studio, btw.

"Documentation"

The patch adds an option to the premake script (pass --unity to update-workspaces to enable), and when enabled it generates a source file per library/sub-project according to the existing structure - it simply takes the list of sources for each project and generates a foo_unity_unit.cpp file that #includes all the other files, then registers that single source file as the only source of the project as far as the generated makefile/VS project is concerned. This means that no changes in premake were required, only a few isolated changes in the lua scripts. (Which is nice when we plan to upgrade premake...)

A couple of hacks were involved to make this work (beyond the actual code fixes):

• LOG_CATEGORY is defined by many source files, so it is automatically undefed before including a source. I contemplated adding a #undef to each source file, but rejected that idea based on the number of source files involved
  
• In the lib code there's a macro for generating "unique id:s" based on line numbers. This causes a bit of problems when many similar source files include about as many headers and then define a couple of error code associations just after those headers. Anyway, the generated unity unit redefines the macro to give different unique id:s before each file.
  
• Errors.cpp is currently ill-suited for compilation along with any code that includes headers that declare error types. A better patch should update the generation of Errors.cpp to work well in unity-builds.
  
• Oh, and actually only .cpp files are included in the unity unit. All other kinds of files (e.g. assembly files) get added to the source file list as usual.
  

One problem is that the "engine" build seems to be a bit too big (gcc uses a lot of memory for it), so my parallel builds don't work as well on the unity build. I had expected to be able to run at least one gcc per core but I quickly get I/O-limited rather than cpu-limited. I'm guessing from linking the objects into .a's in parallel (which seems hard to teach make not to do - it doesn't categorize actions), or simply that the total memory use of the gcc instances cause swapping.

I'm also getting mahaf link errors from acpi.cpp - but the code looks like it couldn't ever work on linux so I'm thinking it can't be caused by my changes but rather something that is actually an error in the source - especially since I also get the same error in unity builds and non-unity builds.

As for build time results: with -j1 I get a build time of 1m10s, -j2 gives 37s and -j4 25s (4 core machine). On the same machine, a non-unity build with -j4 takes around 1m15s, so I guess that means a theoretical speedup of around 4x, although the actual speedup for me was only about 2x.

unity.patch.txt

[Ykkrosh]

LOG_CATEGORY is defined by many source files

The rough plan (#247) was to get rid of all that category stuff, since I've never ever needed that functionality and it's a waste of time. (I thought it was a good idea when adding it...) That was partly done, but all the old log statements throughout the code need to be fixed (hopefully just a simple regexp replace to make it use LOGERROR("msg") etc) and the LOG_CATEGORYs removed. If anyone wants to make a patch for this, please feel free

One problem is that the "engine" build seems to be a bit too big

(As mentioned on IRC) This should be improved at some point by removing the old simulation code (which is currently all in "engine"; the new code is in "simulation2" instead), so there's probably no need to try splitting engine up before then.

I'm also getting mahaf link errors from acpi.cpp

I don't think I've ever seen errors like that (on Linux). What's the error? (Mahaf is Windows-only and it looks like it's getting used in acpi.cpp even on non-Windows, which is probably not good, but it hasn't caused me any problems.)

[janwas]

It's good to see some numbers; what a great speedup, thanks for doing this!

(Mahaf is Windows-only and it looks like it's getting used in acpi.cpp even on non-Windows, which is probably not good, but it hasn't caused me any problems.)

I may be able to look at this later today..