Improve build system

[Ykkrosh]

I'm copying this from Trac #257 since it seems like a topic that needs discussion, and the forum is a more convenient medium for discussions:

It's great to see you guys have open-sourced this project; however, the build system for Linux and OS X seems a little bit insane right now. I'd like to offer some assistance in porting the build system to SCons, a Python-based build system that is much simpler to work with and won't require the use of Premake or workspaces. I think making Makefiles is generally a pretty frightful workaround for the inadequacies of Make, and a more modern build system would simplify development of this project a great deal. Using scons, the Premake and workspace stuff would be replaced by a scons-local installation, allowing Unix developers to simply run 'python scons.py' to compile. In doing this, I'd also like to update the Mac build process to:

1) Build proper .app bundles

2) Make use of all the libraries the developer has available, but link statically against any libraries not included by default in OS X. In this way, for example, the developer can compile with FFMpeg from MacPorts?, but not require the user to install MacPorts?. The alternative is to include the libraries in the application bundle, but that would just result in a larger, slower application and increase the amount of work required in the build system (especially since this would involve changing install_names so the loader could locate everything). The only requirement I've found so far that's required that doesn't have static libraries by default in MacPorts? is wxWidgets

I'm still in the process of checking the project out, but I thought it would be worthwhile to gauge your interest in this. There are other good build systems like CMake available as well, but SCons is the one I'm most familiar with and thus the one I think I could contribute the most with. If you guys think this would be a good idea I should be able to start working on it fairly soon (obviously, I don't want to waste a lot of time on it if not). Just let me know and I can get started on this ASAP.

I totally agree that the current build system is hard to use.

Maybe CMake would be a good option? It has a good .deb .rpm ,windows installer, mac package creation support. (CPack) http://www.itk.org/Wiki/CMake:CPackPackageGenerators

[Ykkrosh]

I think the requirements for the build system include:

* Needs to work on Windows, Linux, and OS X.

* Needs to generate Visual Studio (2005, 2008) project files for Windows.

* Needs to be portable to the more common Linux distributions.

* Needs to be efficient - clean builds and incremental builds shouldn't be much slower than the current make system on Linux and the VC++ build system on Windows, since (at least in my opinion) long build times are really painful and must be minimised.

* Needs to handle precompiled headers (for GCC and VC++).

* Needs to be relatively easy to maintain (adding source files, adding new project libraries, adding new external libraries).

* Possibly other things that I can't think of right now.

I'm not familiar with SCons, but it looks like it should be able to handle those things; the only one I'm particularly unsure about is performance, since the only data I know is this article indicating it was awfully slow many years ago. I'm not familiar with CMake either, but it looks like it should handle those things too, but I have no idea about its performance.

There's also the option of extending Premake to handle whatever new features are needed. (We use a slightly patched copy of an old version, and I'd be happy with patching its C code further - I don't think there's much value in trying to maintain a clean copy or tracking new upstream versions, since all we need is a version that works for our one specific project). My natural preference is for that conservative option, since it avoids the risk of switching to a completely new system and finding a load of new problems and wasting time catching up to the state we already had - I think there would have to be really compelling reasons to switch, in terms of providing features that would be impossible or infeasible to implement in our current system. So I suppose an important question is how complex it would be to do .app-bundling and static linking for OS X, just by extending the Lua or C code in Premake. (Unfortunately I have no idea how to answer that question!)

[iksaif]

I think the requirements for the build system include:

* Needs to work on Windows, Linux, and OS X.

* Needs to generate Visual Studio (2005, 2008) project files for Windows.

* Needs to be portable to the more common Linux distributions.

* Needs to be efficient - clean builds and incremental builds shouldn't be much slower than the current make system on Linux and the VC++ build system on Windows, since (at least in my opinion) long build times are really painful and must be minimised.

* Needs to handle precompiled headers (for GCC and VC++).

* Needs to be relatively easy to maintain (adding source files, adding new project libraries, adding new external libraries).

* Possibly other things that I can't think of right now.

I worked with CMake and SCons, and I think CMake is really a good choice.

For example, KDE Desktop wich is a really big C++ project now use CMake.

Performance are here, and it's very easy to maintain.

On thing you may have forgoten, it supports parallel builds (make -j3 for example).

[illushin]

This is my first post in this forum, i only know that you use SVN and release (or at least plan to) on Windows, Linux nad OS X.

I work in a relatively big open source strategy game project, which uses SCons since more then 4 years, and added CMake 2 years ago, to slowly replace SCons. so now we still use both in parallel, but CMake clearly the favourable method. it is much easier and painless to maintain, and has petter performance, plus it can, as you want it, create visual studio projects (and for many other IDEs, eg KDevelop, Code::Blocks, ...), which scons can not.

The only reason why we still use SCons is, that CMake does not support building in parallel under windows, while SCons does. This applies to MinGW, it may work wiht other compilers.

Our project uses GIT as versioning system, and in certain scenarios, SCons allows a smoother working process (less compile time) for devs, if they work with many branches in a single repository. This is the case because SCons's support for caching object files (like ccache).

An other build system you may consider, is Waf, which is kind of a predecessor of SCons:

http://en.wikipedia.org/wiki/Waf

I have no personal expericne with it, but its said to be less a hassle to maintain then SCons.

[Belisarivs]

I work in a relatively big open source strategy game project

Why didn't you say its name? Glest?

[Yggdrasil]

it's probably not glest. they use jam for building on linux. there are no official build files for windows.

battle for wesnoth uses scons and cmake but svn and not git.

spring has also both, scons and cmake, and uses git. so that's my guess.

i think cmake is the way to go if you want to generate project files for different IDEs. i never used scons (except for building). so i'm not sure if this is possible with scons.

[janwas]

Thanks for the suggestions and comments! Having had fairly frequent contact with the build system, I'd like to weigh in.

however, the build system for Linux and OS X seems a little bit insane right now.

Yeah Even the first step, having to manually gather all dependencies, really sucks. I have little dev experience on *nix, but it'd be nice to make things as easy there as they are with our Windows infrastructure.

I think the requirements for the build system include:

Further requirements include being able to handle dirty Windows-specific stuff (.rc files, manifests) and assembling .asm files.

(We use a slightly patched copy of an old version, and I'd be happy with patching its C code further - I don't think there's much value in trying to maintain a clean copy or tracking new upstream versions, since all we need is a version that works for our one specific project).

Actually this part has always made me nervous. Newer versions of Premake may well include helpful or even necessary features since its dev model is "only add stuff when people ask for it" (i.e. it's not equipped to handle all needs out of the box). For example, v4.1 includes support for cross-compiling 32/64 bit and other platforms (e.g. Mac), which may be useful.

However I agree that we could probably get by with the current version for quite a while.

My natural preference is for that conservative option, since it avoids the risk of switching to a completely new system and finding a load of new problems and wasting time catching up to the state we already had - I think there would have to be really compelling reasons to switch, in terms of providing features that would be impossible or infeasible to implement in our current system.

Amen. It's worth mentioning that the build system has 1300 lines of Lua customization plus the logic in Premake proper - replacing that would be a pretty big undertaking.

In light of that, isn't there another option besides makefiles or SCons/CMake? Premake also generates Code::Blocks workspaces - would that help, or is using it a pain? Is there anything specific that could be done to make the generated makefiles less painful?

CMake clearly the favourable method. it is much easier and painless to maintain, and has petter performance, plus it can, as you want it, create visual studio projects (and for many other IDEs, eg KDevelop, Code::Blocks, ...), which scons can not.

Ah, interesting. That's a good argument, creating VC projects is pretty important for Windows devs.

The only reason why we still use SCons is, that CMake does not support building in parallel under windows, while SCons does. This applies to MinGW, it may work wiht other compilers.

hm, is that really a killer feature? What kind of speedup are you seeing from parallel builds?

[Matei]

I definitely get speedups from make -j 3 on my dual core machine, so I think keeping the option for parallel build would be nice. If SCons really doesn't scale to the size of 0AD though, we can drop it. Many developers will have the option of working in Visual C++, which does parallel builds.

[Cilyan]

Hello all !

I worked a little bit on 0AD so it uses as much as possible system libraries and tools. As a first step, there was a very simple tweak to use premake 3.7. I saw you are planning on moving to SCons or CMake but by the mean time, maybe you would be interested by this little patch. I wanted the script to be able to run with premake 3.7 (last version in 3.x series) on my Linux (ArchLinux). Finally, it comes out that it also runs fine with premake 3.1 (the one shipped in svn). The problem was a simple deprecation in lua, the syntax

for i,v in dirs do

is obsolete and should be replaced by

for i,v in pairs(dirs) do

.

I suppose it will run on other 3.x versions of premake and on other plateforms, but I have not conducted more tests. Here is the final patch :

Index: build/premake/premake.lua
===================================================================
--- build/premake/premake.lua	(révision 7027)
+++ build/premake/premake.lua	(copie de travail)
@@ -260,13 +260,13 @@
 	-- (e.g. "lib/precompiled.h")
 	tinsert(package.includepaths, source_root)

-	for i,v in rel_include_dirs do
+	for i,v in pairs(rel_include_dirs) do
 		tinsert(package.includepaths, source_root .. v)
 	end


 	if extra_params["extra_files"] then
-		for i,v in extra_params["extra_files"] do
+		for i,v in pairs(extra_params["extra_files"]) do
 			tinsert(package.files, source_root .. v)
 		end
 	end
@@ -288,7 +288,7 @@
 				pch_dir..source
 			})
 		end
-		for i,v in project.configs do
+		for i,v in pairs(project.configs) do
 			tinsert(package.config[v].defines, "USING_PCH")
 		end
 	end
@@ -489,7 +489,7 @@
 		windows = { "lib/sysdep/os/win", "lib/sysdep/os/win/wposix", "lib/sysdep/os/win/whrt" },
 		macosx = { "lib/sysdep/os/osx", "lib/sysdep/os/unix" },
 	}
-	for i,v in sysdep_dirs[OS] do
+	for i,v in pairs(sysdep_dirs[OS]) do
 		tinsert(source_dirs, v);
 	end

@@ -903,7 +903,7 @@
 	-- desired */tests/* files. this is a bit slow, but hey.

 	local all_files = matchrecursive(root .. "*.h")
-	for i,v in all_files do
+	for i,v in pairs(all_files) do
 		-- header file in subdirectory test
 		if string.sub(v, -2) == ".h" and string.find(v, "/tests/") then
 			-- don't include sysdep tests on the wrong sys
Index: build/premake/functions.lua
===================================================================
--- build/premake/functions.lua	(révision 7027)
+++ build/premake/functions.lua	(copie de travail)
@@ -1,6 +1,6 @@
 function sourcesfromdirs(root, dirs)
 	local res = {}
-	for i,v in dirs do
+	for i,v in pairs(dirs) do
 		local prefix
 		if v == "" then prefix = root..v else prefix = root..v.."/" end
 		local files = matchfiles(
@@ -14,13 +14,13 @@
 end

 function trimrootdir(root, dirs)
-	for i,v in dirs do
+	for i,v in pairs(dirs) do
 		dirs[i] = strsub(v, strlen(root))
 	end
 end

 function listconcat(list, values)
-	for i,v in values do
+	for i,v in pairs(values) do
 		table.insert(list, v)
 	end
 end
Index: build/premake/extern_libs.lua
===================================================================
--- build/premake/extern_libs.lua	(révision 7027)
+++ build/premake/extern_libs.lua	(copie de travail)
@@ -260,7 +260,7 @@
 			tinsert(package.linkoptions, "-framework " .. name)
 		end
 	else
-		for i,name in names do
+		for i,name in pairs(names) do
 			tinsert(package.config["Debug"  ].links, name .. suffix)
 			-- 'Testing' config uses 'Debug' DLLs
 			tinsert(package.config["Testing"].links, name .. suffix)
@@ -277,7 +277,7 @@
 -- extern_libs: table of library names [string]
 function package_add_extern_libs(extern_libs)

-	for i,extern_lib in extern_libs do
+	for i,extern_lib in pairs(extern_libs) do
 		local def = extern_lib_defs[extern_lib]
 		assert(def, "external library " .. extern_lib .. " not defined")

I hope it will help.

[janwas]

I definitely get speedups from make -j 3 on my dual core machine, so I think keeping the option for parallel build would be nice. If SCons really doesn't scale to the size of 0AD though, we can drop it. Many developers will have the option of working in Visual C++, which does parallel builds.

OK When I last tried it, VC didn't manage too big of a speedup (probably due to heavy IO).

The problem was a simple deprecation in lua

Thanks! I've gone ahead and committed the changes.

(there was one more "for i,x construct" in extern_libs.lua:244)

[Ykkrosh]

I saw you are planning on moving to SCons or CMake but by the mean time

We're not currently planning that - it's just something that someone suggested, and so I started this forum topic to see if anyone had compelling concrete reasons to switch (but I haven't seen any so far, but I'm happy to wait )

I wanted the script to be able to run with premake 3.7 (last version in 3.x series) on my Linux (ArchLinux).

Be aware that we have some patches in our version of Premake, and I don't think they were all sent/accepted upstream into standard Premake. You might experience some obvious or subtle errors, or missing features, when using the standard version, so it's not really recommended.

[illushin]

Yggdrasil guessed right, i come from spring.

The only reason why we still use SCons is, that CMake does not support building in parallel under windows, while SCons does. This applies to MinGW, it may work wiht other compilers.

hm, is that really a killer feature? What kind of speedup are you seeing from parallel builds?

I personally never use parallel builds, as i like to still be able to work painlessly with my machine (Dual Core), even when compiling. Also, i usually dont work under windows.

There is actually just one of the main devs of spring i know of, which uses SCons, because of this reason.

I definitely get speedups from make -j 3 on my dual core machine, so I think keeping the option for parallel build would be nice. If SCons really doesn't scale to the size of 0AD though, we can drop it. Many developers will have the option of working in Visual C++, which does parallel builds.

i would not say that SCons does not scale well.. it may scale up even better, as it is actually phyton, so you have the flexibility of a fully working programming language. CMake is rather limited in that sence, which is why you end up with more simple/clean build files in the end, and its still powerfull, i would say.

i guess this is no problem for you, but both of them are really tailored towards native sources. it is possible to make them work with others too. eg we did with Java, where SCons ... theoretically has better support, but it is pretty bad in practice for anything more then a hello world example. CMake only supports source file change detection for native sources by default.

[idle]

Hey guys,

Sorry that I didn't read the whole tread...

but _PLEASE_ don't use such exotic build-systems!

If possible take Makefile (self written from scratch) otherwise autotools or CMake.

Nothing is perfect but the others are horrible (maybe not even all but all i know about...).

They don't respect *FLAGS (LDFLAGS/LIBS, -Wl,--as-needed) which is important if you'd like to link just _really needed_ stuff etc.

IMHO there are many other things against such stuff.

And if a build-system doesn't know how to build in a "proper" way.. well then....

I assume you're not going to write Makefiles yourself so I only suggest autotools or CMake then.

I just saw the thread here, its 3am now... maybe I'll point it out a bit more to a later moment.

[janwas]

Sorry that I didn't read the whole tread...

but _PLEASE_ don't use such exotic build-systems!

Reading the thread would have pointed out the reason for a more complicated build system - Makefiles or autotools do not meet the requirements set out above.

but the others are horrible

Does that include Premake and scons?

They don't respect *FLAGS (LDFLAGS/LIBS, -Wl,--as-needed) which is important if you'd like to link just _really needed_ stuff etc.

hm, I don't have much Linux experience, but only linking in the bare necessities doesn't exactly sound like a killer feature. Is that so important?

[idle]

Sorry that I didn't read the whole tread...

but _PLEASE_ don't use such exotic build-systems!

Reading the thread would have pointed out the reason for a more complicated build system - Makefiles or autotools do not meet the requirements set out above.

more complicated sounds ok, as long as it is only complicated

but the others are horrible

Does that include Premake and scons?

at least premake, there were a few things with scons too but i don't remember which things..

They don't respect *FLAGS (LDFLAGS/LIBS, -Wl,--as-needed) which is important if you'd like to link just _really needed_ stuff etc.

hm, I don't have much Linux experience, but only linking in the bare necessities doesn't exactly sound like a killer feature. Is that so important?

yes it is.

for some linker specific stuff i recommend:

http://blog.flameeyes.eu/ (tags: as-needed, linker)

and also interesting is:

http://www.makelinux.net/make3/make3-CHP-2-SECT-8.html

some major issues has been fixed now, see http://trac.wildfiregames.com/changeset/7077

[janwas]

more complicated sounds ok, as long as it is only complicated

heh, true

In fairness I'd say our build script isn't horribly complicated, and it remains readable by virtue of being Lua (not my favorite language, but still way more intelligible than CMake or m4).

I didn't know about --as-needed, thanks for the pointers.