Working On Upgrade To Premake4

[Yves]

Hi

I've spent the last few days converting the existing premake3 lua-scripts to the premake4 syntax. I'll still need some more time to finish and test everything, but it looks like I'm on the right way.

I just wanted to inform the people here, because it would be a waste if someone else worked on that too.

It could still take some weeks or even months because my vacations are over now, but I guess this doesn't hurry.

So... everyone keep it going and hopefully we'll soon be able to take advantage of premake4

EDIT:

I've released of first version of the updated scripts in this post.

Edited June 19, 2011 by Yves

[feneur]

Splendid, will this (help with) supporting Xcode on the Mac in any way? Would be nice to make things easier for people using that platform both for the sake of Mac programmers and other developers using Mac, so if that's the case you can rest assured your work will be even more appreciated

[Kimball]

Excellent news! Mainly for the reasons Erik described.

[Yves]

Won't promise anything, but ...

Premake is a build configuration tool. Describe your C, C++, or C# software project using a simple, easy to read syntax and let Premake generate the project files for:

* Microsoft Visual Studio 2002, 2003, 2005, and 2008, including the Express editions

* GNU Make, including Cygwin and MinGW

* Apple Xcode

* Code::Blocks

* CodeLite

* IC#Code SharpDevelop

* MonoDevelop

[feneur]

Sounds promising at the very least

[janwas]

Thanks for tackling this

I am very busy ATM, but wrote a good deal of the existing scripts and might be able to help out if you get stuck somewhere (or are wondering 'why did they do it this way') - just send an e-mail.

[Yves]

A short status report:

More or less the whole premake-script is converted to permake4 syntax in a "quick and dirty" style.

My approach is getting it working quickly and improving it as soon as everything works. I didn't change most of the concepts used in the original scripts even though some things could probably be solved different (and better) with the new premake4 features.

At the moment I'm working on the test generation. I'm trying to implement it using prebuildcommands, which already works except with parallel builds. Premake has a bug with parallel builds and make (e.g. make -j5). In parallel builds, the compiling is started at the same time the prebuildcommands are run, which results in missing test sourcefiles during the first build-process.

My goal was not to patch the premake sources, but it looks like I'll have to fix this. In my opinion this is a bug in premake and not just a problem of how I'm trying to use it. Prebuildcommands should always be called before the build-process and not during the build-process. Maybe I can get a fix upstream so we won't have to use patched sources in the future.

@janwas

Thank you for offering your support! I guess there will be some things to discuss after I get the first version working.

[Yves]

I just wondered if there's any need for the .d files created by make/gcc.

At first view it looks like they aren't needed. Dependencies are recreated using premake which is invoked by update-workspaces.sh.

The -MF flag for creating dependencies is appended by premake, but the customized premake code used for .asm files tries to create a .d file too. Because of a little typo it doesn't create it properly, so I guess nobody needed it.

What do you think? Should I remove it in the premake4 scripts(if not needed) or should we fix it now(if needed)?

I'm talking about these lines in gnu_cpp.c:

				if (!g_verbose)
					strcat(g_buffer, "@");

				strcat(g_buffer, "nasm "); strcat(g_buffer, opts); 
				strcat(g_buffer, " -i"); strcat(g_buffer,input_dir );
				strcat(g_buffer, " -M -o $@ $< >$(OBJDIR)/$(<F:%%.asm=%.d)\n");

There's a percent-sign too much.

[Ykkrosh]

Hmm, I'm not quite sure what you mean. Premake3 doesn't output .d files by itself, so it has to get GCC to do it via -MF (so that incremental rebuilds work correctly). Do you mean Premake4 is different for C++ and computes all the inter-file dependencies by itself? (I'd be surprised since I don't really see how it's possible to do that correctly, and incremental rebuilds would fail if you added an #include to a file (which is very common) and then rebuilt without running update-workspaces again.)

About .asm files: Good catch! (Though is it one % too much or too few? The line for .c files says "$(<F:%%.c=%%.d)" but I don't know if that's right or wrong). The .asm files are edited very rarely, which is probably why nobody noticed, but it would be nice to do it correctly and generate the .d files if it's not much extra work.

[Yves]

Sorry I got that wrong. This line is the missing piece:

-include $(OBJECTS:%.o=%.d)

I don't have much experience with make and gcc and didn't catch how these files are used.

Now I understand.

About the other thing I'm a bit confused. I've tested the .asm-part with both two and one percent sign.

It only works with one. I don't know why there are two here: "$(<F:%%.c=%%.d)".

EDIT: I guess the double percent sign was made with character escaping in mind, but it's not needed there.

At the moment, there aren't any .c files used in the build-process and the premake-scripts don't even search for *.c.

In my opinion this part can be completely removed or should be corrected with one percent sign instead of two (even if it doesn't matter at the moment).

			else if (matches(ext, ".c"))
				strcat(g_buffer, "$(CC) $(CFLAGS) -MF $(OBJDIR)/$(<F:%%.c=%%.d) -MT $@ -o $@ -c $<\n");

Premake 3.7 does it this way in the original version:

			else if (matches(ext, ".c"))

				strcat(g_buffer, "$(CC) $(CFLAGS) -o \"$@\" -c \"$<\"\n");

Edited January 30, 2011 by Yves

[Yves]

For clarity and convenience I've created a patch of the suggested changes.

The dependencies of objectfiles created by nasm are now written to .d files correctly instead of a file with .asm ending. I didn't test the part with .c files because there are no .c files at the moment, but I'm quite sure double percent signs are wrong there too. The script uses the same functions to write these strings to the makefile.

Feel free to apply this patch... or just wait for me to finish the update to premake4 .

Patch.txt

Edited January 30, 2011 by Yves

[Yves]

I've just executed the first build of 0ad created using premake4 and make.

Unfortunately there was no way around extending premake. It's a bit easier with premake4 than it used to be with premake3 because nearly all the logic is now in lua-scripts instead of c-code and because some features now work out of the box.

I had to customize premake for meeting the following requirements:

- Using the output of pkg-config and sdl-config for library dependency input

- Integrating assembler-code using nasm assembler

- Using cxxtestgen

- Using special linker options to work around circular references (-Xlinker --start-group $(LDDEPS) -Xlinker --end-group $(LIBS))

The next step will be getting visual studio solution/project creation working (at least 2005, 2008 and 2010 I guess) and testing it on a mac (make-projects first, xcode later).

I hope I can get the required hardware and software for testing...

[Ykkrosh]

Sounds like great progress

In the long term, it'd be nice if the changes we need were merged into the standard Premake distribution - the main developer has seemed interested in extending Premake based on real use cases, so hopefully that should be possible. But our main priority is just to have it work for us, and if we have to distribute a customised version of Premake then that's fine for now (and better than waiting months for a new upstream release).

I don't think we really need to keep VS2005 compatibility, but we should support 2008 and 2010. But I think the 2005 format is almost identical to 2008, so if it's easy to support both then that'd be worthwhile.

If you need help with testing, I have access to various environments and other people here can probably help too, so you could upload your code somewhere (attach here or on a Trac ticket or whatever) for people to try out. (Also it'd be nice for us to have a copy of the work-in-progress in case you discover later that you don't have time to complete it, so someone else could finish it off - hopefully that won't happen but it's good to be safe )

[Yves]

It's a bit too much "work in progress" at the moment and there are stupid "TODO" comments everywhere, some in german and some in bad english

I'd prefer to wait with a first release until visual studio is supported.

Something else...

	if OS == "windows" or not options["without-pch"] then
		package.pchheader = header -- "precompiled.h"
		package.pchsource = source -- "precompiled.cpp"
		...

It looks like on windows the option "without-pch" will not disable precompiled headers.. does anyone know the reason for that?

[Ykkrosh]

I expect that's just because there was no good reason to disable PCH on Windows, and supporting it would mean more build configurations to test and fix, so it's easier to consistently require PCH. (GCC PCH can cause problems, at least with old versions or certain external tools, so it was worth supporting without-pch for GCC.)

[Yves]

Today/yesterday I've finished the vs2010 scripts. Vs2005 and vs2008 aren't working yet, because they have a different syntax for their project files.

During the last week, I've been looking a bit closer at premake's development. The good thing is that there's quite fast progress.

On the other hand, it means that the source is changing very frequently and it will be hard to get something upstream.

Porting changes from the current 4.3 stable release to the development release and making them "final" would be a lot of work and requires deep knowledge of premake's internals and the different IDEs and their build-systems.

The main developer has announced that he is trying to make contributions easier, mainly by creating a more unique code-structure... we'll see.

For now I'm NOT focusing on getting changes upstream.

Maybe I can give it a try again after premake4.3 is integrated and premake4.4 is released.

Good night...

[feneur]

Really nice to hear about the progress.

[Pureon]

Great news Yves! Thanks for the update

[janwas]

Good news.

About pushing stuff upstream: one upside of the rapid development is that others might have the same need, and it might get taken care of automagically So waiting sounds good; as Philip says, living with a custom version of Premake is fine for the mid-term at the very least.

As for VC2005 support, I agree it's no longer needed, but IIRC the only difference is the version number at the top of the XML files.

I was recently suggesting to Philip that we only support VC2010 for simplicity (one big upside is avoiding the .manifest hell). Since the hwdetect framework now also reports the compiler used, we'll soon see how many people still need VC2008.

[Yves]

I've been working on xcode support the last weekends, but encountered a lot of difficulties.

It was the first time I've really tried working with MacOS, which didn't make it easier. One of the difficulties was setting up a VM with an acceptable low frequency of kernel panics .

After some kernel patches, manually entered bus-ratio-settings and other ugly stuff, it works more or less, but It's still a pain to use.

Make works, but xcode support is a bit more difficult. There were problems with architecture detection, pgk-config and sdl-config, assembling, linking, bugs in the code that got fixed in newer revisions and more.

Currently I have a working xcode project and I was able to run 0ad. I've made many changes to the project manually and there's still work to be done until these changes are all integrated into premake.

I keep working on it and hopefully there will be more progress in my vacations in the week after next week.

[historic_bruno]

Ah, good thing I didn't start working on this, too

I'm using Visual C++ 2010, and with our current version of premake I have to convert 2008 projects every time a source file is added or removed (requires several steps and 2008 projects seem to be broken anyway). That was enough for me to at least look at the new premake, but as the structure is totally different I didn't pursue it. It's great that someone else has! This will make a huge difference in ease of development on different platforms.

[Yves]

... a detail:

What do you think about changing the shebang of update-workspaces.sh from #!/bin/sh to #!/bin/bash?

I've found out that some of the arch-detection problems could be related to the missing HOSTTYPE variable.

According to the manpage of bash this variable is defined by bash.

On my Ubuntu 10.10 system /bin/sh points to /bin/dash which does not define HOSTTYPE.

Often /bin/sh is a link to bash anyway.

On my Snow Leopard installation both /bin/sh and /bin/bash are the same version of bash.

[Ykkrosh]

If the architecture detection is currently only a problem on OS X (is that the case?), then using /bin/bash won't make a difference (since OS X always uses bash) and there's a danger it might introduce problems on other platforms that currently work, so it'd be safer not to change it unless the change fixes a real problem.

(Using HOSTTYPE is a terrible hack anyway - probably what we ought to do is something like autoconf (maybe actually use autoconf) to detect what the compiler is really targeting.)

[Yves]

Currently this situation only exists on my Ubuntu and other systems where /bin/sh points to a shell that does not define HOSTTYPE.

It isn't really a problem because the check with "gcc -dumpmachine" works on my system, but could be a problem if both checks fail (happens rarely I guess).

There's a new function in Premake 4.4(not stable yet) that could be interesting:

http://industriousone.com/osis64bit

Currently architecture detections is nothing more than choosing between 32 and 64 bit, so if it's reliable we could replace a lot of code with calling os.is64bit() in the future.

I also had a look at "platforms" introduced in premake 4.1 but currently I think that's not needed.

Maybe I'll look at it later when premake 4.4 is released, but probably it's not worth the efforts.

http://industriousone.com/platforms

Edit:

gcc -dumpversion won't work on the mac, but currently it looks like HOSTTYPE-detection will work.

Edited April 8, 2011 by Yves

[Ykkrosh]

It says "The os.is64bit function determines if the host is using a 64-bit processor" which doesn't sound like the right idea - some users run 64-bit kernels on 64-bit processors but with 32-bit userspace, so the game needs to be compiled in 32-bit mode. (Looks like that's what this change was for). I don't know whether that means Premake's code is doing the wrong thing or just that the documentation is imprecise, though.