Working On Upgrade To Premake4

[Yves]

You are right... I will have to check that.

I keep it as it is for the first release of the premake4 scripts. There's going to stay some "room for improvement" anyway .

EDIT - I have some questions about linking:

Currently the pyrogenesis project/solution has all the library linking flags set (-lenet -lnvtt etc...) and some of those are also set in other projects.

When trying to link on a Mac with xcode, this configuration leads to linking problems with all the libraries where only a dylib is avaliable and no static lib (*.a).

The affected libraries currently are:

-nvimage

-nvmath

-nvtt

-Mozjs-ps-debug

-Mozjs-ps-release

I couldn't find the documentation confirming it, but it looks like libtool doesn't allow linking of dynamic libraries into static libraries.

I've tried removing all the linking flags in all projects except pyrogenesis, and it works.

Currently I don't understand all the details of what's happening and probably this is even different on the various toolsets we are using.

Just to give an example: I'm not sure if the library will be included twice in the executable if it's linked statically by the static library and also by pyrogenesis itself.

Could anyone give some basic guidelines/ideas of what the concept should be and what would probably be the best way to take?

Should I just remove all the linking flags in all the static libraries and only keep them in pyrogenesis?

Edited April 9, 2011 by Yves

[Yves]

The progress is currently slow, but at least there is some progress .

The problems described in the last post are solved and I was able to compile, link and run the game on osx with an xcode project generated by premake.

Test generation is still missing for xcode, but I don't think that will be very difficult.

Hopefully supporting older versions of Visual Studio will more or less just require copying the existing c++ code to the right place and converting it to lua syntax.

Chances are good, but it could also be more complicated.

EDIT- 28th of April 2011:

All the main features are integrated and I'm now doing tests and fixing problems I find during testing.

I can't really tell how long it will take until I'm ready to release a first version for public testing.

If everything works well it could already be on this weekend... but I don't know what kind of problems I could still find.

EDIT: I've found and fixed some problems, prepared the source for distribution and started writing some documents about known issues and test cases.

Currently I'm comparing the know issues to the premake3 environment. This gives me some hints about how some of them could probably be solved.

Premake4's problems with make and parallel builds are now fixed too.

Edited May 8, 2011 by Yves

[Yves]

I have some questions about the release of my work and the integration into the svn repository.

It is really a lot of testing necessary to make sure everything works as before (or better). The problem is that I don't have enough time for that and whenever I'm close to finishing my testing, there are new changes in the current premake3 files I have to integrate and test again.

I've also red some threads in the forums where people are working on improvements that will affect premake3 files.

Currently I'm a few weeks behind with my files and will have to merge a lot of changes made during that time.

Would it be possible to just integrated it and resolve the problems later? After merging the changes, of course!

It would save a lot of work for me and would allow a much faster integration of premake4, but some things would probably be broken in the beginning.

I think the basic building on all platforms is quite stable, but some compiler- or linker-flags could be different or some less often used parameters for the premake-scripts could cause problems.

We could also have preamke3 and premake4 in parallel. If there are problems, people could switch back to premake3.

I've made it that way for testing. You can pass --use-old-premake to update-workspaces.sh for using premake3.

Obviously the problem is the same... the work has to be kept in sync as long as we use both.

What do you think?

I've attached a list of know issues.

I don't know if it helps much, because the bigger problem are the unknown ones .

I' going to attach the sources as soon as they are ready for the current revision.

Known-issues.pdf

0ad_premake4_1_0.zip

Edited June 13, 2011 by Yves

[Sebovzeoueb]

If I understand correctly (not very clued up on all the technical terms), you are making it so the game will be easier to compile for OS X, so we may be seeing an official OS X binary soon? Does this include the Atlas editor?

Keep up the good work!

[k776]

@Yves: Philip looked into integrating CMake a few weeks ago. Not sure where he got with it.

You may want to have a chat with him on IRC about it.

[Yves]

My current work is attached in the last post. I've merged the recent changes and made some basic tests.

Feel free to do with it whatever you want .

I'm also willing to support if you find bugs, have questions or need changes.

"Installation":

Copy everything into your working copy according to the directory-structure replacing files with the same name.

Usage:

By default Premake4 will be used, but you can switch to premake3 by calling update-workspaces.sh with --use-old-premake(Linux and Mac) or editing update-workspaces.bat(windows) as described in the comments.

@Sebovzeoueb

It will support xcode projects, which is apple's development environment for the Mac.

This first release doesn't really improve anything else on the Mac and Xcode support is still a bit "experimental".

I've learned a bit about the Mac and I've also seen some of the problems, so maybe I'll be able to do more in the future.

@k776

In this case it would be interesting to know if my work on premake4 will actually be used or not.

It wouldn't be a total waste because I've learned a lot, but I won't spend more time on boring things like testing if nobody will every use it.

Edited June 13, 2011 by Yves

[Sebovzeoueb]

I've learned a bit about the Mac and I've also seen some of the problems, so maybe I'll be able to do more in the future.

Nice.

[historic_bruno]

@k776

In this case it would be interesting to know if my work on premake4 will actually be used or not.

It wouldn't be a total waste because I've learned a lot, but I won't spend more time on boring things like testing if nobody will every use it.

My guess is we'll use whatever works best and is easiest to maintain in the long run. I haven't seen a firm decision yet, so I would second k776's advice to discuss this with Philip (Ykkrosh). You would have some useful input into how well Premake4 works, what it can and can't do.

[Yves]

Ok I'll try talking to Philip around the end of the week or on the weekend. Currently I'm a bit too busy.

Probably he will already have seen this thread until then.

[janwas]

Yeah, it's hard to hit a moving target I think your suggestion of

integrating the current status is a good one. Only if there are real

dealbreakers would we have to specify the --use-old-premake flag.

This would greatly ease testing, since so many people would actually

be using it.

I have some comments on your open issues:

1. isn't a major problem, since it should be possible to upgrade to VC2010
   (this used to be annoying before the advent of the Express Edition,
   but the price is now right )
   
2. typical usage is to create the makefile on the system that will compile.
   I don't see a need to exchange makefiles between systems, so it's all good.
   
3. can't comment here, don't know anything about *.app.
   
4. does the POSIX dlopen API not support dylib on OS X?
   What would be required to make it work on OS X? Something like a
   sys_LoadLibrary interface could easily be added with platform-specific backends.
   Currently, we've been using POSIX dlopen as the "lowest common denominator",
   but if that's incorrect, we can adapt.
   

Thanks for your work on this

Once you and Philip have agreed on a time, I'd appreciate you letting me

know and will also try to be there. Philip recently mentioned a bit of

trouble with assembly code and his CMake (which I understood to be an

experiment and not at all a decision), so I have ripped out all the .asm

locally, and it's no longer something we need to worry about.

It's not yet committed, though, due to lack of testing on OS X, which is kind of

ironic, hehe.

[Yves]

Thank you for the feedback!

I'll let you know if/when we meet on irc.

[*]does the POSIX dlopen API not support dylib on OS X?

What would be required to make it work on OS X? Something like a

sys_LoadLibrary interface could easily be added with platform-specific backends.

Currently, we've been using POSIX dlopen as the "lowest common denominator",

but if that's incorrect, we can adapt.

Honestly I didn't look very deep into this. The first problem is that the current code simply looks for a *.so file instead of a *.dylib which should be easy to change.

As far as I know dlopen can be used too, but I don't know if there are any differences that could be a problem. I can try to fix it when I have time to.

[janwas]

Cool!

Apparently dlopen does indeed support dylib:

http://en.wikipedia.org/wiki/Dynamic_loading

I'd suggest additional cases for the primarySuffix definition, and then we should be good to go

[Ykkrosh]

(I'm not predictably available until after this weekend, so I probably can't do synchronous chatting until then, but I'll be occasionally around on the forums.)

I tested the new build code on Linux and it seemed to compile everything fine, which is good . (Haven't looked at the code in any detail yet, though.)

I started experimenting with CMake because of two main issues with the old Premake system. (I didn't want to conflict with the work on upgrading Premake, I just wanted to get a better perspective on the build process.)

* Poor integration with bundled third-party libraries - currently we have the update-workspace.sh wrapper script to build them, which is slow (it builds both Debug and Release versions of everything, regardless of which you need, and it re-runs the configure commands every time you run the script) and inflexible (we can't e.g. skip building the bundled copy of ENet if the system already provides the right version of it, without a load of complexity in the shell scripts) and ugly.

* Lack of a configure step for Unix platforms - currently we can't reliably tell whether the compiler is 32-bit or 64-bit (which still causes problems for some users), we have to write manual tests with 'os.execute("gcc -dumpversion > .gccver.tmp")' etc to detect compiler versions (which isn't nice), we can't reliably find libraries automatically (which causes problems when people have unexpected names for Boost libraries), and if you want to compile with a non-default compiler (icc, clang, etc) then you have to remember to set the right environment variable when running premake and also every time you run make.

CMake 2.8 seems to address the first problem fairly nicely - it only takes about seven lines to use ExternalProject_Add to integrate an external project. It gets built when you run 'make' (which seems the most sensible time to build things), it can build just the current configuration, it works with parallel builds, it runs the external project's 'make install' properly (while our shell scripts just use 'cp'), and it's running in the same build script that does find_package() etc so it can easily depend on whether an acceptable library is already found in the system.

CMake also does a normal configure step, which provides all the important information to the script (architecture, compiler version, etc, plus it has loads of standard modules to locate libraries), and which stores all the relevant data (compiler path, configuration selection, etc) and uses them each time you run 'make' (so it's easy to build with non-standard compilers).

I've not yet seen anything indicating that Premake 4 handles these issues better than Premake 3 does (though I haven't tried confirming that by looking carefully). I only played with CMake for a few days, a couple of weeks ago - enough to get it to compile a runnable executable on Linux and to almost compile on Windows, but far from complete (e.g. I didn't look at CxxTest integration or OS X at all), and it probably has other issues I haven't encountered yet. Currently I really don't know what's the best direction to go in

[Yves]

Thank you for the input!

I haven't yet looked at CMake, but my experience is that usually there's no perfect solution and we will most likely have different problems with CMake we don't know about yet. Probably (I don't know) CMake fits our needs better, but in my opinion the situation is as follows:

I've put a lot of work in the integration of premake4 and we are very close to integrating this and use it as a base for further improvement.

I'm not willing to do the same from scratch with CMake probably just to see that It isn't any better after months of work.

Maybe someone of you could do that much faster, but you'd still need a lot of time. You wouldn't expect how much time you loose for preparing environments and operating systems, switching between environments, recompiling, reintegrating changes, testing, finding bugs etc...

I can help with the integration of Premake4 and probably some of the problems you mentioned can also be solved with the current release or future releases of Premake.

There are some improvements coming with Premake4 like support for vs2010 and Xcode(probably still experimental). There are also other IDEs that could be integrated (Code::Blocks, CodeLite).

I would appreciate a clear decision.

EDIT:

I've attached a patch for fixing known issue number 4 (dllloader not looking for *.dylib).

In my opinion this should be changed no matter if premake or cmake will be used.

The code now tries loading *.so, *.dylib, *_dbg.so and *_dbg.dylib in that order for release builds on the Mac.

I've also changed the behaviour errors are reported. It now reports errors for all tries and not only the first one, if the loading fails.

I've tested the error reporting and successful loading on both Linux and Mac, but not yet on Windows.

Please do a careful review because I'm not yet familiar with the coding conventions you use and because I haven really coded c++ for quite a long time .

patch.txt

Edited June 19, 2011 by Yves

[janwas]

Thanks for this patch! I've committed it after some modifications (using ARRAY_SIZE

magic to avoid the need for constants, simpler loop logic and error reporting).

It has been successfully tested on Windows, but I'd appreciate a quick test run on OS X,

because of the extra #if that changes things (you never know)

I personally am sympathetic to Premake - partially because it gives us a

full programming language, which is nice when we need some unforeseen project customization

(asm, PCH, other complexities).

CMake's scripting language sucks, but it does have more logic for common cases built in.

I guess one thing we can do to narrow the gap is to implement some of that in Lua rather

than the shell script - should be nicer + more flexible.

However, my understanding of Linux builds is too vague, so let's talk this over together with Philip

Just let me know a time.

[Yves]

Thank you!

I'll have to look closer at it later.. I just discovered that I probably didn't test loading dylibs properly.

I couldn't test with atlas because building it fails and collada doesn't seem to be loaded always (only the first time for converting something?).

Otherwise it seems to work. The only small problem is that there could be a maximum of four error messages which will be written to one line that won't fit on most screens.

[janwas]

hrm. Shall we try to get Atlas to compile and run, while we're at it?

Yes, Collada is only loaded on demand - a heavy-handed way to ensure that would be to delete the entire cache (not a bad idea anyway, that gets rid of old/no longer needed files).

The only small problem is that there could be a maximum of four error messages which will be written to one line that won't fit on most screens.

Yep. Doesn't the console wrap around, though? The final error messages are probably uninformative anyway, so I don't think this is a problem.

[Ykkrosh]

(Sorry, still been kind of busy and not had time to think about this . I should be around on IRC all weekend if there's things to discuss, though.)

I haven't yet looked at CMake, but my experience is that usually there's no perfect solution and we will most likely have different problems with CMake we don't know about yet. Probably (I don't know) CMake fits our needs better, but in my opinion the situation is as follows:

I've put a lot of work in the integration of premake4 and we are very close to integrating this and use it as a base for further improvement.

I'm not willing to do the same from scratch with CMake probably just to see that It isn't any better after months of work.

Maybe someone of you could do that much faster, but you'd still need a lot of time. You wouldn't expect how much time you loose for preparing environments and operating systems, switching between environments, recompiling, reintegrating changes, testing, finding bugs etc...

Hmm, yeah, I think I agree with all that - I'd prefer to avoid spending time testing and fixing all the little details on all the platforms for CMake, and the benefits probably don't outweigh the cost plus the drawbacks, whereas adopting the Premake4 upgrade has little cost and gives immediate advantages (assuming it means VS2010 will work properly). It doesn't need to be perfect now - as long as it works, we can make incremental improvements later. So I think I'd be happy to integrate and commit it now, and fix any problems that come up in testing, and not worry about any further changes until it's stable, if that sounds alright to people.

collada doesn't seem to be loaded always (only the first time for converting something?)

The converted files are cached in ~/.cache/0ad/ so you can delete that to force it to reconvert everything.

[janwas]

So I think I'd be happy to integrate and commit it now, and fix any problems that come up in testing, and not worry about any further changes until it's stable, if that sounds alright to people.

Integrating+committing would be great I'm not sure if you are planning on keeping Premake4 as the default, as suggested by Yves. I think that would be a good thing in terms of testing, but could also live with the reverse, where we have to specify an additional option to get Premake4 until it has proven itself.

[historic_bruno]

So I think I'd be happy to integrate and commit it now, and fix any problems that come up in testing, and not worry about any further changes until it's stable, if that sounds alright to people.

Sounds good to me

[janwas]

Philip mentioned the following during the meeting:

1) he will be busy for the next few days upgrading the server to one with more (and expandable) disk space

2) after that, the plan is to commit the Premake4 build, replacing the old one at the same time

3) any problems that may arise should be easy to fix, there's no point maintaining a second version.

Sounds good to me

[Yves]

Thank you, that's good news .

I'll try to look at the forums more regularly in the next weeks so I can try to help if there are questions or problems.

hrm. Shall we try to get Atlas to compile and run, while we're at it?

....

Doesn't the console wrap around, though? ...

I've tested with deleting the cache and can confirm that it(still) works.

The console doesn't wrap around, it's all on one line (see screenshot)... but that's just cosmetics.

Hopefully I'll have some time next week to work on the problems with compiling atlas on OSX. Maybe It's just a small thing.

[janwas]

Thanks for confirmation

You may find the `watch this thread' (blue button near top of page) useful - that will notify you via email when a reply to this particular thread is posted.

Wraparound: oh, right, I thought we were talking about the system shell. I can live with the lack of wrapping in this case, though it's good to avoid in general.

Would you appreciate you giving Atlas a go if/when time permits!

[Chakakhan]

There are issues with the Premake4 scripts. The solutions will build, but they throw exceptions when they try to initialize OpenGL.

I have tested this with VS2010 Express on XP, VS2010 Professional on Windows 7 and VS2010 Ultimate on XP. All have the same problem. Side note: on my Windows 7 box, the exception dialogs are not showing up and the application hangs waiting for input into the invisible dialog box (I have multiple monitors on that machine as well, so that may be a contributing factor to the dialogs not showing up).

I also tested the Premake4 2008 scripts by upgrading them into the same 3 environments above and got the same error.

I checked all the compile and link switches and it all looks the same. I am going to take a look at the Lua scripts and see if I can figure anything out.

[Yves]

@Chakakhan

I had errors with OpenGl initalization too, no matter if using premake3 or premake4.

The solution was updating my ati graphics driver.

@janwas

I've looked at the atlas problems.

The version of wxwidgets coming with macports doesn't support std:wstring for some functions.

Changing two lines of code makes it compile, but linking still fails.

The problem is that there is no 64bit version of wxwidgets provided by macports.

I'm still looking for appropriate solutions...

Thank you for the hint about watching a topic.

I already know about that feature, but it suddenly stops working from time to time. I don't know why this happens and haven't yet tried figuring it out.