Jump to content

Build environment and deployment on the Mac

Yves
 Share

Recommended Posts

  • Replies 333
  • Created
  • Last Reply

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Popular Posts

wraitii
wraitii

Success! So what I did above is sufficient, I had merely forgotten to recompile the libraries with the new iconv. After using force-rebuild (specifically, only wxwidgets and libxml2 ought to be needed

Yves
Yves

I decided to open a thread about this topic because I think we need to collect the available information somewhere. There are different people who already worked on improvements of Mac support, but we

historic_bruno
historic_bruno

Some updates on the OS X build front. I've decided the way I was doing SDK builds wasn't entirely correct. I was building against the SDK of the oldest version of OS X we want to support (currently 1

stwf Sesquiplicarius

stwf

Posted
Posted (edited)

Hello All,

I made it my goal to get you a double clickable app for 0AD on the Mac, and I am almost there. It all compiles into a framework in XCode, just need to finish getting the linking to work...

.. It has been a pain.

I am going the route of a hand built XCode project, so I'm not sure it fits in with your vision should how it works. Last I read you seemed to want to automate creation of the project? Anyway I could just maintain this myself in github and use it to provide you with an app that will run on a clean system, my goal!

As for the storage issue, things that the app absolutely needs should go inside the app. Things it creates and manipulates itself should go in application support, and things the user can manipulate should go in docs.

So a good app would have a basic maps, themes, scenarios bundled inside it. Space willing all maps should go in there. This way it can be drag copied to a new machine and still work. If that takes up too much space you'll need an installer to put large maps, music, etc inside Application Support.

It would be nice to have an optional 0AD folder inside the Documents folder where I could add themes/mods. I'd like to see saved games there too, so I could manipulate, copy and archive them myself.

Edited by stwf
Link to comment
Share on other sites
Guest afeder

Guest afeder

Posted
Posted (edited)

A little note about Android, just to make sure people have it in mind if a port to this platform materializes:

1) At least one of the bundled libraries (spidermonkey) need to be patched, if using the approach I've come up with so far. I don't know how this would fit into the plans. Probably the patch can be included in the SVN and then update-workspaces.sh can apply it if instructed to target Android...

2) The filesystem on Android is different from any other platform, so all the paths need to be rethought. As far as I understand now, all apps can access the SD card (path: "/sdcard"). It might be a good idea to designate this as the place to put mods - then anyone can distribute mods through Android Market and have them install themselves in this shared space. Due to space requirements, things like game assets might also be stored on the SD card. Other things, like config and savegames, might be placed in the game app's "special data directory" - this is a directory which no other apps can access, residing in the device's internal flash memory, which is erased when the app is uninstalled.

Edited by afeder
Link to comment
Share on other sites
Echelon9 Discens

Echelon9

Posted
Posted (edited)
  stwf said:
1328899996' post='233922']

Hello All,

I made it my goal to get you a double clickable app for 0AD on the Mac, and I am almost there. It all compiles into a framework in XCode, just need to finish getting the linking to work...

.. It has been a pain.

I am going the route of a hand built XCode project, so I'm not sure it fits in with your vision should how it works. Last I read you seemed to want to automate creation of the project? Anyway I could just maintain this myself in github and use it to provide you with an app that will run on a clean system, my goal!

A hand-rolled Xcode4 project file already exists to build an OAD .app bundle.

However, as you noted it needs to be kept updated when source code files are added or deleted vs the automated Premake solution that works nicely on Windows, but poorly on Xcode project files.

http://www.wildfiregames.com/forum/index.php?showtopic=15145&view=findpost&p=233049

Edited by Echelon9
Link to comment
Share on other sites
stwf Sesquiplicarius

stwf

Posted
Posted
  On 11/02/2012 at 3:13 AM, Echelon9 said:

A hand-rolled Xcode4 project file already exists to build an OAD .app bundle.

However, as you noted it needs to be kept updated when source code files are added or deleted vs the automated Premake solution that works nicely on Windows, but poorly on Xcode project files.

http://www.wildfireg...ndpost&p=233049

missed that! I'm downloading it now and will try it out. So what work still needs to be done on the project to produce a stand alone app?

My strategy was to make the pyrogenesis part a sub-project that builds a framework. So that might make the Premake part easier. Since Premake only needs to build the pyrogenesis subproject. Then you can keep the stuff thats nasty inside the hand made project, stuff that won't change very much.

Link to comment
Share on other sites
historic_bruno Primus Pilus

historic_bruno

Posted
Posted
  On 11/02/2012 at 12:04 AM, afeder said:

1) At least one of the bundled libraries (spidermonkey) need to be patched, if using the approach I've come up with so far. I don't know how this would fit into the plans. Probably the patch can be included in the SVN and then update-workspaces.sh can apply it if instructed to target Android...

This is no problem, there are already plans to separate the game's libraries by OS. In other words, we'll have \libraries\win32, \libraries\osx, \libraries\linux, etc. Developers will only checkout whichever directory they need. It wouldn't be too hard to add some Android-specific patches, including whatever eases 0ad development on Android, assuming we decide to support Android.

  Quote

2) The filesystem on Android is different from any other platform, so all the paths need to be rethought. As far as I understand now, all apps can access the SD card (path: "/sdcard"). It might be a good idea to designate this as the place to put mods - then anyone can distribute mods through Android Market and have them install themselves in this shared space. Due to space requirements, things like game assets might also be stored on the SD card. Other things, like config and savegames, might be placed in the game app's "special data directory" - this is a directory which no other apps can access, residing in the device's internal flash memory, which is erased when the app is uninstalled.

Sounds good to me. Maybe start a new topic or Trac ticket for this, as you get closer to a working version of the game.

Link to comment
Share on other sites
historic_bruno Primus Pilus

historic_bruno

Posted
Posted

OK, my libraries build script is nearly done. The game libraries directory will change to look like this: (edit: updated per Philip's corrections)

libraries
    \-- source (source, headers, and build scripts shared between different OSes)
        \-- cxxtest
        \-- enet
        \-- fcollada
        \-- nvtt
        \-- spidermonkey
        \-- valgrind
    \-- osx
        \-- build-osx-libs.sh
    \-- win32
        \-- bundled static libs and headers for Windows builds, with wxWidgets included

"source" is used to avoid duplicating the same files on different platforms, since our NVTT and FCollada are highly modified from the official releases. Its contents are shared by Unix-likes, OS X, and Windows.

"osx" will contain only a build script, used to download all dependency source tarballs and build them in one step. 64-bit Windows and 32-bit OS X are not supported, yet.

The way of making this clear, I believe, is to have an empty "libraries" directory in SVN, with a readme that explains the further process. You will need to do a separate checkout of the libraries for your OS of choice (separate from the rest of the source code). This also has the benefit of reducing clutter and the checkout size, since Linux users won't be downloading prebuilt Windows libs. Of course BuildInstructions will be modified to reflect this change.

Every new OS X developer will run "build-osx-libs.sh", make some tea/coffee, wait maybe 10-30 minutes until the script finishes, then have all the dependencies necessary to build the game, without needing Mac Ports, though Xcode may still a dependency to get the Unix build tools (there's probably other ways to get them if someone is so inclined, but we're not supporting that yet). Currently it only builds static libs, I see this posing the least difficulties in creating a portable version of the game.

For simplicity, this will be the only build method for OS X, unless we want to complicate things with one or more of the following:

  1. Support Mac Ports / other package managers - this would require a lot of conditional handling in Premake because search paths are different and pkg-config would be needed
  2. Offer prebuilt libs in libraries\osx as we do on Windows - would save some time in the build, especially on slower computers, but again requires lots of special handling in Premake and hardcoding the results of sdl-config and wx-config (similarly increases maintenance for us)
  3. Build dynamic/shared libraries instead - not difficult to do in the build script, but it does require the extra install_name_tool steps. Benefit would be people could swap different library versions in a bundle without having to rebuild the whole game, in case of incompatibilities. Is this is a reasonable concern? I don't know :)

And finally here is what the script builds:

  1. SDL 1.2.15
  2. Boost 1.48.0
  3. wxWidgets 2.9.3
  4. libjpg 8c
  5. libpng 1.5.8 (the reason for this is even though it's included on Snow Leopard, its version 1.2 is ancient and the game won't build with it due to missing some API features. Seems like less effort to just build it ourselves, unless someone is familiar with libpng development)
  6. libogg 1.3.0
  7. libvorbis 1.3.2
  8. libxml2 2.7.8 (this is also bundled with SL, but I need xml2-config from a consistent version, I think, so that's why I'm building it. Other suggestions welcome)
  9. ENet 1.3.3
  10. Spidermonkey 1.8.5
  11. NVTT (custom)
  12. FCollada (custom)

All these libraries are built as 64-bit against SDK 10.6 with minimum required OS X version 10.6, since we're not supporting 10.5 yet (if ever). They are make install'ed into the local directory, so no root permissions are needed, but again this means the absolute path gets built into the *-config utilities. Even moving your build directories will require a full rebuild.

Link to comment
Share on other sites
Sonarpulse Sesquiplicarius

Sonarpulse

Posted
Posted (edited)

Nice work! Is this is based around premake or something else? I think your current method is good, but I would rename some folders so as other platforms, package managers, etc are support, stuff still makes sense. (I put those other features in the hierarchy too, not saying you should add them now)


Prerequisites
\-- Source
\-- Include
	\-- just library headers (of course)
\-- Dynamic Lib
    \-- android arm (windows 8)
	\-- osx x86_64
	\-- osx x86_32
	\-- iOS arm (windows 8)
	\-- windows x86_64
	\-- windows x86_32
	\-- windows arm (windows 8)
\-- Static Lib
    \-- android arm (windows 8)
	\-- osx x86_64
	\-- osx x86_32
	\-- iOS arm (windows 8)
	\-- windows x86_64
	\-- windows x86_32
    	\-- current contents of libraries
	\-- windows arm (windows 8)

Edited by Sonarpulse
Link to comment
Share on other sites
historic_bruno Primus Pilus

historic_bruno

Posted
Posted
  On 18/02/2012 at 11:48 PM, Sonarpulse said:

Nice work! Is this is based around premake or something else? I think your current method is good, but I would rename some folders so as other platforms, package managers, etc are support, stuff still makes sense. (I put those other features in the hierarchy too, not saying you should add them now)

Well the library build script has nothing to do with Premake, but of course the results are used by Premake when building the game, so I did have to modify it a bit (mostly extern_libs4.lua). Now I just need to make sure I can copy a build from Lion to Snow Leopard, if it works I'll be happy, attach the script and hopefully we can get the ball rolling on this :) I don't know if the change will be made for Alpha 9 as that's less than a month away.

I think the libraries could possibly be organized further as you say, but that's not really my goal. For instance, there's a lot of Windows dlls in binaries\system that have no use except on Windows, it would be nice to do something about that. Some day :)

Link to comment
Share on other sites
historic_bruno Primus Pilus

historic_bruno

Posted
Posted

Ran into a few tricky problems while working on the script. I couldn't figure out why it kept linking to the wrong version of libpng (the one bundled with OS X Lion, not from my script). It turns out a bunch of system search paths get added before any of the game's bundled library paths, and they'll be searched in that order. So I'm thinking we should move those to the end of the search list. I don't know if this causes problems on other platforms, but it shouldn't? Seems like all that's needed is factoring out the few offending lines into their own function and calling it in the correct order.

Also Premake needs a "sysroot" option to handle SDK builds, otherwise you have to define environment variables like CPPFLAGS every time you build, which is less than ideal. This would also help the Android cross-platform build if I'm not mistaken. For OS X specifically, we need to handle the minimum required version parameter, as that tells the compiler how to link certain parts of the API depending on which version of OS X introduced them. Perhaps a "min-osx-version" option, or it could be done with an environment variable, but an option is more explicit and better documented. Both "sysroot" and "min-osx-version" would be set by update-workspaces.sh for the convenience of OS X devs (defaulting to /Developer/SDKs/MacOSX10.6.sdk and 10.6 respectively)

  On 19/02/2012 at 5:59 AM, General Octavian said:

Will this build allow for atlas and sound available on the mac version?

Atlas - yes, sound - no. It doesn't change the sound problems, but if there's no other solution forthcoming we could always add OpenALsoft to the script.

Link to comment
Share on other sites
Ykkrosh Primus Pilus

Ykkrosh

Posted
Posted
  On 18/02/2012 at 10:53 PM, historic_bruno said:
osx (should it be named osx86_64? since it only supports Intel 64-bit)

(I don't think it should - if we add other OS X platforms later, they should probably be added to the same build script (to avoid redundancy) inside libraries/osx/, and they don't need a new top-level directory. Platform-specific compiled output could perhaps go in libraries/osx/x86_64/ if there's a real desire to support multiple platforms in parallel in a single checkout, but that sounds quite a theoretical concern to me.)

  Quote
"linux" will contain only headers.

That doesn't sound right - it needs to contain the source for (if I remember correctly) nvtt, fcollada, spidermonkey, enet, and probably valgrind (only headers), and shouldn't contain anything else. It would be nice if we didn't duplicate the forked source in both libraries/osx/ and libraries/linux/ - maybe put them into a single libraries/sources/ (along with any shared-by-Linux-and-OSX custom build scripts for each).

  Quote
The way of making this clear, I believe, is to have an empty "libraries" directory in SVN, with a readme that explains the further process. You will need to do a separate checkout of the libraries for your OS of choice (separate from the rest of the source code).

Could it be made part of the existing update-workspaces.{sh,bat} step? That can automatically do a "svn co" to get or update the appropriate libraries/{linux,osx} (and libraries/sources/{nvtt,...}), and then (on Linux/OSX) automatically run the library build script, so we don't need to worry about users getting it wrong or forgetting to update when we change the libraries.

(In that case, update-workspaces should probably specify an explicit SVN revision of the libraries that it wants, and we should update that specification when updating the libraries, so that people can still build old versions of the game successfully with the contemporary libraries, instead of it always requesting the latest versions. Also, need some alternative for Linux release downloads so that they bundle everything necessary, since package build scripts shouldn't perform their own network access.)

(update-workspaces should quite possibly be moved and/or renamed, perhaps to "build-setup" in the root directory or something like that, since currently it's weirdly hidden away.)

  Quote
OS X [...] Currently it only builds static libs, I see this posing the least difficulties in creating a portable version of the game.

Linux packagers generally insist on dynamic linking for various reasons, but OS X development never seems to bother with anything like that, so static linking on OS X sounds reasonable and least problematic. (Static linking also means bigger executables (which I wouldn't like on Windows since it would probably make the autobuilds much bigger), and seemingly much worse linking times (which I wouldn't like on Linux since I compile the game myself on Linux and am impatient) ((unless I'm misinterpreting the cause of the slow linking of Android builds)), but those don't seem particularly important on OS X.)

Link to comment
Share on other sites
historic_bruno Primus Pilus

historic_bruno

Posted
Posted

By the way, my test build works on both Lion and Snow Leopard, the only thing I don't know is if it will work on Mountain Lion ;)

  On 19/02/2012 at 2:46 PM, Ykkrosh said:

That doesn't sound right - it needs to contain the source for (if I remember correctly) nvtt, fcollada, spidermonkey, enet, and probably valgrind (only headers), and shouldn't contain anything else. It would be nice if we didn't duplicate the forked source in both libraries/osx/ and libraries/linux/ - maybe put them into a single libraries/sources/ (along with any shared-by-Linux-and-OSX custom build scripts for each).

You're quite right about that, and I also like the idea of a shared directory, so I'll work on that :)

Link to comment
Share on other sites
historic_bruno Primus Pilus

historic_bruno

Posted
Posted

Output of otool -L for pyrogenesis, test, ActorEditor, libAtlasUI, libCollada:

  Reveal hidden contents

Notice the minimal number of dylibs they link against :P What I don't know is if Apple plans to maintain compatibility with the 10.6 SDK going forward, nothing they do would surprise me, we'll just have to wait and see. It would be annoying if e.g. Mountain Lion ships without some system dylib we depend on.

Link to comment
Share on other sites
historic_bruno Primus Pilus

historic_bruno

Posted
Posted

Build script for OS X libraries, but note you can't really run it because paths have changed:

  Reveal hidden contents

By default, it builds against OS X 10.6 SDK with minimum required OS X version 10.6 (what this means: only system libraries available on Snow Leopard or later are linked. Further, you will likely get compile errors if you try building on 10.5. Changing these variables and it might work. However, the minimum version also tells the compiler how to link system functions: for 10.6 or earlier functions they are linked at compile-time, i.e. "strong linking", but 10.7 and later functions (assuming you changed the SDK) are linked at run-time, i.e. "weak linking". We don't handle that distinction yet, nor do we have much cause to, we'll just need to be aware of this).

Because I was so tantalizingly close, I made a quick app bundle script which does all the things needed for a release. I'm sure it can be improved, but at least this way we might have something in place for Alpha 9 like we originally planned:

(note: this script won't work for you, because I've changed libraries paths and Premake scripts, it's just to show what's involved :P Very much subject to change)

  Reveal hidden contents

As you can see the steps to creating a bundle are:

  • Clean and generate workspaces with Premake
  • Build game
  • Confirm tests succeed
  • Create standard app bundle directories
  • Run archive builder on public mod (precaches XML, textures, and COLLADA models)
  • Copy files into the bundle (shared libs into Contents\Frameworks, data into Contents\Resources, binary into Contents\MacOS) - for the reasons see Apple's Bundle programming guide
  • Create Contents\Info.plist using PlistBuddy - this is just an XML property list file that contains info about the bundle

Almost everything is done on the command line with the exception of creating the 0ad icon (icns seems to be the only format OS X recognizes and I created it using http://iconverticons.com from our crappy low resolution 0ad.png) and compressing the bundle into a dmg. I'll investigate whether/how these steps can also be performed on the command line. Final package ends up being around 375MB and I don't know anywhere to upload this for testing :( I tried copying the bundle from Lion to different locations on Snow Leopard, and it ran perfectly including Atlas.

As far as making the bundle paths work, I've got a patch that adds some sysdep code so we can detect if the binary is in a valid bundle (tests if the bundle identifier is "com.wildfiregames.0ad") and a few functions to get the internal Resources and Frameworks paths (unrelated to previous paths patch). Paths are not hardcoded into the build unlike most of the solutions used previously.

Link to comment
Share on other sites
historic_bruno Primus Pilus

historic_bruno

Posted
Posted
  On 19/02/2012 at 2:46 PM, Ykkrosh said:

Could it be made part of the existing update-workspaces.{sh,bat} step? That can automatically do a "svn co" to get or update the appropriate libraries/{linux,osx} (and libraries/sources/{nvtt,...}), and then (on Linux/OSX) automatically run the library build script, so we don't need to worry about users getting it wrong or forgetting to update when we change the libraries.

(In that case, update-workspaces should probably specify an explicit SVN revision of the libraries that it wants, and we should update that specification when updating the libraries, so that people can still build old versions of the game successfully with the contemporary libraries, instead of it always requesting the latest versions. Also, need some alternative for Linux release downloads so that they bundle everything necessary, since package build scripts shouldn't perform their own network access.)

(update-workspaces should quite possibly be moved and/or renamed, perhaps to "build-setup" in the root directory or something like that, since currently it's weirdly hidden away.)

That's a bit more complicated than what I've got working now, but it sounds like it would make the build process slightly less painful. That would be more of an incremental improvement than an initial change. I don't think it will ever be foolproof, compiling software is hard and at least some technical knowledge is needed :P

  Quote

Linux packagers generally insist on dynamic linking for various reasons, but OS X development never seems to bother with anything like that, so static linking on OS X sounds reasonable and least problematic. (Static linking also means bigger executables (which I wouldn't like on Windows since it would probably make the autobuilds much bigger), and seemingly much worse linking times (which I wouldn't like on Linux since I compile the game myself on Linux and am impatient) ((unless I'm misinterpreting the cause of the slow linking of Android builds)), but those don't seem particularly important on OS X.)

I haven't noticed linking taking longer than usual, though I do run the build with "-j5" :) The binary size is about 14MB using the static libs, so not huge but not small either.

  On 23/02/2012 at 2:39 AM, Echelon9 said:

I think we can get the SVN or release version in the plist file as well.

Yes, the version keys should be strings formatted like "X.Y.Z", but I have no idea how we want to format release numbers.

For anyone interested, Apple has good documentation on the Info.plist keys here and a list of recommended keys here - I added the recommend ones.

  On 23/02/2012 at 3:31 AM, General Octavian said:

Is sound working in this build?

Sound is a totally separate issue from building and deploying bundles for OS X :) Nothing works in this bundle that wouldn't already work without a bundle. Not saying it won't be fixed, but you have to take one thing at a time.

Link to comment
Share on other sites
MishFTW Primus Pilus

MishFTW

Posted
Posted
  On 23/02/2012 at 2:25 AM, historic_bruno said:

Almost everything is done on the command line with the exception of creating the 0ad icon (icns seems to be the only format OS X recognizes and I created it using http://iconverticons.com from our crappy low resolution 0ad.png) and compressing the bundle into a dmg. I'll investigate whether/how these steps can also be performed on the command line. Final package ends up being around 375MB and I don't know anywhere to upload this for testing :( I tried copying the bundle from Lion to different locations on Snow Leopard, and it ran perfectly including Atlas.

Bro, please let me get my hands on that! I don't care how big the file size may be, Geek needs updates! Like I've said before, I would take testing 0 AD over doing my boring history homework any day. (It's a win-win situation :))In the meantime, I have a pretty decent icon file.

Link to comment
Share on other sites
gudo Duplicarius

gudo

Posted
Posted (edited)

.torrent? Or, you could break it into smaller archive files and upload to zippyshare.com. You don't need an account, but file size is limited to 200MB.

Actually, now that I think about, would distributing the various alphas as a .torrent be possible? (with the current DL server as a web seed) Seems a good idea all around in terms of bandwidth and speed.

Edited by gudo
Link to comment
Share on other sites
historic_bruno Primus Pilus

historic_bruno

Posted
Posted
  On 23/02/2012 at 3:42 AM, Geek377 said:

Bro, please let me get my hands on that! I don't care how big the file size may be, Geek needs updates! Like I've said before, I would take testing 0 AD over doing my boring history homework any day. (It's a win-win situation :))In the meantime, I have a pretty decent icon file.

Nice icon! I'm using it now :D In return I'll provide you with an app bundle: http://hotfile.com/dl/147504648/be7bb7e/0ad-r11112-osx.dmg.html

(Edit: Fixed app bundle with correct icon, new link)

Link to comment
Share on other sites
wraitii Primus Pilus

wraitii

Posted
Posted (edited)

That's great, historic_bruno. Very nice work :)

Just one question: you seem to export the collada models directly, ie not use the library... Any reason for it?

Edit: just tried it. Downloaded, started from the dmg (which sometimes can cause bugs), and everything worked fine on 10.7.2 on a MacBook Air. You've really done a great work.

Edited by wraitii
Link to comment
Share on other sites
historic_bruno Primus Pilus

historic_bruno

Posted
Posted
  On 23/02/2012 at 6:34 AM, wraitii said:

Just one question: you seem to export the collada models directly, ie not use the library... Any reason for it?

There's no technical reason, but that's how Alpha 9 data will be packaged :) If you think about it, it doesn't really make sense to require conversion of archived COLLADA documents at runtime.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
 Share
Go to topic listing
×
×
  • Create New...