Build environment and deployment on the Mac

[Yves]

Saved games are definitely managed by the game, as the user doesn't get to choose the names or where they are located. We'll be adding a "delete" button to the load game menu, so at that point they will be fully managed by the game.

The user doesn't choose where to put the files, but he at least triggers creating/deleting of savegames.

Also the savegames aren't tied to a specific system and someone could want to copy them from one computer to another, share them with friends or whatever.

I would tend to say they are more user-managed than game-managed.

About the mods I'm not sure because I don't know enough details about how they will work and how they will be distributed and used.

Probably Application Support is better.

[Echelon9]

Discussion is progressing well.

In relation to mods and total conversions, I'd like to suggest that they should go in /Applications/0ad/ and then be dealt within the VFS system (which I'm not fully across just yet) - whether that means in practice a zip file in the top level folder, a sub-folder or someother compressed archive.

To my mind where a mod changes -- or even potentially totally converts gameplay elements in a total conversion -- the adoption of those changes should be quite clear and deliberate, almost forming a new game. This is why it makes sense to have them within the /Applications/0ad/ folder.

[historic_bruno]

The VFS (virtual file system) means you can mount real OS paths as relative to a virtual root in the VFS, then almost every file the game uses goes through VFS (the virtual path gets translated to a real path by I/O functions). You can mount multiple real paths to the same VFS directory, with different priorities. Basically it's a way of abstracting how different OSes and file systems handle paths, and allows us to easily support mods that overwrite parts or all of the game data. Atlas-based tools also use VFS, they pass messages from the UI to the game engine, which handles the I/O. Currently it's a bit messed up because although VFS reads follow the priorities system, writes do not, making the concept of a user mod useless until that's resolved.

IMO, total conversions should consist of a new app bundle, because it's likely the engine itself will be changed. In the case of other mods, I don't think we should place anything in or with the app bundles after distributing them.

[Yves]

I think we have enough overview and can start doing some work. We can still change any of those paths later if we need to.

I've added some description to each task on the wiki-page and made a note who is assigned to that task.

Feel free to take an unassigned task.

I've only taken those tasks I already started working on, but If nobody wants to do the others, I will take care of them.

@historic_bruno

I've assigned you one of the tasks because I think it covers one of the ticket's you are assigned to.

Is that OK?

edit: wrong link.

[Juicyfruit]

I would also like to contribute some libraries or look at the build script for them.

Maybe we can collaborate on github or gitorious or something as a sort of staging area before it goes into the main tree.

For the short time we could make a script that just does a fix-up on the resulting binary by moving files into it and calling install_name_tool.

I think we should focus on making one bundle that works on snow leopard and lion i386 and x86_64. We should links some things static ( thinking about boost specifically ). But most of all we should work to get it working really well soon then use that experience to fix the building process properly.

[Yves]

Maybe we can collaborate on github or gitorious or something as a sort of staging area before it goes into the main tree.

Not sure if we need that. I think we won't change many existing files.

For the short time we could make a script that just does a fix-up on the resulting binary by moving files into it and calling install_name_tool.

That's quite similar to what I'm doing with my script except that I'm not using the bundle the current build-process creates but create a new one instead.

What else could we do in the future? The installnames are predefined by our repository structure because after building it should work as it is for testing and development.

I think we should focus on making one bundle that works on snow leopard and lion i386 and x86_64.

Currently we tend to only support x86_64 on the Mac. Most of the clients support that and it makes it a lot easier in the first place.

We should links some things static ( thinking about boost specifically ). But most of all we should work to get it working really well soon then use that experience to fix the building process properly.

Why should we link it static? Changing the installname is easy.

[Juicyfruit]

Why should we link it static? Changing the installname is easy.

It seems common practice for boost on os x bundles I could find that use boost. Also when we dont use much functionality of a library and only one executable is going to use it anyway we might as well link it static to get a smaller total size faster load time etc.

Not sure if we need that. I think we won't change many existing files.

For getting together a set of well build libraries to populate the libraries directory to build agains. I used some extra flags to make nvtt not build support for cude/cg/jpeg/png/openexr for example. Its not like the game uses much more from there then the texture encoding things. It would be nice to hash out and share the libraries we build. Also due to the api changes between 10.6 and 10.7 we need to check that the SDL patches or snapshot we end up using does indeed work nicely on both platforms. I think the lion api it uses also works on snow but not sure, the comment in the code states the it uses the newer functionality, but it uses the old one for 10.6 even though the new way is available. Its the little things that make the resulting binary work well for a lot of people. Also I would like confirmation if openal-soft works as well for other people as it does for me. I dont have access to snow leopard at the moment but would love the feedback.

One of the reasons I am trying to make cmake work is that I dont like premake or know much about it. However for building gnu makefiles it seems very well suited. But then we need a script to add some OS X fancy things to the bundle it produces. Or am I wrong. For that we need an icon, a plist a direcotry to store libraries / frameworks. (Do we store dylib's in libraries and frameworks in frameworks ? ).

[Yves]

About the repository, It's probably useful if we want to share the data more often and collaborate more closely.

I've never made a repository on github/gitorious, so if you could create it, I'd support it and upload my work in progress there too.

It seems common practice for boost on os x bundles I could find that use boost. Also when we dont use much functionality of a library and only one executable is going to use it anyway we might as well link it static to get a smaller total size faster load time etc.

I will have to think a bit more about it because I don't really have important pro or contra arguments .

But then we need a script to add some OS X fancy things to the bundle it produces. Or am I wrong.

Yes, that's true.

I guess a few lines of code can explain best what I meant (see attachment).

It's a very early work in progress, so don't expect too much.

bundle-script.zip

[Echelon9]

Keep in mind that Xcode4 can integrate the processes of creating properly formed bundles, as a post-compilation step, including copying files/folders into the app bundle.

[Sonarpulse]

Seeing that github is brought up:

https://github.com/0ad

this is official, right?

[Jeru]

@Sonarpulse: Yes!

[Yves]

Task nbr 6 completed in changeset 10946. I've closed ticket #947 because it's about the broken app bundle.

I've now also taken Task nbr 1 because that's a requirement for even a first basic app bundle.

Btw. I was looking at some codeblocks-problems on OSX but gave it up. Premake is really a mess on those "less supported" platforms and I hope CMake is much better.

[feneur]

Task nbr 6 completed in changeset 10946. I've closed ticket #947 because it's about the broken app bundle.

I've now also taken Task nbr 1 because that's a requirement for even a first basic app bundle.

Btw. I was looking at some codeblocks-problems on OSX but gave it up. Premake is really a mess on those "less supported" platforms and I hope CMake is much better.

Does those tasks have "proper" Trac tasks?

[Yves]

Does those tasks have "proper" Trac tasks?

Some of them have more or less .

Different people created trac-tickets about the same thing or similar things. You see in the article that I tried matching some of the existing tickets to tasks but some tasks are still without ticket. I will create new tickets for those tasks without tickets if that's how it should be done.

[feneur]

Some of them have more or less .

Different people created trac-tickets about the same thing or similar things. You see in the article that I tried matching some of the existing tickets to tasks but some tasks are still without ticket. I will create new tickets for those tasks without tickets if that's how it should be done.

The main thing is that things are being done, the exact way doesn't matter that much. Since we do use Trac in general it's probably best to use it in this case as well, to make it easier to keep track on what's being worked on and not, and as you say there are some pre-existing already etc, so it's probably the least confusing in any case

[historic_bruno]

About prebuilt dependencies: I have a few concerns about how this is progressing:

• By getting rid of the Mac Ports requirement, we lose the nice pkg-config tool, which is what non-Windows builds currently use to get include and lib paths for most dependencies. For some this is not a problem, because they are simpler and self-contained, but for example SDL and wxWidgets need many frameworks and other dependencies as well as compiler directives which need to be included based on the build configuration.
  
• Luckily we have sdl-config and wx-config, but it should be noted those tools only return absolute paths from where they were installed (with make install). So my dependencies build script runs make install as a final step for each of them (to a nearby directory so sudo is not required), but that means the results are not really reusable across systems (imagine you have an SVN checkout in /path/to/ps/ but I have it in /some/other/path/to/ps/, well then my bundled wx-config and sdl-config wouldn't work for you).
  
• Normally these files go in standard OS-specific locations, so it's the same for each system. So Philip suggested that the dependencies build script should get run every time a new developer wants to build the game on OS X (or more specifically, every time they want to build in a new location).
  
• I feel that defeats the concept of prebuilt dependencies, the goal should be to offer them in SVN so all that needs to be done is a checkout of the libraries package for your OS and the game's source. We already do this for Windows, except wxWidgets but I believe we can even include that, since only Windows developers would be forced to download it.
  
• The only solution I can think of to fix the lack of pkg-config and the absolute paths of e.g. sdl-config, is to hardcode their results in extern_libs4.lua for OS X, for developers who don't want to build the libraries.
  
• So that means we would need two build paths for OS X alone: one for developing with the downloaded prebuilt dependencies (default), and one for those who use the rebuild script (maybe because they can't use the others) or Mac Ports/Homebrew or some other alternative. I guess a flag passed to update-workspaces.sh would select the second case.
  
• The worst part about that is the maintenance required, whenever the OS X dependencies changed, they would need to be rebuilt and e.g. sdl-config run to retrieve the necessary info, which would be placed by hand in extern_libs4.lua. That introduces a lot of potential for error, but maybe acceptable if it rarely happens.
  

Any other ideas? Blender SVN offers prebuilt static libs for multiple versions of OS X, Linux 32/64-bit, and Windows 32/64-bit, and they never ask you to build any of them, the result is an otherwise daunting and time-consuming process becomes much simpler. Of course they don't use Premake, but I assume they had other similar concerns in their build system

As far as the time it takes to rebuild the dependencies with my current script, I would say maybe 5 minutes on my PC using 4 parallel build tasks, and considerably longer on older hardware, that doesn't include time required to download the tarballs if necessary (I do a test in the script, if the file doesn't exist, it assumes it should be downloaded first - still need to test that actually works).

[historic_bruno]

As far as data paths go, expect a patch for #1145 very soon, taking care of both Windows and OS X at the same time.

[Echelon9]

As far as data paths go, expect a patch for #1145 very soon, taking care of both Windows and OS X at the same time.

Sorry, but I really don't like the idea of using ~/Documents on Mac as the file path location of saved games and screenshots. It just doesn't tie into the way any other (broad statement: but I haven't seen any examples) games or Mac apps utilise the file system.

Mac is often about hiding away the complexity of the filesystem (at least for novice, casual users) and providing a nice interface to underlying files and features. Accessing save games makes most sense *within* the 0AD application, I really don't care where they are on the physical file system most of the time. Storing them in Documents is too in the users face, as that location is primarily for user created documents, not outputs of an application or game.

I'm strongly of the view that all generated data should be within ~/Library/Application Support or ~/Libraries/Caches.

[k776]

I agree with Echelon. Saved games should be in ~/Library/Application Support/0ad

Only screenshots should make it into the Documents folder (~/Documents/0ad/)

Same goes for windows. %appdata%/0ad for saved games, %localdata%/0ad for screenshots (I think it's localdata for My Documents on Windows)

[Yves]

As far as data paths go, expect a patch for #1145 very soon, taking care of both Windows and OS X at the same time.

It's good to see some progress, but it's also frustrating that you have done the same thing I've worked on the last two weekends and was close to getting a patch too.

One purpose of this thread and the Wiki article was to avoid such a situation.

I said I'd do this task and also updated the wiki article accordingly in this post.

About the savegame location. That has been discussed in this thread too.

Savegames are data the user could like to access for making backups or sharing it between computers/people.

At least on Windows some games I know place savegames in mydocuments. My opinion is that they belong into ~/Documents on OSX.

EDIT - Some feedback to the patch:

Game data root:

Your patch doesn't change the behaviour as specified in the table. However, it's still possible to define INSTALLED_DATADIR to make it work in a bundle.

My approach was doing some basic validation of the paths at runtime which is checking if the directory exists in this case. If it doesn't exist, the default directory executable_path/../data/ is used.

However the question is if this path is even needed at all if we can have separate paths for its subdirectories (like mods).

Mod-paths:

I've had a quick discussion with Philip about this topic but didn't update the wiki-article.

We sorted out that we currently have at least three kinds of mod-directories.

1. Deployedmod: Usually public.zip which contains the "default mod" we deploy with the game.

2. mods: the directory containing all other mods.

3. usermod: the directory where atlas saves maps etc...

(4. downloadedmods) there's another potential directory for automatically downloaded mods... but current it should be sufficient to just have "mods" instead.

The reason to have a separate "Deployedmod" directory is to allow us including it in a simple app-bundle.

We can't have the mods-directory in the app bundle.

Directory validation:

That's another problem I tried solving with my path. Currently the game just crashes in most cases if some required directories don't meet the requirements (don't exist, wrong permissions etc...).

User-customizations:

A nice feature would be allowing user-customized paths for all those directories... but that's just nice to have.

My approach

My approach was bit more radical.

I've created an interface-class for well-known system paths and derived an OS-specific class from it for each OS.

class WellKnownSystemPaths
{
public:

virtual ~WellKnownSystemPaths();

// Directory containing the game's data (e.g. binaries/data).
// Required access: Read
	virtual const OsPath DeployedDataDir() = 0;	
// Directory where the main executable is started from.
// Required access: Read
	virtual const OsPath ExecutableDir() = 0;
// Directory containing the bundled libraries
// Required access: Read
	virtual const OsPath CacheDir() = 0;
// Directory containing User-Data. Userdata is basically all data whose creation is triggered
// by the user and which the user might also want to access from outside of the game.
// Required access: Read, Write
	virtual const OsPath UserDataDir() = 0;
// Directory containing the users customized settings for the game
// Required access: Read, Write
	virtual const OsPath UserConfigDir() = 0;

private:
};

I've also added a new class representing a game directory and containing all properties and functions such a directory could have.

The definition of this class:

// Validation flags
#define GAMEDIRECTORY_VALIDATE_EXISTANCE	1
#define GAMEDIRECTORY_VALIDATE_WRITE		2

class GameDirectory
{
public:
	// The user is responsible for checking the return-value and acting appropriately!
	bool ValidatePathsAndChoose();

	void SetUserPath(OsPath UserPath) { m_Paths[DirectorySources.USER] = UserPath; }
	void SetOSDefaultPath(OsPath OSDefaultPath) { m_Paths[DirectorySources.OS_DEFAULT] = OSDefaultPath; }
	void SetGameDefaultPath(OsPath GameDefaultPath) { m_Paths[DirectorySources.GAME_DEFAULT] = GameDefaultPath; }
	void SetValidationBitmask(Byte ValidationBitmask) { m_ValidationBitmask = ValidationBitmask; }

	// Can return undefined results if ValidatePathsAndChoose isnt used properly!
	OsPath GetPath() { return m_Paths[m_ChosenDirectorySource]; }

private:
	// Directory sources in order of priority
	enum m_DirectorySources { USER, OS_DEFAULT, GAME_DEFAULT };
	const int m_DirectorySourcesNbrOfElements = 3;
	OsPath m_Paths[m_DirectorySourcesNbrOfElements];
	Byte m_ValidationBitmask;
	DirectorySources m_ChosenDirectorySource;

	bool Validate(OsPath Path);
}

As you can see each GameDirectory can contain three paths (USER, OS_DEFAULT, GAME_DEFAULT).

It also contains a validation bitmask specifying the requirements of that directory.

ValidatePathsAndChoose() validates each directory according to its requirements specified in ValidationBitmask in the given order (User, OS, Game).

It will use the first valid path it finds.

A nice aspect of this solution it that you can define all the directories at one place in a consistent way:

	#ifdef USER_CONFIG_DIR
	m_Paths[m_GameDirectoriesEnum.CONFIG].SetUserPath(OsPath(STRINGIZE(USER_CONFIG_DIR))/"");
	#endif
	m_Paths[m_GameDirectoriesEnum.CONFIG].SetOSDefaultPath(WellKnownPaths->UserConfigDir()/subdirectoryName/"");
	m_Paths[m_GameDirectoriesEnum.CONFIG].SetValidationBitmask(GAMEDIRECTORY_VALIDATE_EXISTANCE | GAMEDIRECTORY_VALIDATE_WRITE);

I don't know if this approach is better and I can't provide a patch yet because it's not yet working.

Now it probably doesn't make sense working on it as you already have a working patch.

[Ykkrosh]

Philip suggested that the dependencies build script should get run every time a new developer wants to build the game on OS X (or more specifically, every time they want to build in a new location). I feel that defeats the concept of prebuilt dependencies, the goal should be to offer them in SVN so all that needs to be done is a checkout of the libraries package for your OS and the game's source.

I think the important goal is that building the game should be easy and automatic, not that it should be fast. For first-time developers, the time-consuming part will be downloading the game's code and data, and waiting extra tens of minutes for building won't be a major issue. For repeat developers, they'll only need to rebuild the libraries every few months when changes are made, so slowness won't matter.

On Windows we provide prebuilt libraries because there's no easy automatic way to compile them on the user's machine, and because Windows allows prebuilt libraries to be portable, but it's a bit of a mess since we end up with outdated libraries and mismatched compiler versions. On Linux we build from source because that can be easy and automatic, and prebuilt libraries wouldn't be portable. OS X seems much more like Linux in this.

[historic_bruno]

Sorry, but I really don't like the idea of using ~/Documents on Mac as the file path location of saved games and screenshots. It just doesn't tie into the way any other (broad statement: but I haven't seen any examples) games or Mac apps utilise the file system.

The advantage of having screenshots in a hidden folder would be...?

If it was so important to Apple that apps use Application Support for *all* data, they shouldn't hide it by default, now they are forcing us to consider alternatives. Also if we were writing a native Mac app, I would agree, we would use the standard file open/save UI. But we're not going to have an OS-specific UI in game AFAIK, the path will be chosen by the game but it's the user who takes the action to create the file. Saved games is a debate worth having, but if there's no strong resistance to having them in My Documents on Windows, then I don't see the problem on OS X either. We can't pretend "OS X users don't care about paths but Windows users do" because that's a false statement.

Everyone can have a preference about this, which is why something like what Yves suggests might work.

It's good to see some progress, but it's also frustrating that you have done the same thing I've worked on the last two weekends and was close to getting a patch too.

One purpose of this thread and the Wiki article was to avoid such a situation.

I said I'd do this task and also updated the wiki article accordingly in this post.

Oh. I think I did read that post, but interpreted it as working on bundle stuff, my patch doesn't have anything to do with bundles. Also I looked on Trac and didn't see a ticket and there was little activity on the forum or IRC, so I just started working on it. I wasn't trying to duplicate effort or step on toes. I don't think our work is totally incompatible, and also it was good to get some experience with Cocoa and Objective-C++ (and I discovered some unrelated bugs meanwhile)

I mostly started working on it because I knew it would only take a few days, and then we would have code to discuss what's right or wrong, like we are now so I guess it worked No need to throw away what you've been working on.

Game data root:

Your patch doesn't change the behaviour as specified in the table. However, it's still possible to define INSTALLED_DATADIR to make it work in a bundle.

INSTALLED_DATADIR should be an absolute path, so that doesn't sound like a good idea for bundles which are portable by nature. The paths I changed have nothing to do with bundles, but will be the same for SVN vs. releases.

My approach was doing some basic validation of the paths at runtime which is checking if the directory exists in this case. If it doesn't exist, the default directory executable_path/../data/ is used.

However the question is if this path is even needed at all if we can have separate paths for its subdirectories (like mods).

I don't understand. Can't we just use CFBundleGetMainBundle or something to see if we're a bundle or not and go with that? I haven't tried it to know if it works reliably (we could also check if there's a .app in the path as an additional validation, which is an "ugly" hack we use somewhere else, but it might work). My feeling is we should use it anyway to get bundle info, like the bundle name, so we can set the paths accordingly (e.g. ~/Library/Application Support/BundleName/). If I'm not mistaken, the VFS already checks if full paths exist or not and creates them accordingly, of course that doesn't help reading the root game data if it's missing, but that's something either the engine (with simple logic) or the installer should get right, rather than being left to user decisions or compile-time options.

Mod-paths:

I've had a quick discussion with Philip about this topic but didn't update the wiki-article.

We sorted out that we currently have at least three kinds of mod-directories.

1. Deployedmod: Usually public.zip which contains the "default mod" we deploy with the game.

2. mods: the directory containing all other mods.

3. usermod: the directory where atlas saves maps etc...

(4. downloadedmods) there's another potential directory for automatically downloaded mods... but current it should be sufficient to just have "mods" instead.

The reason to have a separate "Deployedmod" directory is to allow us including it in a simple app-bundle.

We can't have the mods-directory in the app bundle.

Yeah, my patch doesn't do anything about mods, and honestly the VFS is in such a state currently that it's not even worth thinking of multiple mods IMO (in fact I recall recent discussions about only allowing a single mod to be mounted as a simplifying case). But if we do, all that needs to happen is, for example: (1) is already mounted in VFS root (readonly), (2)/(4) m_GameData / "mods"/* get mounted in VFS root (readonly), and (3) m_UserData / "mods"/* get mounted in VFS root with read/write permission. Am I missing something?

Directory validation:

That's another problem I tried solving with my path. Currently the game just crashes in most cases if some required directories don't meet the requirements (don't exist, wrong permissions etc...).

Permissions is a tricky issue and one we try to avoid so far, but it's easy to fix in this case. With sensible locations for all the game's data, there's no need to worry about permissions, because the current user will have full permission for their home directory (where all the runtime data should be created) and read permission elsewhere. And there's no need to worry about whether they exist, because if they are writable runtime data paths, then the VFS will create them for us. The only thing that absolutely has to exist in advance and be readable is the data and binaries bundled with the game, in which cases are these missing or not accessible?

I don't know if this approach is better and I can't provide a patch yet because it's not yet working.

Now it probably doesn't make sense working on it as you already have a working patch.

It's always worth working on something if it can be done better, because nobody has exclusive rights to change a part of the game's source code

I think the important goal is that building the game should be easy and automatic, not that it should be fast. For first-time developers, the time-consuming part will be downloading the game's code and data, and waiting extra tens of minutes for building won't be a major issue. For repeat developers, they'll only need to rebuild the libraries every few months when changes are made, so slowness won't matter.

That would simplify things quite a bit, right now the simplest case is downloading libraries/osx from SVN, and running the single ./rebuild-libraries.sh script, which downloads, builds, and installs everything, or we can choose to provide the source tarballs, but then it could be argued that doesn't help much because it adds to SVN checkout/update time and doesn't reduce build time, but does increase maintenance.

[historic_bruno]

Argument against saved games in (My) Documents: the files have no value without the rest of the game. Screenshots have some intrinsic value, whereas saved games might as well be deleted with the rest of the game data, if it's deleted. Also, I notice Steam used to store all data in Documents, but then a bunch of people complained, so they changed to Application Support.

Related question: should we have an "uninstaller" for OS X? Otherwise we leave a bunch of files behind after the bundle is deleted. The same question could be posed for Windows and Linux.

[fcxSanya]

Argument against saved games in (My) Documents: the files have no value without the rest of the game. Screenshots have some intrinsic value, whereas saved games might as well be deleted with the rest of the game data, if it's deleted.

The game can be easily redownloaded/reinstalled, but you can't get your savegames back once you deleted them. In my experience games either don't delete saved games or explicitly ask about this in the uninstaller dialog. Our savegames probably not so worth as savegames of some RPG with tens of hours invested to gameplay, but still, we can get some unhappy users if they will reinstall the game by some reason and notice that they lost all their savegames.

[Yves]

Oh. I think I did read that post, but interpreted it as working on bundle stuff, my patch doesn't have anything to do with bundles. Also I looked on Trac and didn't see a ticket and there was little activity on the forum or IRC, so I just started working on it. I wasn't trying to duplicate effort or step on toes.

No problem, it's not your fault. I guess my current situation (very little time available during the week but a lot time needed to get all the required background knowledge) doesn't fit well for this kind of task and how we are working on it.

Collaboration means being able to give feedback or answer to feedback within a relatively short period of time, being available on irc. etc...

I started working on this because I remember that about a year ago we already discussed about Mac deployment but nobody wanted to work on it and nothing really happened since then.

Now it looks like there's actually some progress and people working on it. I don't want to constrict that progress by taking tasks and not "allowing" other people who can do it faster to work on it. But I also don't want to spend my time on something I don't know if it will be used even if the quality of the final result is good.

At the moment I have even a bit less time than normal and next weekend I'll have to work at least on Saturday. Working with the required pace on this task would be quite hard in that situation. For those reasons I decided to stop or at least interrupt my work on Mac deployment. I'll think about this decision again in about a month, but probably it will already be done by then.

I hope nobody misunderstands that in any way.

I'll still participate in the discussing when I have time to and I will probably do some smaller tasks in the meantime... and would be happy to get feedback to the minimap-patch.