Build environment and deployment on the Mac

[wraitii]

Echelon: I'm interested in your fix.

Here's a patch for OSX compilation (tested against SVN at the time of A16, not sure of the release nb). I've stopped the script from force-rebuilding libraries since that redownloads the broken libiconv.

To install, you should first run build-osx-libs.sh --force-rebuild, then apply the patch, then replace libraries/osx/iconv/include/iconv.h with the attached version, then run in the terminal (cd to your "libraries/osx/" folder)

rm ./gloox/.already-builtrm ./wxwidgets/.already-builtrm ./libxml2/.already-built

Then run my modified build-osx-bundle.sh script. It should only rebuild gloox, wxwidgets, and lixml2, and with hope it works. I can't guarantee though, as I might have missed/misplaced a step.

It's currently compiling against the 10.9 SDK with OSX 10.8 as min version.

OSXFix.patch

iconv.h.zip

[MathiasB]

When I have bit of time, I will give it a try!

[Echelon9]

It appears the change to libraries/osx/iconv/include/iconv.h was either overwritten or not applied correctly, as following through your steps I was getting this error on running your modified build-osx-bundle.sh.

$ ./build-osx-bundle.sh==== Building network (release) ======== Building mocks_real (release) ======== Building tinygettext (release) ======== Building lobby (release) ======== Building glooxwrapper (release) ======== Building simulation2 (release) ====iconv.cpp==== Building scriptinterface (release) ======== Building engine (release) ====../../../source/third_party/tinygettext/src/iconv.cpp:118:18: error: no matching function for call to 'libiconv'    size_t ret = tinygettext_iconv(cd, &inbuf, &inbytesleft, &outbuf, &outbytesleft);                 ^~~~~~~~~~~~~~~~~../../../source/third_party/tinygettext/include/tinygettext/iconv.hpp:40:35: note: expanded from macro 'tinygettext_iconv'#  define tinygettext_iconv       iconv                                  ^~~~~../../../libraries/osx/iconv/include/iconv.h:81:15: note: expanded from macro 'iconv'#define iconv libiconv              ^~~~~~~~../../../libraries/osx/iconv/include/iconv.h:83:15: note: candidate function not viable: no known conversion from 'const char **' to 'char **' for 2nd argumentextern size_t iconv (iconv_t cd,  char* * inbuf, size_t *inbytesleft, char* * outbuf, size_t *outbytesleft);              ^../../../libraries/osx/iconv/include/iconv.h:81:15: note: expanded from macro 'iconv'#define iconv libiconv              ^==== Building graphics (release) ====1 error generated.make[1]: *** [obj/tinygettext_Release/iconv.o] Error 1make: *** [tinygettext] Error 2make: *** Waiting for unfinished jobs....

When I open libraries/osx/iconv/include/iconv.h at line 83 it does show the included const.

...extern size_t iconv (iconv_t cd, const char* * inbuf, size_t *inbytesleft, char* * outbuf, size_t *outbytesleft);...

Perhaps your description of the step you took is not quite in the right order, or there was an intermediate rebuild somewhere?

Edited June 3, 2014 by Echelon9

[Echelon9]

Okay, so I tried just omitting the need to link that function call into iconv by changing to this line in source/thirdparty/tinygettext/src/iconv.cpp:

...     // Try to convert the text.    size_t ret = static_cast<size_t>(-1);   // HACKY HACKY    if (ret == static_cast<size_t>(-1))    {...

Unfortunately, while the compilation proceeded far past the previous error, it reported a subsequent error as so from source/ps/Filesystem.cpp when calling std::wstring GetWstringFromWpath(const fs::wpath& path):

==== Building pyrogenesis (release) ======== Building test (release) ====Linking pyrogenesisld: warning: directory not found for option '-L../../../libraries/source/cxxtest-4.3/lib'Linking testld: warning: directory not found for option '-L../../../libraries/source/cxxtest-4.3/lib'Undefined symbols for architecture x86_64:  "boost::filesystem::path_traits::convert(char const*, char const*, std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >&, std::codecvt<wchar_t, char, __mbstate_t> const&)", referenced from:      GetWstringFromWpath(boost::filesystem::path const&) in libengine.a(Filesystem.o)  "boost::filesystem::path_traits::convert(wchar_t const*, wchar_t const*, std::string&, std::codecvt<wchar_t, char, __mbstate_t> const&)", referenced from:      CTextureManagerImpl::GetConverterSettings(boost::shared_ptr<CTexture> const&) in libgraphics.a(TextureManager.o)      GetAIsHelper::Callback(Path const&, CFileInfo const&, unsigned long) in libsimulation2.a(ICmpAIManager.o)  "std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::find_last_of(char const*, unsigned long, unsigned long) const", referenced from:      boost::filesystem::path::m_parent_path_end() const in libboost_filesystem-mt.a(path.o)      boost::filesystem::path::filename() const in libboost_filesystem-mt.a(path.o)      boost::filesystem::path::m_path_iterator_decrement(boost::filesystem::path::iterator&) in libboost_filesystem-mt.a(path.o)  "std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::find_first_of(char const*, unsigned long, unsigned long) const", referenced from:      (anonymous namespace)::root_directory_start(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned long) in libboost_filesystem-mt.a(path.o)      boost::filesystem::path::filename() const in libboost_filesystem-mt.a(path.o)      boost::filesystem::path::m_path_iterator_increment(boost::filesystem::path::iterator&) in libboost_filesystem-mt.a(path.o)      boost::filesystem::path::m_path_iterator_decrement(boost::filesystem::path::iterator&) in libboost_filesystem-mt.a(path.o)  "std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::rfind(char, unsigned long) const", referenced from:      boost::filesystem::path::extension() const in libboost_filesystem-mt.a(path.o)      boost::filesystem::path::stem() const in libboost_filesystem-mt.a(path.o)  "std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::compare(char const*) const", referenced from:      boost::filesystem::path::begin() const in libboost_filesystem-mt.a(path.o)      boost::filesystem::path::m_path_iterator_decrement(boost::filesystem::path::iterator&) in libboost_filesystem-mt.a(path.o)  "std::__1::__basic_string_common<true>::__throw_length_error() const", referenced from:      boost::filesystem::path::path<char const*>(char const*, char const*) in libboost_filesystem-mt.a(path.o)  "std::__1::locale::use_facet(std::__1::locale::id&) const", referenced from:      boost::filesystem::path::imbue(std::__1::locale const&) in libboost_filesystem-mt.a(path.o)      __GLOBAL__I_a in libboost_filesystem-mt.a(path.o)  "std::__1::codecvt<wchar_t, char, __mbstate_t>::do_length(__mbstate_t&, char const*, char const*, unsigned long) const", referenced from:      vtable for boost::filesystem::detail::utf8_codecvt_facet in libboost_filesystem-mt.a(utf8_codecvt_facet.o)

[wraitii]

Odd. Your error message says

./../../libraries/osx/iconv/include/iconv.h:83:15: note: candidate function not viable: no known conversion from 'const char **' to 'char **' for 2nd argumentextern size_t iconv (iconv_t cd, char* * inbuf, size_t *inbytesleft, char* * outbuf, size_t *outbytesleft);

Which seems to be different from your actual file.

Edit: that second error seems like the old boost using libstd++. Check out this post. I wasn't sure if that was necessary, but it seems to be.

Edit: Hmmm, I got the same error you do, tried adding MishFTW's stuff back and still get it. This is weird.

[historic_bruno]

I can provide a patch for the UPnP memory corruption, I just can't verify it works on all cases until there's a reproducible means of building on Mac OS X so am keen to see wraitii's SVN commits

The patch should be attached to the ticket (#2338), it was already requested but there was no reply. We can review it and verify if it fixes a bug (even if it's not the worst or only bug, that's better than nothing).

What should we do for the OSX bundle? Release a somewhat patched A16 with no Atlas and some potential bugs?

We can work around both bugs if necessary: don't bundle Atlas (pass --disable-atlas to update-workspaces.sh, or leave out the libAtlasUI.dylib, so it doesn't simply crash) and don't use UPnP. I think the wxWidgets/Atlas bug is going to be harder to fix properly, unless they update 3.0 or we downgrade to 2.9.4. But doing nothing would be bad

[thamlett]

Something is better than nothing .

[Echelon9]

Edit: Hmmm, I got the same error you do, tried adding MishFTW's stuff back and still get it. This is weird.

Good to know it's not my dev setup -- there's an intermediate required step here that is yet to be really nailed down in the clear description of Mac building steps.

historic_bruno: I'll add a patch to the Trac ticket, but caveat that it's unable to have been reviewed so doesn't meet the normal standards I'd expect of myself.

Update: Two alternative patches attached to the ticket.

Edited June 4, 2014 by Echelon9

[wraitii]

Hmmm, I've got some weird news. I get the error you get, but only when trying on Xcode. If I run "build-osx-bundle", I get no such error. For some reason.

Try removing the boost and gloox folder and running build-osx-bundle, which should call build-osx-libs to rebuild those, and see what happens.

[Echelon9]

Hmmm, I've got some weird news. I get the error you get, but only when trying on Xcode. If I run "build-osx-bundle", I get no such error. For some reason.

Try removing the boost and gloox folder and running build-osx-bundle, which should call build-osx-libs to rebuild those, and see what happens.

Hi -

I tried as suggested and deleted the gloox/boost folders then re-ran ./build-osx-bundle.sh -j3. The error reported was:

Undefined symbols for architecture x86_64:  "std::runtime_error::runtime_error(char const*)", referenced from:      (anonymous namespace)::UnixAssertHandler::assert(char const*, char const*, int, char const*) in libnvcore.a(Debug.cpp.o)ld: symbol(s) not found for architecture x86_64clang: error: linker command failed with exit code 1 (use -v to see invocation)

I also tried the similar step of deleting gloox/boost and then running the ./build-osx-libs.sh script standalone. It reported the same error about missing symbols and use_facet() unfortunately.

So sadly both tests didn't progress the work to create a working bundle or loose binary on Mac OS X 10.9.

[wraitii]

Okay I have succeeded at compiling against latest SVN on my iMac which was fairly clean. The steps before SHOULD alone work, only you might have to re-copy iconv.h after the first build-osx-libs.sh and retry. Also you will most likely need to pass --disable-atlas to update-workspaces.sh since for me it crashed (wxXml… issue).

Edit: success on my iMac too. Both of my computers are on 10.9.3 and my system works for A16 and svn. I would like someone else to try and confirm, possibly with help over IRC.

[Echelon9]

With wraitii's assistance on IRC, I was able to build an 0ad.app for the first time on OS X Mavericks 10.9. Wraitii and I will separately write up and settle the building steps for this platform.

This pleasing development did allow me to test the UPnP-related double free patch (#2338) on the Mac platform.

Happily to say, the latest patch permitted no crashes when running the UPnP setup routines affected by the double-free bug. My testing confirmed the UPnP Internet Gateway Device was successfully contacted and the port forward registered (run one).

A second run which cached the location of the UPnP IGD also worked as expected without any crashes.

So this is confirmation from at least one testing platform (OSX 10.9) that the latest patch attached here does fix the memory corruption bug.

N.B this memory corruption will in fact affect each platform (Win, Linux, Mac) but it appears to only cause a hard crash for now on Mac.

Subtler bugs may be being happening on other platforms and it would be good to get it fixed up promptly. I've seen reported on other platforms of memory issues in and around this function.

[Yves]

How's the whole build process now? Do you still manually change iconv.h?

[Echelon9]

How's the whole build process now? Do you still manually change iconv.h?

It's still messy, and manual. Yes, there is a need to manually change iconv.h

wraitii and I will connect on IRC again this weekend to clarify and where possible coalesce the steps into a clean patch and command line steps. My goal here is to have single button / single command building returned to Mac OSX 10.9.

[wraitii]

Some update on the iconv.h issue: basically the configuration script does some magic around lines 14000-15000 that makes it decide against setting ICONV_CONST, which results in the iconv header being non-const, which is different from the win32 version we're shipping. I'm really not sure why, or how we can change this since I have the hardest time reading configure.

There are a few ways to fix this, we could:

-find some way to have configure pass ICONV_CONST — a proper fix, but can it be done?

-Patch libiconv after downloading to always act like we passed ICONV_CONST. This could be done in a variety of way: patching include/iconv.h after compilation, patching configure, …

-change our code in source/something so that it doesn't expect a const iconv.

I'm leaning towards patching so far.

The rest is basically already fixed, and I've mostly figured out why imo, so I'll try to get it clean and commit the code.

Edit: wait there's actually something in extern_libs for that which could fix the issue.

[wraitii]

I've got a commit ready to go that would fix compilation on Mavericks but might break it on earlier systems (10.7 downwards ), should I commit it?

Attached is the patch for latest SVN.

OSXFix.patch

[historic_bruno]

I've got a commit ready to go that would fix compilation on Mavericks but might break it on earlier systems (10.7 downwards ), should I commit it?

Attached is the patch for latest SVN.

Note that CPPFLAGS isn't C++ flags (as I once mistakenly assumed), it's actually the C preprocessor flags, so sometimes it does what you intended but not always. There's almost no reason to use it though. For C++ compiler flags, you actually want CXXFLAGS; most of build-osx-libs.sh has been changed to reflect that, but I guess NSPR and ICU were merged in from Git branches that had an older revision of build-osx-libs.sh. If you could make those changes, it would be nice and might even fix the need to flag libstdc++ manually there

Other than that, the patch seems to be reasonable changes. Don't worry about the pre-10.7 build, as: 1) it's too few users to spend much time on, 2) it's broken anyway due to a bug in Spidermonkey. The 10.7 build is one that IMO we should still care about, it should take much less effort for it to work compared with, say, 10.6. I don't see anything in the patch that would break it irreparably, so it's only a matter of someone with 10.7 testing and possibly fixing the build, if they want. Once Yosemite (10.10) is released, that should be even less of a concern.

You can try upgrading wxWidgets to 3.0.1 as was done in SVN, it should fix the Atlas crash on OS X.

Once you have a bundle you're happy with, I would check the binaries (pyrogenesis, libAtlasUI.dylib, libCollada.dylib) with "otool -L binaryname", to see if it's pulling in any unexpected libs. Someone with a "clean" system (no Xcode or command line tools) should test the bundle on Mountain Lion and Mavericks if those are what we'll be supporting, to catch any unpleasant surprises, but a lot can be inferred from otool.

Edited June 22, 2014 by historic_bruno
Misread the ICU part of the patch

[wraitii]

Those are actually changes I directly took from MishFTW before, I'll revert them as they should be unneeded.

I checked the upgrade to 3.0.1 on SVN and there's a compilation error, but it seems unrelated to 3.0.1 so I'll try against A16 and see if it fixes our error.

Echelon fixed the UpNp bug, if someone else confirms I could include that patch.

[wraitii]

Okay I've committed the fix, I'd like someone to try and run build-osx-bundle, it should work. I might have forgotten something since I haven't run a complete recompilation of all libraries (too long) but I don't think I did. Against SVN, this will work. A16 requires a few more stuffs, but I'll do that by hand for the bundle.

I'm now going to try and do a proper bundle.

Edit: OK I have a 600Mb RC. It's not perfect (atlas doesn't load up, so you end up being stuck in nothingness if you click on it) and I disabled uPnP but it should work otherwise.

[Stan`]

So those tickets are not finished then ? http://trac.wildfiregames.com/query?status=assigned&status=new&status=reopened&group=status&milestone=Alpha+16

[leper]

Why didn't you apply the patch from #2338?

[MathiasB]

I have tried wraitii's commit, and it worked on my Mac.

Thank!

[wraitii]

Okay, I have a 544Mb 7z archive and a 744 .dmg version of A16, with uPnP but not Atlas because I had some weird error and don't really want to bother. I intend to try and send the latter to Philip, asking for his feedback. Not sure how to upload it.

Anyways I have succeeded at compiling SVN pyrogenesis on xCode this morning, apparently to compile it properly I should pass "MIN_OSX_VERSION" as 10.9 too or the compiler defaults to libstdc++ or something. Anyways, it worked so I'll consider OSX compilation is mostly OK now.

On the other hand, Atlas compilation fails because TransformObject.cpp uses some XML stuff, in <wx/xml/xml.h>, but for some reason in the OSX config wxUse_XML is set to "false" so it can't work. Not sure how to fix that.

Edit: got it up and running on an http server if someone wants to try it.

[wraitii]

Mac users: http://releases.wildfiregames.com/rc/

[Thorfinn the Shallow Minded]

I am downloading right now. Awesome work.