Build environment and deployment on the Mac

[MathiasB]

I'm stuck at the CXXTests... (after having solved SDL and ICU)

The library that needs to be include isn't there:

ld: warning: directory not found for option '-L../../../libraries/source/cxxtest-4.3/lib'

How to build this library?

http://pastebin.com/vC9jabPv

Edited May 28, 2014 by MathiasB

[wraitii]

Same error as the pastbin above, though I don't think the warning quoted is related. I tried forcing libiconv to be 64 bit (apparently by default it's only 32?) but that didn't seem to change much. Maybe I did it wrong. I seem to recall this issue having been fixed before though.

I applied Philip's changes form a later revision and run into no issues with libc++ ad OSX 10.9.3 otherwise.

[MathiasB]

Same error as the pastbin above, though I don't think the warning quoted is related.

The warning is indeed not related to the linking error...

[historic_bruno]

The cxxtest warning always occurred and is completely harmless.

Same error as the pastbin above, though I don't think the warning quoted is related. I tried forcing libiconv to be 64 bit (apparently by default it's only 32?) but that didn't seem to change much. Maybe I did it wrong. I seem to recall this issue having been fixed before though.

I don't think arch has anything to do with it, the linker error is slightly ambiguous in that respect. If it can't find the symbols for x86_64, that doesn't mean they exist for i386. That can happen, but it's more likely that some libs simply haven't been linked. You could check the arch on compiled output with the file tool, and possibly nm, too (I did this in the past until I better understood the linker).

[MathiasB]

I have already checked the lib file with nm, and there is no _iconv in there. (pastebin.com/88NqXGyA)

I have tried to build the library myself, but there are compile and linking errors in de package i downloaded from Apple.

Edited May 29, 2014 by MathiasB

[historic_bruno]

For one thing, extern_libs4.lua doesn't have any OS X-specific handling for iconv, which looks problematic. I think when I tested, it at least had line 373 uncommented (probably commented out to not break Linux builds), but it needs more. We build iconv as part of build-osx-libs.sh, so extern_libs4.lua should be calling add_default_include_paths and add_default_lib_paths on OS X, similarly to on Windows (I don't know if the definitions are also needed). See how most other libs are being handled in extern_libs4.lua.

Getting that straightened out should fix the iconv missing symbols, what the linker probably means is we didn't even bother to link iconv on OS X (it just can't be so explicit, since it doesn't know where the symbols might come from) That could be confirmed by doing "make verbose=1" or something and checking the linker flags when it fails - if there's no -L/path/to/iconv and -liconv, we know the problem.

[wraitii]

I tried linking iconv. Seems to have made a difference, but it still fails. Error on compilation:

../../../source/third_party/tinygettext/src/iconv.cpp:118:18: error: no matching      function for call to 'iconv'    size_t ret = tinygettext_iconv(cd, &inbuf, &inbytesleft, &outbuf, &o...                 ^~~~~~~~~~~~~~~~~../../../source/third_party/tinygettext/include/tinygettext/iconv.hpp:40:35: note:       expanded from macro 'tinygettext_iconv'#  define tinygettext_iconv       iconv                                  ^~~~~/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/include/iconv.h:69:8: note:       candidate function not viable: no known conversion from 'const char **' to      'char **' for 2nd argumentsize_t iconv (iconv_t /*cd*/,       ^In file included from ../../../source/lobby/XmppClient.cpp:19:In file included from ../../../source/lobby/XmppClient.h:27:In file included from ../../../source/scriptinterface/ScriptVal.h:21:In file included from ../../../source/scriptinterface/ScriptTypes.h:72:In file included from ../../../libraries/source/spidermonkey/include-unix-release/jsapi.h:28:In file included from ../../../libraries/source/spidermonkey/include-unix-release/js/HashTable.h:14:

[historic_bruno]

I tried linking iconv.

Linking, or fixing all the missing parts in extern_libs4.lua? It shouldn't be using the iconv from the SDK, but the one built in build-osx-libs. Seems like it doesn't have the paths set up properly (I don't know if the char** vs const char** issue would still exist if it built the correct version, but that is where the HAVE_ICONV_CONST define is relevant) as explained above.

[wraitii]

Hmm, for some reason the second part of my message was erased. I changed extern_libs4.lua to the following and had the above error.

iconv = {compile_settings = function()if os.is("windows") or os.is("macosx") thenadd_default_include_paths("iconv")defines { "HAVE_ICONV_CONST" }defines { "LIBICONV_STATIC" }endend,link_settings = function()if os.is("windows") or os.is("macosx") thenadd_default_lib_paths("iconv")endadd_default_links({win_names  = { "iconv" },-- TODO: glibc provides symbols for this, so we should only include that (and depend on libiconv) on non-glibc unixosx_names = { "iconv" },dbg_suffix = "",})end,},

[MathiasB]

Hmm, for some reason the second part of my message was erased. I changed extern_libs4.lua to the following and had the above error.

iconv = {compile_settings = function()if os.is("windows") or os.is("macosx") thenadd_default_include_paths("iconv")defines { "HAVE_ICONV_CONST" }defines { "LIBICONV_STATIC" }endend,link_settings = function()if os.is("windows") or os.is("macosx") thenadd_default_lib_paths("iconv")endadd_default_links({win_names  = { "iconv" },-- TODO: glibc provides symbols for this, so we should only include that (and depend on libiconv) on non-glibc unixosx_names = { "iconv" },dbg_suffix = "",})end,},

I've changed that part in the extern_libs4.lua file.

This didn't give me any compile errors on gettext, but I still got the same linking errors while trying to link pyrogenesis.

[wraitii]

I changed the OSX folder to be the same as on Windows (iconv and not "libiconv"). The linking should now work. I still get errors though.

../../../source/third_party/tinygettext/src/iconv.cpp:118:18: error: no matching

function for call to 'libiconv'

size_t ret = tinygettext_iconv(cd, &inbuf, &inbytesleft, &outbuf, &o...

^~~~~~~~~~~~~~~~~

../../../source/third_party/tinygettext/include/tinygettext/iconv.hpp:40:35: note:

expanded from macro 'tinygettext_iconv'

# define tinygettext_iconv iconv

^~~~~

../../../libraries/osx/iconv/include/iconv.h:81:15: note: expanded from macro

'iconv'

#define iconv libiconv

Here's the output of make verbose=1 for that file:

The "-I../../../source" bit seems suspicious. Maybe I still have a linking issue.

clang++ -arch x86_64 -Iobj/tinygettext_Release -include obj/tinygettext_Release/precompiled.h -MMD -MP -DNDEBUG -DCONFIG_FINAL=1 -DLIB_STATIC_LINK -DBUNDLE_IDENTIFIER=com.wildfiregames.0ad -DUSING_PCH -DHAVE_ICONV_CONST -DLIBICONV_STATIC -I../../../source/pch/tinygettext -I../../../source -I../../../libraries/osx/iconv/include -I../../../source/third_party/tinygettext/include/tinygettext -g -Wall -O3 -Wno-switch -Wno-reorder -Wno-invalid-offsetof -Wextra -Wno-missing-field-initializers -Wunused-parameter -Wredundant-decls -Wnon-virtual-dtor -Wundef -fstack-protector-all -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstrict-aliasing -fno-omit-frame-pointer -fpch-preprocess -msse -msse2 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk -mmacosx-version-min=10.8 -fvisibility=hidden -isystem../../../libraries/osx/boost/include -MF obj/tinygettext_Release/iconv.d -MT "obj/tinygettext_Release/iconv.o" -o "obj/tinygettext_Release/iconv.o" -c "../../../source/third_party/tinygettext/src/iconv.cpp"

[MathiasB]

I changed the OSX folder to be the same as on Windows (iconv and not "libiconv"). The linking should now work. I still get errors though.

../../../source/third_party/tinygettext/src/iconv.cpp:118:18: error: no matching

function for call to 'libiconv'

size_t ret = tinygettext_iconv(cd, &inbuf, &inbytesleft, &outbuf, &o...

^~~~~~~~~~~~~~~~~

../../../source/third_party/tinygettext/include/tinygettext/iconv.hpp:40:35: note:

expanded from macro 'tinygettext_iconv'

# define tinygettext_iconv iconv

^~~~~

../../../libraries/osx/iconv/include/iconv.h:81:15: note: expanded from macro

'iconv'

#define iconv libiconv

Here's the output of make verbose=1 for that file:

The "-I../../../source" bit seems suspicious. Maybe I still have a linking issue.

clang++ -arch x86_64 -Iobj/tinygettext_Release -include obj/tinygettext_Release/precompiled.h -MMD -MP -DNDEBUG -DCONFIG_FINAL=1 -DLIB_STATIC_LINK -DBUNDLE_IDENTIFIER=com.wildfiregames.0ad -DUSING_PCH -DHAVE_ICONV_CONST -DLIBICONV_STATIC -I../../../source/pch/tinygettext -I../../../source -I../../../libraries/osx/iconv/include -I../../../source/third_party/tinygettext/include/tinygettext -g -Wall -O3 -Wno-switch -Wno-reorder -Wno-invalid-offsetof -Wextra -Wno-missing-field-initializers -Wunused-parameter -Wredundant-decls -Wnon-virtual-dtor -Wundef -fstack-protector-all -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstrict-aliasing -fno-omit-frame-pointer -fpch-preprocess -msse -msse2 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk -mmacosx-version-min=10.8 -fvisibility=hidden -isystem../../../libraries/osx/boost/include -MF obj/tinygettext_Release/iconv.d -MT "obj/tinygettext_Release/iconv.o" -o "obj/tinygettext_Release/iconv.o" -c "../../../source/third_party/tinygettext/src/iconv.cpp"

How do you know the linking should work now? You still got compile errors on gettext...

Your compile errors seem very strange though. In the args, you can see that it includes the iconv folder, but apparently it doesn't include the needed header files.

I don't have your compile errors, which is even more strange. gettext builds here without any issue (apart from linking the whole thing for pyrogenesis )

I will try your fix in the lua file.

[Yves]

../../../source/third_party/tinygettext/src/iconv.cpp:118:18: error: no matching

function for call to 'libiconv'

size_t ret = tinygettext_iconv(cd, &inbuf, &inbytesleft, &outbuf, &o...

^~~~~~~~~~~~~~~~~

../../../source/third_party/tinygettext/include/tinygettext/iconv.hpp:40:35: note:

expanded from macro 'tinygettext_iconv'

# define tinygettext_iconv iconv

^~~~~

../../../libraries/osx/iconv/include/iconv.h:81:15: note: expanded from macro

'iconv'

#define iconv libiconv

Well, basically it's still the same problem.

That "thing" must be called iconv or libiconv but not iconv here and libiconv there.

EDIT:

Or it could also be a mismatch with const/non const... but I think the compiler would list availaible candiates with different argument types then.

[wraitii]

Okay, I found the fix for that particular issue.

To recap what I did so far:

Changed extern_libs4.lua to this:

iconv = {compile_settings = function()if os.is("windows") or os.is("macosx") thenadd_default_include_paths("iconv")defines { "HAVE_ICONV_CONST" }defines { "LIBICONV_STATIC" }endend,link_settings = function()if os.is("windows") or os.is("macosx") thenadd_default_lib_paths("iconv")endadd_default_links({win_names  = { "iconv" },-- TODO: glibc provides symbols for this, so we should only include that (and depend on libiconv) on non-glibc unixosx_names = { "iconv" },dbg_suffix = "",})end,},

Changed build-osx-libs to this (lines 211/212) and also added MishFTW's changes but I don't think those are really useful with Philip's fixes:

mkdir -p iconvpushd iconv > /dev/null

Applied Philip's change in r15231 in order to compile against libc++ and not pass any further flags.

Recompiled against 10.8 SDK on OSX 10.9.3

Then I got the above error about const. This is because there is a difference between the osx libs and the win32 libs, apparently.
Windows version line 92:
extern LIBICONV_DLL_EXPORTED size_t iconv (iconv_t cd, const char* * inbuf, size_t *inbytesleft, char* * outbuf, size_t *outbytesleft)
OSX version line 92:
extern LIBICONV_DLL_EXPORTED size_t iconv (iconv_t cd, char* * inbuf, size_t *inbytesleft, char* * outbuf, size_t *outbytesleft)

You might notice that there's a "const" missing in the OSX version. If you put it back, compilation works. So that works like you'd expect.

Now I get a different linking error but I'll update shortly.

Edit: new error, seems like Collada is linking improperly

Undefined symbols for architecture x86_64:  "_libiconv", referenced from:      _xmlIconvWrapper in libxml2.a(encoding.o)  "_libiconv_close", referenced from:      _xmlFindCharEncodingHandler in libxml2.a(encoding.o)      _xmlCharEncCloseFunc in libxml2.a(encoding.o)  "_libiconv_open", referenced from:      _xmlFindCharEncodingHandler in libxml2.a(encoding.o)ld: symbol(s) not found for architecture x86_64clang: error: linker command failed with exit code 1 (use -v to see invocation)make[1]: *** [../../../binaries/system/libCollada.dylib] Error 1make: *** [Collada] Error 2make: *** Waiting for unfinished jobs....

[MathiasB]

Well, before we missed _iconv, now it's _libiconv.

[wraitii]

Full error, I think:

ICmpVision.cppclang++ -arch x86_64 -Iobj/simulation2_Release -include obj/simulation2_Release/precompiled.h -MMD -MP -DNDEBUG -DCONFIG_FINAL=1 -DLIB_STATIC_LINK -DBUNDLE_IDENTIFIER=com.wildfiregames.0ad -DUSING_PCH -I../../../source/pch/simulation2 -I../../../source -I../../../libraries/source/spidermonkey/include-unix-release -g -Wall -O3 -Wno-switch -Wno-reorder -Wno-invalid-offsetof -Wextra -Wno-missing-field-initializers -Wunused-parameter -Wredundant-decls -Wnon-virtual-dtor -Wundef -fstack-protector-all -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstrict-aliasing -fno-omit-frame-pointer -fpch-preprocess -msse -msse2 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk -mmacosx-version-min=10.8 -fvisibility=hidden -isystem../../../libraries/osx/boost/include -MF obj/simulation2_Release/ICmpVision.d -MT "obj/simulation2_Release/ICmpVision.o" -o "obj/simulation2_Release/ICmpVision.o" -c "../../../source/simulation2/components/ICmpVision.cpp"Undefined symbols for architecture x86_64:  "_libiconv", referenced from:      _xmlIconvWrapper in libxml2.a(encoding.o)  "_libiconv_close", referenced from:      _xmlFindCharEncodingHandler in libxml2.a(encoding.o)      _xmlCharEncCloseFunc in libxml2.a(encoding.o)  "_libiconv_open", referenced from:      _xmlFindCharEncodingHandler in libxml2.a(encoding.o)In file included from ../../../source/simulation2/components/ICmpVision.cpp:20:In file included from ../../../source/simulation2/components/ICmpVision.h:21:In file included from ../../../source/simulation2/system/Interface.h:21:In file included from ../../../source/simulation2/system/IComponent.h:22:In file included from ../../../source/simulation2/system/Message.h:21:In file included from ../../../source/scriptinterface/ScriptTypes.h:71:In file included from ../../../libraries/source/spidermonkey/include-unix-release/jspubtd.h:15:In file included from ../../../libraries/source/spidermonkey/include-unix-release/jstypes.h:25:../../../libraries/source/spidermonkey/include-unix-release/mozilla/Util.h:277:31: warning:       deleted function definitions are a C++11 extension [-Wc++11-extensions]    Maybe(const Maybe& other) MOZ_DELETE;                              ^../../../libraries/source/spidermonkey/include-unix-release/mozilla/Attributes.h:197:35: note:       expanded from macro 'MOZ_DELETE'#  define MOZ_DELETE            = delete                                  ^In file included from ../../../source/simulation2/components/ICmpVision.cpp:20:In file included from ../../../source/simulation2/components/ICmpVision.h:21:In file included from ../../../source/simulation2/system/Interface.h:21:In file included from ../../../source/simulation2/system/IComponent.h:22:In file included from ../../../source/simulation2/system/Message.h:21:In file included from ../../../source/scriptinterface/ScriptTypes.h:71:In file included from ../../../libraries/source/spidermonkey/include-unix-release/jspubtd.h:15:In file included from ../../../libraries/source/spidermonkey/include-unix-release/jstypes.h:25:../../../libraries/source/spidermonkey/include-unix-release/mozilla/Util.h:278:48: warning:       deleted function definitions are a C++11 extension [-Wc++11-extensions]    const Maybe& operator=(const Maybe& other) MOZ_DELETE;                                               ^../../../libraries/source/spidermonkey/include-unix-release/mozilla/Attributes.h:197:35: note:       expanded from macro 'MOZ_DELETE'#  define MOZ_DELETE            = delete                                  ^ld: symbol(s) not found for architecture x86_64clang: error: linker command failed with exit code 1 (use -v to see invocation)make[1]: *** [../../../binaries/system/libCollada.dylib] Error 1make: *** [Collada] Error 2make: *** Waiting for unfinished jobs.... 

ANyone else reading this as "we don't link libiconv"? Is that an issue?

[historic_bruno]

Check that libxml2 built with the correct iconv, in particular you said you renamed the iconv folder, so maybe xml2-config is still pointing to the old path (see the FCollada section of build-osx-libs.sh and also libxml2 in extern_libs4.lua). You can try running libraries/osx/libxml2/bin/xml2-config --libs (or --cflags) to see if the paths are correct.

[wraitii]

Success!

So what I did above is sufficient, I had merely forgotten to recompile the libraries with the new iconv. After using force-rebuild (specifically, only wxwidgets and libxml2 ought to be needed, I also recompiled gloox).

I've got an A16 bundle which should be OK, except it's got too many languages.

Now on to catch up to SVN.

[Echelon9]

Great - would love to see the delta to SVN closed and clean steps to follow your build

[historic_bruno]

Now on to catch up to SVN.

Great Also, I think we can release a 10.8 or 10.9 only bundle, since it is already late. Unless you have the time and interest, 10.7 compatibility may have to wait (and pre-10.7 is practically a lost cause at this point). It seems like Mavericks adoption has been pretty good, it's a free upgrade after all...

[historic_bruno]

Also keep in mind there are still a few OS X release blockers: Atlas crashes due to a wxWidgets bug and multiplayer is likely to crash due to a bug in UPnP support.

[thamlett]

Oh wow, you guys sure did some progress.

@historic_bruno: after thinking about it, I shouldn't have tried compiling on 10.6 (silly me). I doubt that anybody who can run the game can't run Mountain Lion or Mavericks. There are a few exceptions, of course (including the macbook i was using yesterday), but those issues can be resolved separately.

[Echelon9]

Also keep in mind there are still a few OS X release blockers: ... multiplayer is likely to crash due to a bug in UPnP support.

I can provide a patch for the UPnP memory corruption, I just can't verify it works on all cases until there's a reproducible means of building on Mac OS X so am keen to see wraitii's SVN commits. To note the memory corruption is not Mac specific, it's just be most demonstrably seen on that platform.

[wraitii]

What should we do for the OSX bundle? Release a somewhat patched A16 with no Atlas and some potential bugs?

[sanderd17]

It would be nice to test the UPnP issue at least. If that makes every multiplayer game unplayable. Either UPnP should be disabled on the mac build (IIRC, there's a flag for it), or it should be fixed.

If there are no other gameplay problems for Mac, I think it should just get released.