Android port

[Guest afeder]

Yep, that seems right:

afeder@ubuntu:~/android/toolchain/sysroot/usr/include/libxml2$ export PKG_CONFIG_SYSROOT_DIR=$SYSROOT 
afeder@ubuntu:~/android/toolchain/sysroot/usr/include/libxml2$ pkg-config libxml-2.0 --cflags
-I/home/afeder/android/toolchain/sysroot/usr/include/libxml2

Sysroot is a GCC option:

--sysroot=dir

Use dir as the logical root directory for headers and libraries. For example, if the compiler would normally search for headers in /usr/include and libraries in /usr/lib, it will instead searchdir/usr/include and dir/usr/lib.

It is used when cross-compiling to make a local directory the root of a 'virtual' filesystem that corresponds to the filesystem of the target device.

However, some packages (e.g. cURL) do indeed take --with-sysroot=dir as an option of configure.

Edited February 1, 2012 by afeder

[Guest afeder]

Yep, that fixed the problem. With PKG_CONFIG_SYSROOT_DIR set, I get no error for libxml2 when building the game.

All external dependencies fixed.

[Guest afeder]

Behold the beautiful error output.

[historic_bruno]

Have you updated the source code recently? ogl_shader was removed in 10984 and there's been a few more recent changes that might help. That still leaves a lot of errors

[Sonarpulse]

Good luck! Hopefully the fixes to those errors will help upstream too.

[Ykkrosh]

About SDL: since a few days, the Hg version of SDL 1.3 is now called SDL2 instead, and has removed compatibility with the SDL1.2 API. I guess we need to either use 1.2, or a slightly older Hg version of 1.3, or update our code to be compatible with SDL2 (probably better in the long term but I haven't tested it to see how much we'd have to change).

Incidentally, don't remember if this has been mentioned before, but it's impossible (or at least discouraged and unportable and seemingly undocumented) to use GLES with pure C++ - you have to use some Java to set up EGL windows before the C++ can render into it. SDL provides the necessary Java code, and SDL's android-project can build it: when you run "ndk-build" it builds libSDL2.so and libmain.so, and when you run "ant debug" it compiles the Java code and packages everything into an .apk file. (Then you do an "adb push whatever.apk /data/app" to install it onto the device). I imagine it would be easier to write a custom script to perform those build steps, instead of trying to integrate our existing code into ndk-build, but I don't know exactly what's required.

Unfortunately the Android emulator doesn't seem to do GLES2 - I can compile and run a simple SDL test application but it just prints "E/libEGL ( 504): called unimplemented OpenGL ES API" to logcat. At least it's possible to test that the application starts up without linker errors or Java exceptions etc, but for any graphical output it'll need to be run on real hardware.

[Guest afeder]

Have you updated the source code recently? ogl_shader was removed in 10984 and there's been a few more recent changes that might help. That still leaves a lot of errors

They are numerous in quantity, but similar in quality. With a good plan, it shouldn't be impossible to get rid of them.

Updated output from latest svn

Incidentally, don't remember if this has been mentioned before, but it's impossible (or at least discouraged and unportable and seemingly undocumented) to use GLES with pure C++ - you have to use some Java to set up EGL windows before the C++ can render into it. SDL provides the necessary Java code, and SDL's android-project can build it: when you run "ndk-build" it builds libSDL2.so and libmain.so, and when you run "ant debug" it compiles the Java code and packages everything into an .apk file. (Then you do an "adb push whatever.apk /data/app" to install it onto the device). I imagine it would be easier to write a custom script to perform those build steps, instead of trying to integrate our existing code into ndk-build, but I don't know exactly what's required.

There's two isolated steps to SDL's approach, as far as I can tell: 1) you build the native shared library component with ndk-build and 2) you build the Java and package it with the native shared library with Ant. So, I imagine we can simply use the toolchain you suggested to do step 1 and use Ant as normal for step 2. Ant "won't care" how the native shared library came about.

[Ykkrosh]

With a good plan, it shouldn't be impossible to get rid of them.

I think a possibly sensibly plan would be to focus on getting just the GUI to work for now (plus the build system and anything else required), and #ifdef out any problematic non-GUI code, so we can have a runnable starting point before porting all the other graphics code. (I was attempting to start with the GUI here, getting rid of some of the fixed-function code). That will probably involve more changes to the CShaderProgram API so that it can use glVertexAttribPointer instead of the not-in-GLES2 glColorPointer etc, and the rest is probably just tedious rearrangement of code to use the new APIs.

There's two isolated steps to SDL's approach, as far as I can tell: 1) you build the native shared library component with ndk-build and 2) you build the Java and package it with the native shared library with Ant. So, I imagine we can simply use the toolchain you suggested to do step 1 and use Ant as normal for step 2. Ant "won't care" how the native shared library came about.

I know very little about ndk-build and ant, but that sounds reasonable... If I understand correctly we could probably use SDL's normal ndk-build step, with its jni/src/Android.mk's YourSourceHere.c just being a simple file that defines SDL_main() to load libpyrogenesis.so and call a startup function inside that shared library, and then we can build libpyrogenesis.so independently using the game's build system, and then get ant to work as with normal SDL but copy libpyrogenesis.so into the package too, or something like that.

Incidentally, about the libraries bundled with the game: there's probably no point porting NVTT to Android, since it's used for S3TC texture compression and most mobile GPUs don't support S3TC; I think we should disable that code and load uncompressed textures instead. (Then, if not everybody has got fed up with Android, experiment with ETC1 to save VRAM, and/or JPEG to save download size, etc.)

[Guest afeder]

I know very little about ndk-build and ant, but that sounds reasonable... If I understand correctly we could probably use SDL's normal ndk-build step, with its jni/src/Android.mk's YourSourceHere.c just being a simple file that defines SDL_main() to load libpyrogenesis.so and call a startup function inside that shared library, and then we can build libpyrogenesis.so independently using the game's build system, and then get ant to work as with normal SDL but copy libpyrogenesis.so into the package too, or something like that.

Yes, that might work. There is a related section in the NDK documentation: docs/PREBUILT.html.

[Guest afeder]

I think a possibly sensibly plan would be to focus on getting just the GUI to work for now (plus the build system and anything else required), and #ifdef out any problematic non-GUI code, so we can have a runnable starting point before porting all the other graphics code. (I was attempting to start with the GUI here, getting rid of some of the fixed-function code). That will probably involve more changes to the CShaderProgram API so that it can use glVertexAttribPointer instead of the not-in-GLES2 glColorPointer etc, and the rest is probably just tedious rearrangement of code to use the new APIs.

So, the main obstacle I am facing now is SpiderMonkey. ENet already builds and I can #ifdef out NVTT and FCollada. But I'm not sure how to build SpiderMonkey and can't seem to get any response from the official mailing lists. It has a --with-android-toolchain configure option, I'm just not sure what other options to combine it with. Any suggestions appreciated.

[Ykkrosh]

Have you seen this? I guess something similar should work for us, though I don't know whether the 1.8.5 release is compatible enough or whether we should use a newer Hg revision. With 1.8.5 we don't need --enable-threadsafe or --with-nspr-* (and don't need to compile NSPR); don't know about newer revisions. Also I presume we can target android-10 (2.3.3+), not android-8 (2.2+), but I guess that won't matter much.

[Pedro Falcão]

Age of Empires 3 Mobile:

[Guest afeder]

Have you seen this? I guess something similar should work for us, though I don't know whether the 1.8.5 release is compatible enough or whether we should use a newer Hg revision. With 1.8.5 we don't need --enable-threadsafe or --with-nspr-* (and don't need to compile NSPR); don't know about newer revisions. Also I presume we can target android-10 (2.3.3+), not android-8 (2.2+), but I guess that won't matter much.

There seem to be some kind of incompatibility issue with android-ndk-r5-crystax-2 (it searches the wrong paths). Someone from Mozilla suggested that I build Fennec and use the libraries that it generates, but that results in a similar error.

Would it be feasible to port the parts of the code using wstrings to some other Android-compatible representation? The wstrings are the only thing preventing me using the latest official NDK, which SpiderMonkey seems set up for.

Edit: I'm going to try if I can use another toolchain in parallel specifically to build Spidermonkey. Theoretically, it should not be a problem since the NDK API is intended to be stable across releases.

Edited February 5, 2012 by afeder

[Ykkrosh]

Would it be feasible to port the parts of the code using wstrings to some other Android-compatible representation? The wstrings are the only thing preventing me using the latest official NDK, which SpiderMonkey seems set up for.

If the only thing missing from the official NDK is the definition of the std::wstring type, then we could easily define the type ourselves (exactly like source/ps/utf16string.h except with wchar_t). But it sounds like the NDK is probably missing most of the functions for operating on wchar_t too, in which case that won't be sufficient. Changing the game to not use wstring/wchar_t should be hypothetically feasible (it may be best to store everything as UTF-8 encoded strings internally, and just convert to/from UTF-16 when interacting with Win32 APIs) but probably a huge amount of work - I expect it'd be much easier to sort out the build system issues instead of changing all the code.

[Guest afeder]

With 1.8.5 we don't need --enable-threadsafe or --with-nspr-* (and don't need to compile NSPR);

How do you prevent it from building with NSPR? update-workspaces.sh seems to run configure with --with-system-nspr, but that just causes it to link in the NSPR in my absolute /usr/lib. If configure it without any nspr options at all, I get this:

/home/afeder/android/js-1.8.5/js/src/jscpucfg.cpp:48:21: fatal error: prtypes.h: No such file or directory

Edited February 6, 2012 by afeder

[Ykkrosh]

update-workspaces.sh seems to run configure with --with-system-nspr

Where does it do that? libraries/spidermonkey/build.sh only mentions --with-system-nspr in a commented-out line.

/home/afeder/android/js-1.8.5/js/src/jscpucfg.cpp:48:21: fatal error: prtypes.h: No such file or directory

That's in a "#if defined(CROSS_COMPILE) && !defined(FORCE_BIG_ENDIAN) && !defined(FORCE_LITTLE_ENDIAN)" section. I presume CROSS_COMPILE is being defined when building for Android, triggering the error. It looks like defining FORCE_LITTLE_ENDIAN (simplest hack is probably to stick it at the top of jscpucfg.cpp) should avoid that dependency on NSPR, hopefully.

[Ykkrosh]

Unfortunately the Android emulator doesn't seem to do GLES2 - I can compile and run a simple SDL test application but it just prints "E/libEGL ( 504): called unimplemented OpenGL ES API" to logcat.

My test code didn't work on real hardware either since I had the SDL 2.0 init code wrong and it was defaulting to GLES1, but it worked after I fixed that. (Then the emulator fails on application startup because it can't find a usable surface type). The Java code in android-project needs modification to support useful things like 24-bit RGB output (instead of default 16-bit) and antialiasing (4xAA is "nearly free on Mali hardware (around 2% performance drop)"), but that's easy enough.

[Guest afeder]

Where does it do that? libraries/spidermonkey/build.sh only mentions --with-system-nspr in a commented-out line.

Indeed, I read that line incorrectly.

That's in a "#if defined(CROSS_COMPILE) && !defined(FORCE_BIG_ENDIAN) && !defined(FORCE_LITTLE_ENDIAN)" section. I presume CROSS_COMPILE is being defined when building for Android, triggering the error. It looks like defining FORCE_LITTLE_ENDIAN (simplest hack is probably to stick it at the top of jscpucfg.cpp) should avoid that dependency on NSPR, hopefully.

Yes, that worked. Now Spidermonkey builds too

The full diff of changes/hacks I had to make is here.

[Guest afeder]

Any idea why this fails:

afeder@ubuntu:~/android/toolchain/sysroot$ ls $PKG_CONFIG_SYSROOT_DIR/usr/local/lib/pkgconfig/mozjs185.pc
/home/afeder/android/toolchain/sysroot/usr/local/lib/pkgconfig/mozjs185.pc

afeder@ubuntu:~/android/toolchain/sysroot$ pkg-config mozjs185 --cflags
Package mozjs185 was not found in the pkg-config search path.
Perhaps you should add the directory containing `mozjs185.pc'
to the PKG_CONFIG_PATH environment variable
No package 'mozjs185' found

According to the man page of pkg-config, /usr/local/lib/pkgconfig is supposed to be one of the default search paths.

Edit: I guess pkg-config doesn't actually search the sysroot, it just prepends it to the absolute paths

Edit2: I fixed this by using the PKG_CONFIG_LIBDIR variable.

Edited February 11, 2012 by afeder

[Guest afeder]

These are the files causing errors in the latest error output:

/source/simulation2/components/CCmpRallyPointRenderer.cpp
/source/simulation2/components/CCmpTerritoryManager.cpp
/source/ps/Profiler2GPU.cpp
/source/ps/ProfileViewer.cpp
/source/ps/CLogger.cpp
/source/ps/Util.cpp
/source/ps/CConsole.cpp
/source/ps/GameSetup/HWDetect.cpp
/source/ps/GameSetup/GameSetup.cpp
/source/maths/Brush.cpp
/source/maths/BoundingBoxAligned.cpp
/source/graphics/Material.cpp
/source/graphics/ShaderTechnique.cpp
/source/graphics/Camera.cpp
/source/graphics/ParticleEmitter.cpp
/source/graphics/ObjectEntry.cpp
/source/graphics/TerritoryTexture.cpp
/source/graphics/LOSTexture.cpp
/source/graphics/Sprite.cpp
/source/graphics/ShaderProgramFFP.cpp
/source/graphics/CinemaTrack.cpp
/source/renderer/DecalRData.cpp
/source/renderer/ParticleRenderer.cpp
/source/renderer/HWLightingModelRenderer.cpp
/source/renderer/TerrainRenderer.cpp
/source/renderer/ShadowMap.cpp
/source/renderer/PlayerRenderer.cpp
/source/renderer/ModelRenderer.cpp
/source/renderer/RenderModifiers.cpp
/source/renderer/PatchRData.cpp
/source/renderer/OverlayRenderer.cpp
/source/renderer/SkyManager.cpp
/source/renderer/InstancingModelRenderer.cpp
/source/renderer/Renderer.cpp
/source/renderer/FixedFunctionModelRenderer.cpp
/source/renderer/TerrainOverlay.cpp
/source/renderer/VertexBuffer.cpp
/source/renderer/TransparencyRenderer.cpp
/source/tools/atlas/GameInterface/ActorViewer.cpp
/source/tools/atlas/GameInterface/Handlers/TerrainHandlers.cpp
/source/gui/CGUI.cpp
/source/gui/GUIRenderer.cpp
/source/gui/MiniMap.cpp
/source/lib/res/graphics/unifont.cpp
/source/lib/res/graphics/ogl_tex.cpp

Which of these would characterize as being "non-GUI"? And how do you suggest I remove them from the build system?

[Ykkrosh]

r11039/r11040 should have fixed the build errors in source/gui/. (At least they work for me with Mesa's GL ES on Linux). I still need to add GLSL versions of the GUI shaders, but that probably should be enough to get the GUI displaying. I think the rest of the GL code can get either fixed easily (translating to the new shader API) or temporarily #if'd out, but that'll depend on the precise details of each case.

[Guest afeder]

Great. But how do I practically go about preventing the Premake system from compiling those that I #ifdef out? Where are the input files for the compiler generally listed?

[Ykkrosh]

It will still need to compile all of the source files, since it needs most of the definitions of classes and methods etc even if they're not working yet, else it'll fail later with linker errors. Generally just small sections of code inside the methods should be disabled, like this.

The source files aren't actually listed anywhere - build/premake/premake4.lua lists directories, and it searches for all *.cpp inside them.

[Guest afeder]

Got it.

[Guest afeder]

I'll be trying to commit any hacks I make here as I go along: https://github.com/afeder/0ad-android