Issue running on Linux

[Guest olelukoie]

After I've updated from svn a couple days ago I can not run the game. When I run pyrogenesis_dbg all I see is a black screen and I can not switch to the console I run the game from. The only thing I can do is to switch to VT (text mode virtual terminal) with Ctrl+Alt+F1 and kill the process. After killing it and returning in X session I see the following:

[olelukoie@olelukoie-linux system]$ ./pyrogenesis_dbg
TIMER| InitVfs: 53.9688 ms
TIMER| InitScripting: 74.476 ms
TIMER| CONFIG_Init: 26.7084 ms
TIMER| write_sys_info: 1.09154 ms
vfs.cpp(111): Function call failed: return value was -110100 (Unknown error (-110100, 0xFFFE51EC))
udbg_bfd_init: loading symbols from ./pyrogenesis_dbg.
Function call failed: return value was -110100 (Unknown error (-110100, 0xFFFE51EC))
Location: vfs.cpp:111 (LoadFile)

Call stack:

(0x00998cd2) ldbg.cpp:101 debug_DumpStack(wchar_t*, unsigned long, void*, char const*)
(0x00949789) debug.cpp:340 debug_BuildErrorMessage(wchar_t const*, char const*, int, char const*, void*, char const*, ErrorMessageMem*)
(0x00949c2c) debug.cpp:498 debug_DisplayError(wchar_t const*, unsigned long, void*, char const*, char const*, int, char const*, unsigned char*)
(0x00949ebb) debug.cpp:553 debug_OnError(long, unsigned char*, char const*, int, char const*)
(0x00961e21) vfs.cpp:111 VFS::LoadFile(boost::filesystem::basic_path<std::string, VfsPathTraits> const&, boost::shared_ptr<unsigned char>&, unsigned long&)
(0x0097a02a) unifont.cpp:90 UniFont_reload
(0x009739a7) h_mgr.cpp:506 call_init_and_reload
(0x00973b8e) h_mgr.cpp:545 alloc_new_handle
(0x00973db9) h_mgr.cpp:580 h_alloc(H_VTbl*, boost::filesystem::basic_path<std::string, VfsPathTraits> const&, unsigned long, ...)
(0x0097ab90) unifont.cpp:210 unifont_load(boost::filesystem::basic_path<std::string, VfsPathTraits> const&, unsigned long)
(0x007875c1) Font.cpp:51 CFont
(0x006896f4) GameSetup.cpp:728 InitPs
(0x0068a930) GameSetup.cpp:1126 Init(CmdLineArgs const&, int)
(0x005a64a9) main.cpp:399 RunGameOrAtlas
(0x005a6561) main.cpp:416 main
(0xe5ef691d) /lib64/libc.so.6:0 __libc_start_main

errno = 0 (?)
OS error = ?


(C)ontinue, (S)uppress, (B)reak, Launch (D)ebugger, or (E)xit?
Убито

[olelukoie@olelukoie-linux system]$

It seems that the game now tries to run in full-screen mode instead of in window mode (as it was before update) and doesn't allow to use Alt+Tab to switch to another application so it's now impossible to react on its console output.

[feneur]

Have you rebuilt the code since your last update?

[Guest olelukoie]

Have you rebuilt the code since your last update?

Yes, of course You can see a couple of my posts about compilation problems after this problem appeared (here and here). Now there are no compilation problems but the game still not run.

And anyway it's not good to run game in full-screen mode if it writes something in console. The default mode should be windowed - at least for debug build.

Edited August 3, 2009 by olelukoie

[aeon]

Same problem here ...

Here's the full how to kill the process for console newbs like me:

switch to text mode via CTRL+ALT+F1

login

run top and find process id (PID) of pyrogenesis_dbg

press q to guit top

run kill -9 PID

switch back to normal mode via CTRL+ALT+F7

Edited August 3, 2009 by aeon

[Ykkrosh]

Hmm, sounds like the new filesystem stuff doesn't quite work as planned . (I guess the reason it's running fullscreen is that it fails to load the config file which sets the windowed=true default, and then it carries on anyway until it dies later). I'll see if I can look into it this evening.

[janwas]

Yep, the error code means "directory not found" in very early init (the config file). Hopefully Philip will find the cause, unfortunately I can't debug on Linux. In the meantime, there remains an escape hatch in the form of the -writableRoot command line argument that should restore the old directory lookup scheme if it worked before.

[Ykkrosh]

(I've changed the code to default to windowed.)

I was just about to post "Unfortunately everything seems to work fine for me on Linux ", but then I tried the game once more and it failed, so I'll see if I can see why...

[Ykkrosh]

sys_get_executable_name doesn't work as expected. In particular, if I run "./pyrogenesis_dbg" then it returns "./pyrogenesis_dbg", and GameSetup.cpp ends up thinking the root is "" (when it should be equivalent to ".."). (But if I run "gdb ./pyrogenesis_dbg" then it returns "/home/philip/0ad/trunk/binaries/system/pyrogenesis_dbg", which is probably why I didn't see the problem at first.)

I notice this code has relevant comments around its use of dladdr. I don't see any way to avoid having to do lots of ugly stuff like they do (resolving against cwd if it's relative, searching PATH if it's just the filename).

[Guest olelukoie]

I notice this code has relevant comments around its use of dladdr. I don't see any way to avoid having to do lots of ugly stuff like they do (resolving against cwd if it's relative, searching PATH if it's just the filename).

Mmm... It seems to me that you have chosen a difficult way of loading the game data. Really there are just two absolutely standard ways of placing and loading (searching) application data:

- on Windows systems all data files are in relative paths and the main executable is in the root of this hierarchy. You don't need to check anything.

- on *NIX the executable is in /usr/bin (or in /usr/local/bin) and the data is in /usr/share/<program_name> (or in /usr/local/share/<program_name>). In package managers this path is coded as 'datadir'. Once again you don't need to check the paths.

Sometimes it's also possible to see windows-way on *NIX in some cross-platform apps (for ex. Intel VTune or Matlab) when app is (usually) installed in /opt/<program_name> and all data are in relative paths. But again you don't need to test the paths as you know what you've done and where data files should be.

As far as you don't use dynamic libs for the game (except the standard ones) and provide monolithic executable without any binary "plugins" you don't need to search and load libs and it simplifies things even more. Just try to load any one data-file from relative path and if it fails (open or fopen or whatever you use returns "no such file or directory" ) try to load it from /usr/share or /usr/local/share. All you may want to check is if the executable is in /usr/bin or in /usr/local/bin but it's not necessary. More accurately you can check the presence of that single data file in one of this three locations without trying to open it but I don't like this way as it is just wasting function calls - checking possible error codes after opening file is necessary anyway and they can be successfully used to check the file presence.

So I don't understand your problem...

[janwas]

- on Windows systems all data files are in relative paths and the main executable is in the root of this hierarchy. You don't need to check anything.

That's currently true for many systems, but newer versions of Windows and least-permission user accounts forbid writing to some directories, especially under Program Files. Hence, we should also separate data and executables on Windows.

- on *NIX the executable is in /usr/bin (or in /usr/local/bin) and the data is in /usr/share/<program_name> (or in /usr/local/share/<program_name>). In package managers this path is coded as 'datadir'. Once again you don't need to check the paths.

hm, I don't know about acceptable practice on Linux. The XDG layout scheme (what we're now using) was recently suggested. Can someone else with more experience comment on whether fixed paths would be acceptable? (Simplicity is a pretty compelling argument )

However, XDG isn't that much more complicated. I think what's "killing" us is correct relative search on Linux, as Philip points out. What if we only enabled the option for that on Windows (where GetModuleFilename is easy+reliable)?

[Guest olelukoie]

- on Windows systems all data files are in relative paths and the main executable is in the root of this hierarchy. You don't need to check anything.

That's currently true for many systems, but newer versions of Windows and least-permission user accounts forbid writing to some directories, especially under Program Files. Hence, we should also separate data and executables on Windows.

Well, lets look at two my favorite strategy (both turn-based) windows games: GalCiv2 and CivIV

They both save all the per-user config data, mods, saves etc into %PROFILE%\My Documents\My Games\<game_name> folder. They never save anything in Program Files. I think it's the best practice for windows.

As for *NIX - all software should write all its per-user config data etc in ~/.<program_name> or in ~/.config/<program_name> folder. Once again you should not write anything in /usr or /opt subfolders - and the user doesn't have enough permissions for this.

- on *NIX the executable is in /usr/bin (or in /usr/local/bin) and the data is in /usr/share/<program_name> (or in /usr/local/share/<program_name>). In package managers this path is coded as 'datadir'. Once again you don't need to check the paths.

hm, I don't know about acceptable practice on Linux. The XDG layout scheme (what we're now using) was recently suggested. Can someone else with more experience comment on whether fixed paths would be acceptable? (Simplicity is a pretty compelling argument )

However, XDG isn't that much more complicated. I think what's "killing" us is correct relative search on Linux, as Philip points out. What if we only enabled the option for that on Windows (where GetModuleFilename is easy+reliable)?

I know nothing about MacOSX, but on *NIX systems I've seen and used so far (Linux, FreeBSD, QNX) the common practice is to put static data files (that do not need to be changed per system or per user) in /usr/share or - sometimes - /usr/lib (/usr/lib64 on x86_64), per-system config files in /etc/<program_name> and per-user config files in ~/.<program_name> or ~/.config/<program_name>. In some cases programs use /usr/local instead of /usr but it doesn't matter. On other systems there can be different paths (/usr/etc instead of /etc for ex.) but I'm not sure you plan to support all this systems.

And as a universal solution you can use cmake to configure sources to use correct paths and it will allow you to get rid of all this path checking - paths would be fully static but configurable before compilation.

For example on my mandriva installation there is a rpm macro for cmake:

%cmake \
	%setup_compile_flags \
	mkdir -p build \
	cd build \
	%define enable_debug 1 \
	%{?_enable_debug: %{expand: %%global enable_debug 1}} \
	%__cmake .. \\\
		-DCMAKE_INSTALL_PREFIX:PATH=%{_prefix} \\\
		-DCMAKE_INSTALL_LIBDIR:PATH=%{_libdir} \\\
		-DINCLUDE_INSTALL_DIR:PATH=%{_includedir} \\\
		-DLIB_INSTALL_DIR:PATH=%{_libdir} \\\
		-DSYSCONF_INSTALL_DIR:PATH=%{_sysconfdir} \\\
		-DSHARE_INSTALL_PREFIX:PATH=%{_datadir} \\\
		-DCMAKE_SKIP_RPATH=ON \\\
		-DCMAKE_BUILD_TYPE=%{_cmake_debug} \\\
%if "%{?_lib}" == "lib64" \
		%{?_cmake_lib_suffix64} \\\
%endif \
		%{?_cmake_skip_rpath: -DCMAKE_SKIP_RPATH:BOOL=ON} \\\
		%{?_cmake_verbose: -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON} \\\
		-DBUILD_SHARED_LIBS:BOOL=ON \\\
		-DBUILD_STATIC_LIBS:BOOL=OFF \\\
		-DCMAKE_MODULE_LINKER_FLAGS="%(echo %ldflags|sed -e 's#-Wl,--no-undefined##')"

As you can see there are all the dirs configured.

[janwas]

Well, lets look at two my favorite strategy (both turn-based) windows games: GalCiv2 and CivIV

They both save all the per-user config data, mods, saves etc into %PROFILE%\My Documents\My Games\<game_name> folder. They never save anything in Program Files. I think it's the best practice for windows.

Right, that's pretty much our plan on Windows, too (Just substituting CSIDL_APPDATA for My Documents)

On other systems there can be different paths (/usr/etc instead of /etc for ex.) but I'm not sure you plan to support all this systems.

That's the advantage of XDG - we write a bit more code once, then users/systems can easily change environmental variables to support their particular path setup.

And as a universal solution you can use cmake to configure sources to use correct paths and it will allow you to get rid of all this path checking - paths would be fully static but configurable before compilation.

hm, that's an idea, but I'm not sure we'd always want to require configuration+compilation. Wouldn't it make sense to provide binary packages (debs or rpms) eventually?

[Ykkrosh]

on *NIX the executable is in /usr/bin (or in /usr/local/bin) and the data is in /usr/share/<program_name> (or in /usr/local/share/<program_name>)

That's pretty much what I suggested here - seems like the proper way to do installation of the game. But there's still the issue that during development we want to run the game directly from the SVN directory with no installation, so it still needs to be able to cope with relative paths as well as installed absolute paths.

As far as you don't use dynamic libs for the game (except the standard ones) and provide monolithic executable without any binary "plugins" you don't need to search and load libs and it simplifies things even more

We do have a couple of dynamic libraries (the Atlas editor UI, and the Collada parser) that are loaded on-demand at run-time, and I'd be happier not to change it to be monolithic, but I think it shouldn't be much of a problem anyway.

I'm not sure we'd always want to require configuration+compilation. Wouldn't it make sense to provide binary packages (debs or rpms) eventually?

Binary packages are configured and compiled for a particular Linux distribution with a particular filesystem layout. With most programs you can change the default directories by passing arguments to the ./configure script before you run make, and the paths get compiled into the executable. You can't change the paths without recompiling, but you never need to change the paths so that's okay. I don't see any reason not to do the same - so we should let people specify the paths (for system data, not per-user data) when calling update-workspaces.sh, and default to the common /usr/... paths, and then the game can use those string constants when loading data and libraries (when it's run in the run-from-system-installation mode, not the run-from-SVN-checkout-directory mode).

[Guest olelukoie]

Today I've remembered one more *NIX-specific situation: the program is installed in windows-way single directory tree with executable in the top of the hierarchy (or in 'bin' subdir) and data files are in subdirs but the executable is not in the $PATH and there is a symbolic (or even hard) link to it that resides in $PATH. This approach is used by several programs I've seen (Matlab for ex). The good solution for this situation is to use startup script that sets up several environment vars and prepares command-line parameters (preconfigured during compilation or setup process) and then runs the real executable that reads these vars and/or params. This approach is sometimes used even when installing in /usr/bin - just to be sure that all the data would be found.

Once again it doesn't require to check the data files directories at runtime.

And I think that supporting both ways of installing (like in windows and in /usr) on *NIX-systems would be the best solution because it would allow people to install the game even without root privileges - by simply downloading binary distribution and unpacking it directly in their own home dirs (boss will never know ). I use this approach to install fresh versions of Eclipse and Scilab as the versions in official repos of my distribution are updated too late for me. Another example is Qt Creator.

OK, I've run out of thoughts on this topic Thank you for your patience reading them.

[Ykkrosh]

sys_get_executable_name doesn't work as expected.

This should be fixed in r7081. (Also I've extended CxxTest so you can run "./test_dbg -test TestSysdep" and "./test_dbg -test TestSysdep::test_sys_get_executable_name" etc to run individual suites/tests, and added build system support for mock objects.)

(It's not fixed very nicely, because in general it seems to be impossible (or at least extremely hard) to find an absolute path corresponding to a relative path - e.g. if I symlink /tmp/a to /tmp/b/c, then /tmp/a/../c/ is equivalent to the absolute path /tmp/b/c/, so you have to mess around with the filesystem just to normalise the path. It would be much easier if we didn't actually try to determine an absolute path for the executable, and were happy with unnormalised relative-to-cwd paths, but I don't know if we can cope with that.)

[janwas]

Binary packages are configured and compiled for a particular Linux distribution with a particular filesystem layout. With most programs you can change the default directories by passing arguments to the ./configure script before you run make, and the paths get compiled into the executable. You can't change the paths without recompiling, but you never need to change the paths so that's okay. I don't see any reason not to do the same - so we should let people specify the paths (for system data, not per-user data) when calling update-workspaces.sh, and default to the common /usr/... paths, and then the game can use those string constants when loading data and libraries (when it's run in the run-from-system-installation mode, not the run-from-SVN-checkout-directory mode).

Ah, ok; that'd just require simple changes to the Paths class (i.e. hardcoding the paths).

And I think that supporting both ways of installing (like in windows and in /usr) on *NIX-systems would be the best solution because it would allow people to install the game even without root privileges - by simply downloading binary distribution and unpacking it directly in their own home dirs (boss will never know ).

That's a good argument. Only supporting the single-directory mode on Windows would've been a copout, to be honest.

OK, I've run out of thoughts on this topic Thank you for your patience reading them.

Thank you for your comments, they are helpful!

This should be fixed in r7081. (Also I've extended CxxTest so you can run "./test_dbg -test TestSysdep" and "./test_dbg -test TestSysdep::test_sys_get_executable_name" etc to run individual suites/tests, and added build system support for mock objects.)

Awesome!

(It's not fixed very nicely, because in general it seems to be impossible (or at least extremely hard) to find an absolute path corresponding to a relative path - e.g. if I symlink /tmp/a to /tmp/b/c, then /tmp/a/../c/ is equivalent to the absolute path /tmp/b/c/, so you have to mess around with the filesystem just to normalise the path. It would be much easier if we didn't actually try to determine an absolute path for the executable, and were happy with unnormalised relative-to-cwd paths, but I don't know if we can cope with that.)

I agree that being 100% robust there is unrealistic (symlinks are tough, but also due to unpredictable contents of argv[0] and getcwd/initial_path - they may be total garbage). However, the source copies the old/deprecated normalization stuff from boost, and I'm wondering why realpath wasn't used - that looks to be more robust and should at least correctly handle symlinks.

One more comment: relying on cwd doesn't work on Windows because it in turn is not always correctly set *sigh* (I forget in what cases, but launching from a shell and explorer and debugger behaved differently).

[Ykkrosh]

the source copies the old/deprecated normalization stuff from boost, and I'm wondering why realpath wasn't used

Because I didn't know it existed. Fixed the code to use it instead of yucky Boost code now

[janwas]

Looks good