Jump to content

0ad SVN - "ERROR: SpiderMonkey build failed"


zyli
 Share

Recommended Posts

Heh, looks like @Itms was a few of hours ahead of me in investigating!

Quote

17:07 <@Itms> I'm really not sure what Debian is doing that fumbles the output of distutils

At a guess, this: https://salsa.debian.org/cpython-team/python3/-/blob/python3.10/debian/patches/distutils-install-layout.diff

(There is also https://salsa.debian.org/cpython-team/python3/-/blob/python3.10/debian/patches/sysconfig-debian-schemes.diff which modifies sysconfig, which appears to be where DEB_PYTHON_INSTALL_LAYOUT is read.)

It adds two new "Install Schemes": unix_local and deb_system.

The default "Install Scheme" for Debian(-derived) distros is unix_local; which has the "purelib" path template of '{base}/local/lib/python{py_version_short}/dist-packages' which matches @zyli's initial report above.

With the DEB_PYTHON_INSTALL_LAYOUT environment variable set to "deb", the "Install Scheme" used changes to deb_system; which has the "purelib" path template of '{base}/lib/python3/dist-packages', which matches the last reported attempt.

Both patches have code that detects virtual environments (and uses the posix_prefix install scheme instead in that case) - however this detection only functions when it is run inside a virtual environment, and this point of the spidermonkey build process is run outside it.

That said, I may have a solution:installscheme.patch


@hyperion Out of curiosity, are setuptools/virtualenv/distutils installed globally in that image? (Or need installing globally as part of the process?)

 

Link to comment
Share on other sites

@hyperion
========

Unfortunately, the links don't work:

cd /home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/_virtualenvs/common/lib/

state before:
------------

ls -l

drwxr-xr-x 3 domowy domowy 4096 01-15 14:45 python3.10


ls -l python3.10/

drwxr-xr-x 10 domowy domowy 4096 01-15 14:45 site-packages


after creating symlinks
-----------------------

ls -l

lrwxrwxrwx 1 domowy domowy  111 01-15 21:33 python3 -> /home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/_virtualenvs/common/lib/python3.10
drwxr-xr-x 3 domowy domowy 4096 01-15 21:33 python3.10

ls -l python3.10/

lrwxrwxrwx  1 domowy domowy  125 01-15 21:32 dist-packages -> /home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/_virtualenvs/common/lib/python3.10/site-packages
drwxr-xr-x 10 domowy domowy 4096 01-15 14:45 site-packages

Compilation

cd ~

cd svn

svn cleanup 0ad

svn up 0ad
[...]
Uaktualniono do wersji 27448.

cd 0ad/build/workspaces/

./clean-workspaces.sh
[...]

./update-workspaces.sh 
[...]

FileNotFoundError: [Errno 2] No such file or directory: '/home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/_virtualenvs/common/local/lib/python3.10/dist-packages'
ERROR: SpiderMonkey build failed

Strange, this time it looks in "local/lib"

But those directories are not there, and the links are gone.

(Nie ma takiego pliku ani katalogu = There is no such file or directory, nie ma dostępu do = no access to)

ls -l /home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/_virtualenvs/common/local/lib/python3.10/dist-packages
ls: nie ma dostępu do '/home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/_virtualenvs/common/local/lib/python3.10/dist-packages': Nie ma takiego pliku ani katalogu

ls -l /home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/_virtualenvs/common/local/lib/python3.10
ls: nie ma dostępu do '/home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/_virtualenvs/common/local/lib/python3.10': Nie ma takiego pliku ani katalogu

ls -l /home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/_virtualenvs/common/local/lib/
ls: nie ma dostępu do '/home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/_virtualenvs/common/local/lib/': Nie ma takiego pliku ani katalogu

ls -l /home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/_virtualenvs/common/
drwxr-xr-x 2 domowy domowy 4096 01-15 21:43 bin
drwxr-xr-x 3 domowy domowy 4096 01-15 21:43 lib
-rw-r--r-- 1 domowy domowy    9 01-15 21:43 python_exe.txt
-rw-r--r-- 1 domowy domowy  202 01-15 21:43 pyvenv.cfg


ls -l /home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/_virtualenvs/common/lib/
drwxr-xr-x 3 domowy domowy 4096 01-15 21:43 python3.10

ls -l /home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/_virtualenvs/common/lib/python3.10/
drwxr-xr-x 10 domowy domowy 4096 01-15 21:43 site-packages

 

Link to comment
Share on other sites

I have now made a local directory with the appropriate links:

cd /home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/_virtualenvs/common/

ls -l
drwxr-xr-x 2 domowy domowy 4096 01-15 21:43 bin
drwxr-xr-x 3 domowy domowy 4096 01-15 22:07 lib
drwxr-xr-x 2 domowy domowy 4096 01-15 22:08 local
-rw-r--r-- 1 domowy domowy    9 01-15 21:43 python_exe.txt
-rw-r--r-- 1 domowy domowy  202 01-15 21:43 pyvenv.cfg

ls -l local
lrwxrwxrwx 1 domowy domowy 100 01-15 22:08 lib -> /home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/_virtualenvs/common/lib

ls -l local/lib/python3/
lrwxrwxrwx  1 domowy domowy  125 01-15 22:06 dist-packages -> /home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/_virtualenvs/common/lib/python3.10/site-packages
drwxr-xr-x 10 domowy domowy 4096 01-15 21:43 site-packages

All for nothing, because the compilation of these links removes.

FileNotFoundError: [Errno 2] No such file or directory: '/home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/_virtualenvs/common/local/lib/python3.10/dist-packages'
ERROR: SpiderMonkey build failed

ls -l /home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/_virtualenvs/common/
razem 16
drwxr-xr-x 2 domowy domowy 4096 01-15 22:12 bin
drwxr-xr-x 3 domowy domowy 4096 01-15 22:12 lib
-rw-r--r-- 1 domowy domowy    9 01-15 22:12 python_exe.txt
-rw-r--r-- 1 domowy domowy  202 01-15 22:12 pyvenv.cfg


Postscriptum.

Creating the local directory and the appropriate links IMMEDIATELY BEFORE issuing the ./update-workspaces.sh command has the same effect - the links disappear.

Edited by zyli
Link to comment
Share on other sites

@zyli

thanks for continuing to try.

For all we know now the issue is due to install scheme and once fixed it should work for you and other Debian users. First try the patch from @s0600204 , it's inlined in the text and so you might have missed it. His patch looks like a clean approach to fix the issue. Apply it on a clean checkout and try to build 0ad.

On the off chance it's not a complete fix you could try to do the symlink trick, if that works we are sure what the issue is and only need to find all places to change.

For the symlink trick approch:

  • export DEB_PYTHON_INSTALL_LAYOUT='deb' (needs to be done every time you open a new terminal)
  • run "./update-wokspace.sh"
  • the build fails then create the symlinks as suggested, don't clean the build or anything the like.
  • run "REBUILD=false ./update-workspace.sh"

If this succeeds we are sure to have found the cause and that there is nothing more to it.

 

@Freagarach you might want to try as well.

Link to comment
Share on other sites

For the symlink trick approch:

svn cleanup 0ad

svn up 0ad
[...]
Uaktualniono do wersji 27455.

cd 0ad/build/workspaces/

./clean-workspaces.sh

export DEB_PYTHON_INSTALL_LAYOUT='deb'

echo $DEB_PYTHON_INSTALL_LAYOUT
deb

./update-workspaces.sh
[...]
FileNotFoundError: [Errno 2] No such file or directory: '/home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/_virtualenvs/common/lib/python3/dist-packages'
ERROR: SpiderMonkey build failed

# Symlinks done

ls -l /home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/_virtualenvs/common/
drwxr-xr-x 2 domowy domowy 4096 01-16 17:46 bin
drwxr-xr-x 3 domowy domowy 4096 01-16 17:48 lib
-rw-r--r-- 1 domowy domowy    9 01-16 17:46 python_exe.txt
-rw-r--r-- 1 domowy domowy  202 01-16 17:46 pyvenv.cfg

ls -l /home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/_virtualenvs/common/lib
lrwxrwxrwx 1 domowy domowy  111 01-16 17:48 python3 -> /home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/_virtualenvs/common/lib/python3.10
drwxr-xr-x 3 domowy domowy 4096 01-16 17:49 python3.10

ls -l /home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/_virtualenvs/common/lib/python3/

lrwxrwxrwx  1 domowy domowy  125 01-16 17:48 dist-packages -> /home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/_virtualenvs/common/lib/python3.10/site-packages
drwxr-xr-x 10 domowy domowy 4096 01-16 17:46 site-packages

# Again

REBUILD=false ./update-workspaces.sh 
Updating bundled third-party dependencies...

FCollada is already up to date.

Building SpiderMonkey...

SpiderMonkey build options: --disable-tests
           --disable-jemalloc
           --disable-js-shell
           --without-intl-api
           --enable-shared-js
           --disable-jitspew         
Re-executing in the virtualenv
checking for vcs source checkout... no
checking for a shell... /usr/bin/sh
checking for host system type... x86_64-pc-linux-gnu
checking for target system type... x86_64-pc-linux-gnu
checking for Python 3... /home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/_virtualenvs/common/bin/python (3.10.9)
Traceback (most recent call last):
  File "/home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/../js/src/../../configure.py", line 226, in <module>
    sys.exit(main(sys.argv))
  File "/home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/../js/src/../../configure.py", line 50, in main
    sandbox.run(os.path.join(os.path.dirname(__file__), "moz.configure"))
  File "/home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/python/mozbuild/mozbuild/configure/__init__.py", line 558, in run
    func(*args)
  File "/home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/python/mozbuild/mozbuild/configure/__init__.py", line 609, in _value_for
    return self._value_for_depends(obj)
  File "/home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/python/mozbuild/mozbuild/util.py", line 1050, in method_call
    cache[args] = self.func(instance, *args)
  File "/home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/python/mozbuild/mozbuild/configure/__init__.py", line 618, in _value_for_depends
    value = obj.result()
  File "/home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/python/mozbuild/mozbuild/util.py", line 1050, in method_call
    cache[args] = self.func(instance, *args)
  File "/home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/python/mozbuild/mozbuild/configure/__init__.py", line 157, in result
    return self._func(*resolved_args)
  File "/home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/python/mozbuild/mozbuild/configure/__init__.py", line 1286, in wrapped
    self._apply_imports(func, glob)
  File "/home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/python/mozbuild/mozbuild/configure/__init__.py", line 953, in _apply_imports
    self._get_one_import(_from, _import, _as, glob)
  File "/home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/python/mozbuild/mozbuild/configure/__init__.py", line 1084, in _get_one_import
    exec_(import_line, {}, glob)
  File "/home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/python/mozbuild/mozbuild/util.py", line 56, in exec_
    exec(object, globals, locals)
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'mozversioncontrol'
ERROR: SpiderMonkey build failed
    

echo $DEB_PYTHON_INSTALL_LAYOUT
deb

ls -l /home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/_virtualenvs/common/lib/python3/
lrwxrwxrwx  1 domowy domowy  125 01-16 17:48 dist-packages -> /home/domowy/svn/0ad/libraries/source/spidermonkey/mozjs-91.13.1/build-debug/_virtualenvs/common/lib/python3.10/site-packages
drwxr-xr-x 11 domowy domowy 4096 01-16 17:51 site-packages

 

Edited by zyli
Link to comment
Share on other sites

1 hour ago, hyperion said:

I'll setup a vm with the installer you linked to and try to reproduce the issue later.

Same, can build pyrogenesis just fine using a Debian installed using the iso here https://cdimage.debian.org/cdimage/weekly-builds/amd64/iso-dvd/

 

*suspects some migration wrt python in the past left some breakage behind ...

Link to comment
Share on other sites

Maybe ask on the Debain forums if my guess is reasonable and if you can fix it manually. Looking at the standalone build of mozjs91 at  https://salsa.debian.org/gnome-team/mozjs/-/tree/debian/91/master I can't see an indication of your bug either.

 

Sadly, I can't really help much with debugging if I can't reproduce the issue, just grasping at straws.

Link to comment
Share on other sites

Hi here! Sorry I didn't post here, @s0600204 thanks for the parallel investigation :D

So basically I imported a fix from SM that fixes the extra "local" in paths. It is not as clean as the fix in your patch, but it is what was included in SM, so better follow them.

However, the remaining issue is that on Debian, distutils proposes a path ending with "dist-packages" while it should be "site-packages" where things are actually installed. This is the part I don't really understand.

If you have any idea @s0600204 I'd be interested, else I'll ask a Debian maintainer. In the worst case, it is easy to hack around this by changing "dist" by "site", the same way we remove "local/" from the path.

Link to comment
Share on other sites

Ok, I've now created a Bookworm VM myself, and my current findings:

  1. I can duplicate the build failure if python3-setuptools is installed from the Debian repos. Uninstalling it clears the issue. (@hyperion this is probably what you're missing.)
  2. My earlier patch doesn't work.
  3. The reason it doesn't work is because of how setuptools resolves the installation scheme - ask it to use "posix_prefix", and setuptools asks sysconfig what the preferred scheme is for "prefix", whereupon it gets the response "posix_local". So setuptools uses that scheme instead.
  4. That reason sysconfig returns that value is due to the modifications Debian has made to that module. (It's also where the detection of virtual environments takes place. And fails because this bit of the process isn't being run inside a virtualenv.)
  5. setuptools attempts to use "posix_prefix" on posix-y systems anyway under certain circumstances (which spidermonkey's build process meets), so my patch - even if it was correct - is redundant.

@Itms I may have an alternate solution - albeit one that will conflict with the patch you've imported from SM - but I need to sleep right now so I shall have to get back to you with it.

Edited by s0600204
Add links, so its more obvious which bits of code I'm referring to.
  • Like 1
Link to comment
Share on other sites

1 hour ago, s0600204 said:

I may have an alternate solution - albeit one that will conflict with the patch you've imported from SM - but I need to sleep right now so I shall have to get back to you with it.

Great, I'll wait for it and we can decide between it, and the hack from SM and my hack idea :D

Link to comment
Share on other sites

2 hours ago, s0600204 said:

I can duplicate the build failure if python3-setuptools is installed from the Debian repos. Uninstalling it clears the issue. (@hyperion this is probably what you're missing.)

Indeed, now I can reproduce.

With python3-setuptools installed the following allows me to build 0ad in Debian

SETUPTOOLS_USE_DISTUTILS=stdlib ./update-workspaces.sh

@zyli phew, no reinstall needed

 

---

Tried Fedora 37 and can build just fine. Wonder what I need to break that one (no, not setuptools)!

  • Like 1
Link to comment
Share on other sites

Thank you so much for help.

SETUPTOOLS_USE_DISTUTILS=stdlib ./update-workspaces.sh

 It works.

cd /home/domowy/svn/0ad/binaries/system/

ls -l pyrogenesis
-rwxr-xr-x 1 domowy domowy 362809392 01-17 12:10 pyrogenesis

ls -l test
-rwxr-xr-x 1 domowy domowy 414498464 01-17 12:10 test

./test 
Running cxxtest tests (397 tests).............................................................................................................................................................................................................................................................................................................................................................................................................OK!

./pyrogenesis 
TIMER| InitVfs: 143.948 ms
FILES| Main log written to '/home/domowy/.config/0ad/logs/mainlog.html'
FILES| Interesting log written to '/home/domowy/.config/0ad/logs/interestinglog.html'
TIMER| CONFIG_Init: 46.7647 ms
Sound: AlcInit success, using OpenAL Soft
FILES| UserReport written to '/home/domowy/.config/0ad/logs/userreport_hwdetect.txt'
TIMER| RunHardwareDetection: 23.2006 ms
FILES| Hardware details written to '/home/domowy/.config/0ad/logs/system_info.txt'
TIMER| write_sys_info: 16.2649 ms
TIMER| InitRenderer: 412.1 ms
TIMER| ps_console: 15.9626 ms
TIMER| ps_lang_hotkeys: 3.58588 ms
TIMER| common/modern/setup.xml: 35.1188 ms
TIMER| common/modern/styles.xml: 1.09618 ms
TIMER| common/modern/sprites.xml: 2.99531 ms
TIMER| common/global.xml: 2.2558 ms
TIMER| common/setup.xml: 1.0498 ms
TIMER| common/sprites.xml: 1.16201 ms
TIMER| common/styles.xml: 348.015 us
TIMER| pregame/backgrounds/: 5.8753 ms
TIMER| pregame/sprites.xml: 596.217 us
TIMER| pregame/styles.xml: 324.337 us
TIMER| pregame/mainmenu.xml: 849.965 ms
TIMER| common/modern/setup.xml: 91.746 us
TIMER| common/modern/styles.xml: 47.682 us
TIMER| common/modern/sprites.xml: 487.593 us
TIMER| common/global.xml: 108.182 us
TIMER| common/sprites.xml: 171.8 us
TIMER| splashscreen/setup.xml: 851.171 us
TIMER| splashscreen/splashscreen.xml: 19.6777 ms
TIMER| common/modern/setup.xml: 323.669 us
TIMER| common/modern/styles.xml: 279.265 us
TIMER| common/modern/sprites.xml: 2.77173 ms
TIMER| msgbox/msgbox.xml: 52.7774 ms
TIMER| shutdown Renderer: 2.53558 ms
TIMER| shutdown SDL: 8.06283 ms
TIMER| shutdown UserReporter: 82.117 us
TIMER| shutdown ConfigDB: 30.779 us
TIMER| resource modules: 176.347 ms
TIMER TOTALS (7 clients)
-----------------------------------------------------
  tc_pool_alloc: 0 c (0x)
  tc_png_decode: 15.8559 Mc (2x)
  tc_dds_transform: 85.754 kc (14x)
  tc_transform: 1330.4 kc (6x)
  tc_plain_transform: 1204.4 kc (6x)
  tc_ShaderValidation: 35.4034 Mc (1x)
  xml_validation: 2056.95 kc (6x)
-----------------------------------------------------
TIMER| shutdown misc: 893.141 us

0AD starts up fine.
Thanks again.
Regards.

Edited by zyli
Link to comment
Share on other sites

Unfortunately SETUPTOOLS_USE_DISTUTILS=stdlib is not a long-term solution, as the python standard lib's distutils is deprecated as of python 3.10 and due to be removed as part of python 3.12 (scheduled to be released October 2023). If spidermonkey intends to keep using distutils past that point, they are going to have to use setuptools' vendored version. (And either spidermonkey, Debian, setuptools - or a combination of all three - are going to have to come up with an official solution to this.)

@Itms with the caveat that as of writing this I haven't completed a full build of spidermonkey on my VM - so there may be failure later in the build - here's the patch:FixInstallScheme.diff Needless to say it's not suitable for applying on Windows systems (as that would, at a guess, use the "nt" scheme).

  • Like 2
Link to comment
Share on other sites

I have read the patch note https://trac.wildfiregames.com/changeset/27457/
"Update clean-workspaces.sh after SpiderMonkey's upgrade to 78.6.0 and 91.13.1."

I have a little question now.
The above compilation worked for me because:
1. I used SETUPTOOLS_USE_DISTUTILS=stdlib ./update-workspaces.sh,
2. Is it because amendment 27457 came out?

P.S.
In the build above, the "svn up 0ad" command showed version 27457.

Link to comment
Share on other sites

1 hour ago, s0600204 said:

@Itms with the caveat that as of writing this I haven't completed a full build of spidermonkey on my VM - so there may be failure later in the build - here's the patch:FixInstallScheme.diff Needless to say it's not suitable for applying on Windows systems (as that would, at a guess, use the "nt" scheme).

Tested on a Debian Bookworm/Sid with Python 3.11, the full mozjs build went smoothly.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

×
×
  • Create New...