Building a 64-bit pyrogenesis (and deps) for Windows

[hyperion]

Was wondering why my wdbg_stub didn't get picked up, turns out due to a bug I introduced into premake5.lua the windows-only cpp files got omitted during compilation

So, some time later that stuff is fixed and builds as well. Came across a curious thing static_assert((void*)intptr_t(-1)) in wmman, gcc barfs on the reinterpret_cast here, but even if not I don't see the purpose of this assert. May someone enlighten me?

 

So, more serious business, I started to look into how to properly implement the changes into premake5.lua. Some googling makes me feel I'm among the first to ever cross-compile with premake. Many issues to address.

• premake-core has no icc support (--cc=icc is invalid) (optional, highly desirable)
• premake-core does not respect RC respectively RESCOMP (required, some symlink with PATH hacking might work)

So how to handle this? Patching the copy in tree seems beyond ugly.

• windres doesn't support -isystem, only -I for includes

So no sysincludedirs permitted. Don't use them? Build rc as a standalone static project? Does VS have something like whole-archive? Simply drop the error_dialog?

 

Below a patch to set cc and arch usable for cross-compilation, anyone sees a setup this might break?

 

premake-cc-and-arch.patchFetching info...

[vladislavbelov]

  On 06/08/2021 at 3:19 PM, hyperion said:

So, some time later that stuff is fixed and builds as well. Came across a curious thing static_assert((void*)intptr_t(-1)) in wmman, gcc barfs on the reinterpret_cast here, but even if not I don't see the purpose of this assert. May someone enlighten me?

Expand  

Allocation in case of failing returns MAP_FAILED, but also there is a code:

*pp = 0;
// make sure *pp won't be misinterpreted as an error
cassert(MAP_FAILED);

It just does check that MAP_FAILED != 0 to not misinterpret *pp == 0 as *pp == MAP_FAILED.

[hyperion]

Just because there is a coment doesn't mean the code does what is intended or meaningful.

 

Isn't one of the points of intptr_t that the round-trip is guaranteed? So why check?

 

[vladislavbelov]

  On 06/08/2021 at 5:52 PM, hyperion said:

Just because there is a coment doesn't mean the code does what is intended or meaningful.

Expand  

In current state it's pretty meaningful.

  On 06/08/2021 at 5:52 PM, hyperion said:

Isn't one of the points of intptr_t that the round-trip is guaranteed? So why check?

Expand  

It's unrelated to the check. When we use MAP_FAILED we don't care about its value. And only that particular place cares. MAP_FAILED is a standard constant for mmap on POSIX. And usually it's defined as (void*)-1. And here it's just defined using intptr_t, maybe for a more explicit type conversion. 

[hyperion]

So it's to guard against someone redefining MAP_FAILED to nullptr?

[vladislavbelov]

  On 06/08/2021 at 6:08 PM, hyperion said:

So it's to guard against someone redefining MAP_FAILED to nullptr?

Expand  

Kind of, we define MAP_FAILED by ourselves on Windows.

[hyperion]

  On 06/08/2021 at 6:26 PM, vladislavbelov said:

Kind of, we define MAP_FAILED by ourselves on Windows.

Expand  

And because we define it ourself I thought it's to test whether we use a doggy compiler or such

[s0600204]

  On 06/08/2021 at 3:19 PM, hyperion said:

a patch to set cc and arch usable for cross-compilation, anyone sees a setup this might break?

Expand  

Whilst autotools may use CBUILD for the build system and CHOST for the system that the built application is to run on, premake5 uses os.host for the build system and os.target for the system the built application is to run on.

Thus, I think you might want to go back to os.istarget("windows") in your patch, as the rest of the file expects arch to be that of the system the built application is to run on, and pulls in the correct architecture-specific headers accordingly.

(Of course, I could be getting confused...)

(That said, the architecture detection on Windows is broken anyway: our autobuilder is 64-bit but spits out a 32-bit executable... the only thing telling it what arch to build for is what you see in that file.)

[Stan`]

  On 06/08/2021 at 8:43 PM, s0600204 said:

That said, the architecture detection on Windows is broken anyway: our autobuilder is 64-bit but spits out a 32-bit executable... the only thing telling it what arch to build for is what you see in that file.

Expand  

It's also in the generated sln

[hyperion]

  On 06/08/2021 at 8:43 PM, s0600204 said:

Whilst autotools may use CBUILD for the build system and CHOST for the system that the built application is to run on, premake5 uses os.host for the build system and os.target for the system the built application is to run on.

Expand  

Agreed, just that premake terminology is rather awful. I pass in --os=windows --cc=mingw (for better or worse), so os.istarget("windows") and os.ishost("linux") become true. Then there is the premake command system { id } which I believe sets --os=id and results in os.istarget(id) to become true. Hard to not be confused at this point.

Anyway I can't take the current os.istarget("windows") path here for cross-compilation. I could rename CHOST to target_triple thou that wont change the logic.

[Stan`]

@hyperion
regarding icc and premake core is this a regression qhen updated premake for A24 ? i remember there is a linux flag for icc

[hyperion]

  On 06/08/2021 at 9:24 PM, Stan` said:

@hyperion
regarding icc and premake core is this a regression qhen updated premake for A24 ? i remember there is a linux flag for icc

Expand  

There is but that's a poor man choice. Premake will reject --cc=icc (cc option used to set toolset analog to os option ), so will default to probably gcc.

 

if cc == nil then
        filter { "toolset:clang" }
                cc = "clang"
        filter { "toolset:gcc" }
                cc = "gcc"
        filter { "option:icc" }
                cc = "icc"
        filter {}
end

So the above only works because we put icc last. Often that doesn't work, if we could use toolset:icc the premake5.lua could be written a lot cleaner.

[hyperion]

Some small updates:

• I have a static pyrogenesis.exe. It runs till after reading configuration but then fails with a pagefault (outside legal mem range). Running gdb within wine seems tricky. By default the backtrace is only framepointers and if I try dwarf-2 symbols which is apparently supported winedbg crashes.  So I put debugging it aside for now.
• I started cleaning up my patches, only over 60 left of more than 150.
• I also packaged premake-5.0.0_alpha16 and rdb's  fcollada and use them as external tool respectively lib; they work both fine it seems
• Building nvtt on mingw-w64 is completely broken I found out.

[hyperion]

Another update:

• If I disable the optimizer in gcc the game actually runs, just played a game in wine. The only issue I noticed 0ad refuses to write a user.cfg.
• Also packaged tinygettext from upstream and using it as external dep.
• libsdl2 and rust got an update and no longer need any changes to be usable.

So the current status is:

• Mostly runs with all optional stuff except for lobby disabled.
• Nothing from ./libraries, nor premake nor tinygettext from 0ad are used.
• Only spidermonkey and 0ad need patching. Fcollada needs a minor fix wrt pkg-config file. The rest is vanilla (Gentoo patches only if there are any).
• No wseh nor wdbg in 0ad, I'm gonna leave that out.

Slowly getting there.

 

[hyperion]

Some time passed and I thought I post an update on the status, haven't given up just got sidetracked:

• Fixed the user.cfg issue mentioned earlier but found one with replays (can't rewind to start of line).
• Fixed spidermonkey for mingw64-runtime >= 9.0.0
• Found out where the debug info got lost. Premake has a very hidden feature to add -s to the linker options unless you explicitly enable debugging within the premake script!
• Got fed up with premake enough to write a cmake script. Can even package mods . Probably only works for Linux right now.
• Most of wdbg restored. Missing stackwalker might not be easily fixable due to dwarf vs pdb? Anyway low priority.
• wseh might be doable as mingw 64 bit unlike 32 bit actually uses seh, have to investigate further.
• collada only used during caching/building mods (lack of nvtt prevented me from even trying) and so I only noticed during cmake porting that some more work might be needed than originally thought.

To conclude, a 64 bit port seems quite doable, the issues I ran into are mostly due to using mingw instead of msvc.

 

Next steps:

• use cmake with mingw
• try building atlas
• find out what's wrong with replays

 

Current wishlist for A26:

• Drop nvtt for something else rather earlier than later, not gonna fix it, I have no clue what an appropriate replacement could be. Seems to be not that much work once you know the replacement and are somewhat familiar with texture conversion in 0ad.
• Only use <thread> instead of pthread and win threads. 80% of the work seems updating mongoose.
• use std:filesystem instead of custom implementation and sometimes boost.
• Use a proper pyrogenesis.exe.manifest instead of those code and linker tricks which seem to tell windows to run in win98 compat mode. Might fix some issues users see currently.

[Stan`]

  On 26/08/2021 at 5:56 PM, hyperion said:

Drop nvtt for something else rather earlier than later, not gonna fix it, I have no clue what an appropriate replacement could be. Seems to be not that much work once you know the replacement and are somewhat familiar with texture conversion in 0ad.

Expand  

Any help in that area is welcome @vladislavbelov
didn't have enough time in A25.

  On 26/08/2021 at 5:56 PM, hyperion said:

Only use <thread> instead of pthread and win threads. 80% of the work seems updating mongoose.

Expand  

@wraitiihad some patches.

https://code.wildfiregames.com/D1923

https://code.wildfiregames.com/D1924

  On 26/08/2021 at 5:56 PM, hyperion said:

use std:filesystem instead of custom implementation and sometimes boost.

Expand  

That has some issues with macOS and its sdk

https://code.wildfiregames.com/D3185

https://code.wildfiregames.com/D4225

  On 26/08/2021 at 5:56 PM, hyperion said:

•  
• Use a proper pyrogenesis.exe.manifest instead of those code and linker tricks which seem to tell windows to run in win98 compat mode. Might fix some issues users see currently.

Expand  

I'd like to help with that. Seemed a bit annoying with premake

https://code.wildfiregames.com/D1338

  On 26/08/2021 at 5:56 PM, hyperion said:

Got fed up with premake enough to write a cmake script. Can even package mods . Probably only works for Linux right now.

Expand  

A lot of maintainers are annoyed that. I personally hate cmake gui but if it can be done transparently why not.

https://trac.wildfiregames.com/ticket/1104

I tried with meson but never got a working executable... And a lot of features we used were missing.

 

[hyperion]

Thanks for the long answer.

 

  On 26/08/2021 at 6:56 PM, Stan` said:

  On 26/08/2021 at 5:56 PM, hyperion said:

use std:filesystem instead of custom implementation and sometimes boost.

Expand  

That has some issues with macOS and its sdk

Expand  

If you want to build on an old mac what is the issue with installing a more recent compiler? Just treat it like any other dep.

 

  On 26/08/2021 at 6:56 PM, Stan` said:

  On 26/08/2021 at 5:56 PM, hyperion said:

• Use a proper pyrogenesis.exe.manifest instead of those code and linker tricks which seem to tell windows to run in win98 compat mode. Might fix some issues users see currently.

Expand  

I'd like to help with that. Seemed a bit annoying with premake

Expand  

The manifest is an xml file, guess there should be a fancy gui on windows to generate it. Then you simply include it in the rc file and the resource compiler will do the right thing. Is somewhat orthogonal to what you do with your patch.

https://docs.microsoft.com/en-us/windows/win32/sbscs/application-manifests

in the rc file add: IDR_RT_MANIFEST1        RT_MANIFEST             "pyrogenesis.exe.manifest"

done.

The minimalist one I use for mingw to replicate what I call the linker hack:

<?xml version="1.0" encoding="utf-8" standalone="yes"?>

<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
  <dependency>
    <dependentAssembly>
      <assemblyIdentity
          type="Win32"
          name="Microsoft.Windows.Common-Controls"
          version="6.0.0.0"
          processorArchitecture="*"
          publicKeyToken="6595b64144ccf1df"
          language="*" />
    </dependentAssembly>
  </dependency>
</assembly>

But there is a lot that can be set like dpi awareness etc, then people don't have to set properties to work around green stripes I suspect

 

  On 26/08/2021 at 6:56 PM, Stan` said:

  On 26/08/2021 at 5:56 PM, hyperion said:

Got fed up with premake enough to write a cmake script. Can even package mods . Probably only works for Linux right now.

Expand  

A lot of maintainers are annoyed that.

Expand  

10 years later still stuck with premake ...

[Stan`]

  On 26/08/2021 at 8:57 PM, hyperion said:

If you want to build on an old mac what is the issue with installing a more recent compiler? Just treat it like any other dep.

Expand  

I'm not exactly sure what the reasoning is for this but I believe you can't install a more recent Apple compiler (e.g XCode versions are only for specific mac versions) so you'd need to install the Clang compiler from brew or macports, which usually comes with another set of issues. @wraitii might know more.

  On 26/08/2021 at 8:57 PM, hyperion said:

The minimalist one I use for mingw to replicate what I call the linker hack:

Expand  

What's the hack it replaces?

  On 26/08/2021 at 8:57 PM, hyperion said:

But there is a lot that can be set like dpi awareness etc, then people don't have to set properties to work around green stripes I suspect

Expand  

Yeah definitely; I chatted a bit with @smiley about it. I suppose the fork went that route.

  On 26/08/2021 at 8:57 PM, hyperion said:

10 years later still stuck with premake ...

Expand  

Honestly I never found it so bad. Maybe stockholm syndrome. Meson was an absolute pain for me to setup and I didn't even get a working build :/

 

[hyperion]

  On 26/08/2021 at 9:39 PM, Stan` said:

What's the hack it replaces?

Expand  

See project_add_manifest() in premake5.lua. There are also plenty of pragma linker comments ignored by mingw and some specific function calls that could be replaced. Don't remember them offhand. They are sometimes even commented with do not really work.

 

  On 26/08/2021 at 9:39 PM, Stan` said:

Honestly I never found it so bad. Maybe stockholm syndrome. Meson was an absolute pain for me to setup and I didn't even get a working build :/

Expand  

premake isn't that bad until your use-case outgrows it. Not really familiar with meson.

Also me thinks the 0ad code is organized as-is mostly to fit premake, where it should be the other way around.

[smiley]

Regarding the manifest, it was and is always the intention from Microsoft, that everything be done on visual studio. When you tell premake to output solutions and project files, it's pretty nice to work with. Essentially, just tick boxes. Premake itself has all functions to setup the config.

I am not sure the alternatives for premake are that much better to be honest, not that premake itself isn't weird.

[Stan`]

@vladislavbelov noted it seems that sadly the manifest hidpi settings only support Windows 10

https://docs.microsoft.com/en-us/windows/win32/sbscs/application-manifests

[smiley]

I read that page back when trying to get rid of the stripes, I recall there being two flags, with one being recently introduced. Premake uses the recent one iirc.

The win API has a fallback function which I think is supported in earlier versions.

[hyperion]

  On 27/08/2021 at 10:07 AM, Stan` said:

@vladislavbelov noted it seems that sadly the manifest hidpi settings only support Windows 10

Expand  

But that probably also means older versions don't do fancy stuff which break user experience if the setting is not recognized. Are there any reported cases of green stripes bug for older versions?

[Stan`]

  On 27/08/2021 at 11:28 AM, hyperion said:

But that probably also means older versions don't do fancy stuff which break user experience if the setting is not recognized. Are there any reported cases of green stripes bug for older versions?

Expand  

Not that I know of. But it's alway hard to estimate the impact of such a bug from the reports.

According to the people developing the spring engine AMD dropped support for compatibility profiles for GL < 3.0

[smiley]

  On 27/08/2021 at 11:28 AM, hyperion said:

But that probably also means older versions don't do fancy stuff which break user experience if the setting is not recognized. Are there any reported cases of green stripes bug for older versions?

Expand  

I think the bug exists elsewhere as well. And the case of green stripes was a quite recent introduction, which came from an Intel driver update at least on my machine. Its usually just a case of blurry and stretched graphics and the SDL2 window being small enough to not even fit the main menu buttons.

Also, I looked at the old commit, and I apparently came to the conclusion of using exclusive full screen (I recall saying the game already uses it somewhere here recently, which is wrong), which bypasses windows scaling stuff completely, or using this flag which means the resolutions reported by SDL2 are actually correct.