Building a 64-bit pyrogenesis (and deps) for Windows

[hyperion]

On 29/06/2021 at 11:38 AM, Stan` said:

8 Ais on a giant maps means certain OOM so the crash is sad but not much we can do.

Offer a 64bit build? AFAIK Microsoft stopped shipping 32 bit versions of Windows.

---

Moderation: Thread split out from https://wildfiregames.com/forum/topic/45178-a25-svn-version-25810-not-playable

[Stan`]

There is a ticket for it iirc but I have no idea how much hidden work there is. Pretty sure there is some stuff that needs to be done for fcollada (at least building a64bit libmxl2 and updating the slns not sure about all the other libs nor whether the windows specific code was made at all for 64 bits.

[hyperion]

22 hours ago, Stan` said:

There is a ticket for it iirc but I have no idea how much hidden work there is. Pretty sure there is some stuff that needs to be done for fcollada (at least building a64bit libmxl2 and updating the slns not sure about all the other libs nor whether the windows specific code was made at all for 64 bits.

I don't see any technical issues. As the code is 64 bit safe on other systems there is really not much to worry here. The issue is mostly a political one. Should one offer both or replace the 32-bit one with a 64-bit? What to do with the autobuild? Should there even be an autobuild in the main repo?

[Stan`]

21 minutes ago, hyperion said:

I don't see any technical issues. As the code is 64 bit safe on other systems there is really not much to worry here. The issue is mostly a political one. Should one offer both or replace the 32-bit one with a 64-bit? What to do with the autobuild? Should there even be an autobuild in the main repo?

The os specific code might make assumptions. Eg checking for WIN32_ and not WIN64_. Or assuming some objects sizes to be 32bit. There is quite a fair bit of code to show callstack on windows which might have hacks and assumptions too and needs to be checked.

Sure there is a decision to be made but it will be easier once it works

As for the last question, if/when we eventually migrate to git probably not but for now it's really convenient. More so because you cannot currently cross compile and it allows us to generate the windows installer on mac.

 

[hyperion]

On 30/06/2021 at 1:54 PM, Stan` said:

The os specific code might make assumptions. Eg checking for WIN32_ and not WIN64_. Or assuming some objects sizes to be 32bit. There is quite a fair bit of code to show callstack on windows which might have hacks and assumptions too and needs to be checked.

Sure there is a decision to be made but it will be easier once it works

As for the last question, if/when we eventually migrate to git probably not but for now it's really convenient. More so because you cannot currently cross compile and it allows us to generate the windows installer on mac.

 

For contributors it's important to know upfront what is desired / the target before doing anything.

The political part needs to be addressed anyway, the work only if 64-bit is desired and depending on what is desired the work may even change.

I don't claim that it's as easy as a compile switch, but I'm sure a 64 bit build is far from an insurmountable obstacle, just a couple hours of work at most.

[Stan`]

Well now that SpiderMonkey forced us to drop XP (and possibly vista) and Windows 7 is almost dead. (There are some ways to get ESU until 2023) I am not against offering a 64 bit build of the game.

I don't think however that it should be a priority over better performance (even though pathfinder threading will help in A25) and Hidpi support.

7 minutes ago, hyperion said:

 

I don't claim that it's as easy as a compile switch, but I'm sure a 64 bit build is far from an insurmountable obstacle, just a couple hours of work at most.

Maybe not but there are quite a lot of edgecases that needs to be checked (I don't have a list in mind but given that just updating to a minor version of the sdl can break a ton of things on Windows, I wouldn't be surprised.

[hyperion]

On 03/07/2021 at 6:21 PM, Stan` said:

There are some ways to get ESU until 2023

So in your opinion dropping 32-bit build before 2023 is undesirable (sounds reasonable) which makes a straight replacement impossible for now. So how should a 64-bit build be handled?

[Stan`]

Well I think it should be handled cautiously. Maybe as a POC first in a git mirror cause we're gonna need a whole bunch of new dlls and .lib and Phabricator is terrible for that. Then once it works well enough (ideally tested by 2-3 people) we can offer a special build/installer for the adventurous people.

[hyperion]

On 05/07/2021 at 4:02 PM, Stan` said:

Well I think it should be handled cautiously. Maybe as a POC first in a git mirror cause we're gonna need a whole bunch of new dlls and .lib and Phabricator is terrible for that. Then once it works well enough (ideally tested by 2-3 people) we can offer a special build/installer for the adventurous people.

As I don't like this approach I created a cross root to build a static 64-bit pyrogenesis.exe. Well, I got all dependencies cross-compiled with only sm and fcollada missing. Got stuck at building a rust cross std lib, guess will take a while until I have time and energy to get back to it.

[Stan`]

You don't like cautious approaches ? XD

Yeah those two were in my annoyance list. I've never managed to build spidermonkey so far on Windows. And fcollada needs a bunch of csproj modifications...

[Ceres]

Can some of the pros here please spend a few moments in this thread to help interested users (like me ) how to install the necessary VS 2017 components? Thank you so much!

 

[Stan`]

To do a 64 bit build ?

[Ceres]

Sorry, no, to do a build in Windows 10 in general (currently 32bit). I read the Wiki instructions but did not make progress. As I don't want to spam this thread here with my problems, I would like to leave it at that. Since @Yekaterina
kindly provided further help in that other thread I might be able to continue and would then update the Wiki for a better understanding by noobs like myself.

[hyperion]

On 12/07/2021 at 8:09 AM, Stan` said:

You don't like cautious approaches

Na, what I don't like is storing binary artefacts in a repo and only providing the last step in a proper manner, instead how it's done for mac is how I see it should be for Windows too. Building from ground up in Windows strikes me as more difficult than cross-compiling for me. Some tricks were required but It was rather straight forward until I ran into rust. Never expected that bootstrapping a rust cross-compiler targeting Windows would be such a headache. I only expected 0ad itself to need quite some work to allow for cross-compilation which would be worthwhile irrespective of a potential win64 build.

[Stan`]

5 minutes ago, hyperion said:

On 12/07/2021 at 8:09 AM, Stan` said:

 

Na, what I don't like is storing binary artefacts in a repo and only providing the last step in a proper manner, instead how it's done for mac is how I see it should be for Windows too. Building from ground up in Windows strikes me as more difficult than cross-compiling for me.

Ah yeah sure! They'll be removed when we eventually switcht o git. Will have to find a way to make it easy to download them from the server/github/gitlab so we can get the same experience at what svn is.

Yeah building from the ground up is tiring. So many build systems/processes/hacks.

If you have a working cross compiled executable build process it could be nice to share it though.

8 minutes ago, hyperion said:

. I only expected 0ad itself to need quite some work to allow for cross-compilation which would be worthwhile irrespective of a potential win64 build.

I suppose making it work with mingw would be a good start.

[smiley]

On 15/07/2021 at 1:30 AM, hyperion said:

Na, what I don't like is storing binary artefacts in a repo and only providing the last step in a proper manner, instead how it's done for mac is how I see it should be for Windows too. Building from ground up in Windows strikes me as more difficult than cross-compiling for me. Some tricks were required but It was rather straight forward until I ran into rust. Never expected that bootstrapping a rust cross-compiler targeting Windows would be such a headache. I only expected 0ad itself to need quite some work to allow for cross-compilation which would be worthwhile irrespective of a potential win64 build.

https://github.com/microsoft/vcpkg

I suggest devs to drop third party libraries and just rely on vcpkg aside from SM and FCollada.

[Stan`]

Might consider it yes. Last time I tried though I didn't have enough control on toolsets and versions which would be an issue.

IIRC @Imarok had some fun with it. Not sure how to integrate it with premake, I know it handles cmake.

[hyperion]

8 hours ago, smiley said:

https://github.com/microsoft/vcpkg

I suggest devs to drop third party libraries and just rely on vcpkg aside from SM and FCollada.

Does this approach have the same shortcomings as using homebrew for mac? If so I don't think it's a great solution but nonetheless a clear improvement over the status quo.

 

On 14/07/2021 at 10:39 PM, Stan` said:

If you have a working cross compiled executable build process it could be nice to share it though.

Well, as said before, I only have most of the deps so far. As for rust, took me a bit of digging to patch it so it would build (bootstrap depends on sanity checks for dist target which I don't care about in the first place).

As for the quirks so far, gcc 11 is to new for the mingw runtime so I had to use gcc 10 configured to use pthreads so I get <mutex> <thread> etc. Then I configured all deps with --enable-static (as intended) and --disable-shared to reduce issues (don't remember which package didn't want to build a shared lib of the top of my head) and --disable-log for boost. In case of miniupnpc I had to use Makefile.mingw and use the static lib as target. The rest can be called ebuild bugs like excess deps or defines which are Linux specific. I don't mind sharing the exact setup once I have it working but it's Gentoo specific, so except for a proof of concept might not have much value for you. Anyway, if you use mingw on windows I think you won't run into much issues up to where I am either.

PS: had a quick first look at SM and Gentoo has a patch for fortify source, remembered you showing me the flags used by the the ubuntu maintainer which apparently caused issues.

 

[smiley]

44 minutes ago, hyperion said:

Does this approach have the same shortcomings as using homebrew for mac?

Yes, its pretty much the same thing to be fair, except that visual studio plays nicely with vcpkg, in that it tries to emulate a /usr/lib/. What other approach is there for Win?

For certain libraries like nvtt, one would need to patch the portfile too. Painful, but its the least painful way I have found.

[Stan`]

1 hour ago, smiley said:

What other approach is there for Win?

There is Conan, https://wildfiregames.com/forum/topic/26142-using-conan-to-get-windows-dependencies/?tab=comments#comment-376743

And of course what we're doing which is not really an approach

[hyperion]

So sm and fcollada compiled. Sm was a bit of a pain and is probably usable. For fcollada the attached patch was already sufficient.

Building 0ad seems to be a major headache thou. Premake pkgconfig module should check for ${CHOST}-pkg-config first and only fall back to pkg-config if not found like autotools does.

After a little hack for pkg-config I got tons of issues with undefined stuff. Probably caused by legacy hack after legacy hack. So a first simple suggestion attached as well. Best to get rid of the whole header thou and use the standard instead of custom types throughout the codebase.

 

fcollada-no-msc.patch pyrogenesis-types-include-stdint.patch

[Stan`]

6 minutes ago, hyperion said:

After a little hack for pkg-config I got tons of issues with undefined stuff. Probably caused by legacy hack after legacy hack. So a first simple suggestion attached as well. Best to get rid of the whole header thou and use the standard instead of custom types throughout the codebase.

I believe it's there because nothing was standard anywhere... @vladislavbelov might know more.

for pkg config @s0600204 is working on stuff

https://code.wildfiregames.com/D3611

https://code.wildfiregames.com/people/revisions/27/

First patch looks like it could use another define or something.

Thanks for keeping it up.

 

[hyperion]

59 minutes ago, Stan&#x60; said:

I believe it's there because nothing was standard anywhere

Long ago VS didn't support C99 I think

 

1 hour ago, Stan&#x60; said:

https://code.wildfiregames.com/D3611

https://code.wildfiregames.com/people/revisions/27/

Both are set to private?

 

1 hour ago, Stan&#x60; said:

First patch looks like it could use another define or something.

Sure, just as the only consumer (0ad) doesn't care either way I didn't bother thinking what could be an appropriate choice.

[Freagarach]

6 hours ago, hyperion said:

Both are set to private?

You'll need an account on Phabricator.

[Stan`]

6 hours ago, hyperion said:

Long ago VS didn't support C99 I think

Could be that yeah. I remember reading about the standard not ensuring the siezs of uint8_t though. But there are indeed a lot of remnants of the past.

6 hours ago, hyperion said:

Both are set to private?

First one is now public, there was a mistake in the visibility, second one does need an account

6 hours ago, hyperion said:

Sure, just as the only consumer (0ad) doesn't care either way I didn't bother thinking what could be an appropriate choice.

Fair