Building a 64-bit pyrogenesis (and deps) for Windows

[asterix]

36 minutes ago, hyperion said:

What do you want to replace and why?

cairo  replaced by ctx which seems to have more options and cairo is getting older.

[Stan`]

Well first step is Freetype then we'll see about text rendering

https://code.wildfiregames.com/D4108

[Trinketos]

Are there any plans to make a Vulkan version of 0 A.D.?

[Stan`]

Yes in the future.

[asterix]

@hyperion btw https://code.wildfiregames.com/rP25933 and https://code.wildfiregames.com/rP25938 have been recently committed which can your porting work make easier hopefully.

[hyperion]

@asterix

I'm fond of both patches, the valgrind one I used prior to it being committed. The pkg-config one was useless for this task. Beside using cmake in the meantime, there is the issue of the pkg-config premake module needing halve a dozen patches to support cross compiling and being made compatible with pkgconf (an alternate implementation) in combination with a cross root and finally compiling for windows using pkg-config isn't foreseen by the external library search logic either. Further the current premake setup feeds flags to the resource compiler which it doesn't understand and so further adjustments would need to be made.

What I did since last update:

• cherry picked patch for mingw runtime 9.0.0 to support gcc 11 (openal build)
• switched to gcc 11.2
• updated binutils to 2.37
• boost updated to 1.77
• curl updated to 7.79

 

After talking with @Stan` I also looked into clang status, cross compiling to windows still can't be considered mature. There is initial pdb support in llvm which probably is desirable and I guess gcc will never have. There is also a very immature resource compiler which would be needed as well. So in a year or two clang is probably a good alternative.

He also told me that antivirus doesn't like 0ad and if built with mingw even less, I thought about it and think the injection of startup code into crt before standard program entry might be a possible reason for this. To be honest this is a first seeing this trick. Might giving changing the startup procedure a try.

I already started feeding changes back to 0ad, but with @wraitii having barely any time for 0ad currently this might be a lengthy process. So don't expect it being available for general consumption anytime soon.

 

 

[Stan`]

1 hour ago, hyperion said:

He also told me that antivirus doesn't like 0ad and if built with mingw even less, I thought about it and think the injection of startup code into crt before standard program entry might be a possible reason for this. To be honest this is a first seeing this trick. Might giving changing the startup procedure a try.

I already started feeding changes back to 0ad, but with @wraitii having barely any time for 0ad currently this might be a lengthy process. So don't expect it being available for general consumption anytime soon.

Hit me up if you need anything. Maybe @vladislavbelov could review your patches.

[hyperion]

I'm currently ahead of trunk by roughly 70 patches. 40-50 are mostly trivial to review and reject/apply, who does it isn't relevant for me. As there is no patch series support in phab this won't be particularly fast anyway. Also I'm not in a hurry either.

There are also upstreams to wait for to some degree, so it taking time isn't necessarily bad as long as there is some progress.

[Stan`]

Alright. That seems like a fair amount of patches to review.

Btw I'm a bit worried about the lack of response on the compressonator library. It was the same with NVTT in the past.

I'm currently considering extracting nvtt bits and ditching the rest of the lib.

[hyperion]

56 minutes ago, Stan` said:

Btw I'm a bit worried about the lack of response on the compressonator library. It was the same with NVTT in the past.

Might be a lengthy vacation, will ping at some point, so even if not fixed anytime soon we can at least know what would need to be done on our side if we choose compressonator.

 

59 minutes ago, Stan` said:

I'm currently considering extracting nvtt bits and ditching the rest of the lib.

This doesn't sound like much of an improvement. Fixing it to fetch a tarball and apply patches as is done for spidermonkey might be desirable but low priority as long as getting away from it is seriously considered. Moving tinygettext and properly build as external dep with option as system dep sounds like time better invested.

If there are other alternatives to nvtt I don't mind looking into them either, but to be clear, I'm still under the impression compressionator to be a good choice for 0ad. Clean api, quality kernels and a whole slew of extras for artists is probably hard to get elsewhere.

[asterix]

12 hours ago, hyperion said:

patch series support

this could be bypassed by making parent and child revisions or name it similarly in here https://developer.blender.org/T88367 as inspiration.

[hyperion]

Can the CI handle it? Do I have to rebase any patch onto master first? Does the committer have to rebase them again when committing? Do I have to manually list any dependencies? What happens if there are changes to master, do I have to rebase them and manually extract and update them again?

Well, of course emulating series support is possible to some degree but that doesn't mean it's helpful. If it were arcanist wouldn't be eager to squash commits and 0ad devs wouldn't add semi random changes to existing patches just because the file was already touched or similar. If series support were a first class citizen of the workflow probably 90% of the commits would be split. For example the valgrind patch you mentioned has an "and" in the subject line, acknowledging that it's at least two logical changes squashed. Then look at the corresponding differential in phab and you will notice it could have become a couple more "and" as well.

Phab is great for multiple people collaborating on a single change but for a simple reject/accept review it's barely adequate/ the overhead unreasonable. In my perception feeding them one by one or in small batches (they are mostly logically independent) isn't fast but causes the least work for me and reviewers alike over all.

[Stan`]

On 25/09/2021 at 10:33 PM, hyperion said:

Can the CI handle it?

Not yet. See: https://code.wildfiregames.com/D4085 (Of course point is moot if we move away from Phab, but one fire at a time)

On 25/09/2021 at 10:33 PM, hyperion said:

Do I have to rebase any patch onto master first

Well it's better if you do, but if nobody touched the files you changed you should be fine.

 

[hyperion]

4 hours ago, Stan` said:

Well it's better if you do, but if nobody touched the files you changed you should be fine.

Well, some of my own patches already conflict / require rebase if order changes. Basically I have to split the mingw branch into up to 70 branches if I go along with this.

The patch I submitted in the meantime already needs rebasing, should I do it and retrigger CI and hope it gets committed before needing another rebase or leave it be for instance? Also having dozens of open differentials doesn't sound that enticing either.

[Stan`]

56 minutes ago, hyperion said:

The patch I submitted in the meantime already needs rebasing, should I do it and retrigger CI and hope it gets committed before needing another rebase or leave it be for instance? Also having dozens of open differentials doesn't sound that enticing either.

Vlad told me he'd do the rebase. I haven't asked since.

Well yeah it sucks, but unless I can review those patches I'm not sure I can do much. Now technically if they are atomic it should be pretty obvious if they are correct.

The utf16 one is slightly out of my reach. I don't even know if we have good tests for it.

[hyperion]

1 hour ago, Stan` said:

Well yeah it sucks

Not a big deal for me

 

1 hour ago, Stan` said:

The utf16 one is slightly out of my reach

Let me see if I can cherry pick a few easier ones.

 

[asterix]

well https://trac.wildfiregames.com/changeset/26040 free type was committed, now we need to decide on rendering as I see it @Stan` is working on inclusion of extension loader library GLAD https://code.wildfiregames.com/D721

[hyperion]

Glad someone still remembers this

freetype builds out of the box with cross mingw, haven't looked into glad2 as of yet. Also form discussion the move to compressonator is very likely and as such another blocker would be out of the way.

[asterix]

here is very interesting blogpost https://aras-p.info/blog/2022/06/23/Comparing-BCn-texture-decoders/ that guy specializes on performance optimisations @Stan` @vladislavbelov what do you think?

[Ceres]

From the above conversations I understand that building a 64bit Windows version is still far away, correct?

[hyperion]

8 hours ago, Ceres said:

From the above conversations I understand that building a 64bit Windows version is still far away, correct?

Well, Microsoft drops support for 32 bit Windows in 2023, after that there is no longer a reason to have 32 bit pyrogenesis. The way windows is handled with prebuilt binaries in repo makes it less straightforward to offer both. The technical issues are mostly understood and not that hard to fix, just work.

[Ceres]

@hyperion

You mentioned lots of patches in your pipeline. Does review of patches involve testing the patches in the game according to the described test strategy "only" (still a lot to do), or in addition code review? I cannot help about the latter, as I have no coding experience, but maybe about the former (testing of simpler patches). Please let me know if you think that I could be of help and where to start.

 

 

[hyperion]

The only way to get 64-bit support fast is to replace the 32-bit version, even that is probably more than a day of work, but more or less straight forward.

What I did is cross compile 0ad which is somewhat troublesome, my changes are from replacing buildsystem, fixing deps or even replace them, fixing cross compile bugs and more. Reviewing means accept the changes and committing them. The developer on which shoulder this would likely fall is busy adding Vulkan support and still has font rendering on his plate which are in my opinion much more important. Also I prefer to wait for the git migration which is in the works.

The takeaway from my work is that there is little in the way of a straight replacement, all else is somewhat cumbersome.

[Ceres]

Ok, I got it and will patiently wait. In the meantime, I can contribute to simpler tasks.