Jump to content

Building a 64-bit pyrogenesis (and deps) for Windows


Recommended Posts

Little had to be changed in cmake to get it working with mingw as well, have scenario and actor editor working now. Returning a fixed duration for replay length makes them work otherwise.

 

So the only big issue now is not being able to generate mods, respectively being able to run in svn, which is blocked by nvtt not liking mingw at all.

Link to comment
Share on other sites

So, the issues with replays was istream seekg being broken for text input with mingw (magically eating crlf as lf and so offsets are off), works when set to binary.

 

@vladislavbelov @wraitii,

The optimizer somehow breaks profiler2 in mingw, and I noticed profiler2 initialization is racy with native Linux build as well, couldn't figure out where it goes wrong so far (thou gcc bug less likely than 0ad bug). Replacing profiler2 with a stub and we have optimizations with mingw and no race with test in Linux; and basically the same execution speed in wine as a native build.

 

@vladislavbelov,

do you have a suggestion for a replacement of nvtt? Might look into it if there is a path forward. Otherwise I have to stick to releases and use the 0ad-data tarball.

Link to comment
Share on other sites

11 hours ago, hyperion said:

So, the issues with replays was istream seekg being broken for text input with mingw (magically eating crlf as lf and so offsets are off), works when set to binary.

w480

11 hours ago, hyperion said:

do you have a suggestion for a replacement of nvtt? Might look into it if there is a path forward. Otherwise I have to stick to releases and use the 0ad-data tarball.

I haven't chosen a final solution for 0ad, temporarily you could use Compressonator. But any different (from nvtt) solution requires a noticeable amount of code to integrate into 0ad.

  • Like 1
Link to comment
Share on other sites

3 hours ago, Stan` said:

The author claims it wont link.

The reason I don't want to invest time into nvtt is not the difficulty in porting (well hard to tell upfront). First reason is nvtt is dead, then latest release and master is broken, else Gentoo wouldn't use the 0ad bundled copy. 0ad or some devs at least have the intention to move away from nvtt. Lastly, the in tree copy is a typical 0ad mess. Import the code and do some random changes in the same commit, then carry some patches in a separate directory which may or more likely may not apply to pristine sources, then forget about them and change the sources directly again, formatting included.

@wraitii did a good job with spidermonky, if not for storing the tarball in svn I'd label it close to perfect ;).

Well, I look into compressonator, even if just to figure out it doesn't work.

Link to comment
Share on other sites

16 minutes ago, hyperion said:

The reason I don't want to invest time into nvtt is not the difficulty in porting (well hard to tell upfront). First reason is nvtt is dead, then latest release and master is broken, else Gentoo wouldn't use the 0ad bundled copy. 0ad or some devs at least have the intention to move away from nvtt. Lastly, the in tree copy is a typical 0ad mess. Import the code and do some random changes in the same commit, then carry some patches in a separate directory which may or more likely may not apply to pristine sources, then forget about them and change the sources directly again, formatting included.

Yeah I am actually in favor of ditching it :) It was on the roadmap for A25, but @vladislavbelov had little time and focused on HiDPI instead, and we only got some part of that. I'm also not really happy of the fact we don't patch it correctly (as in its already patched)

Gentoo doesn't use the bundled copy at all :D (Source I've been talking with a few of the maintainers on gentoo-games) They just go --without-nvtt for the most part and use the tarballs. Anyway.

I'm curious to see how it'll go with compressonator. Might event try on Windows if I get the chance.

Link to comment
Share on other sites

9 minutes ago, Stan` said:

Gentoo doesn't use the bundled copy at all

You made me check the ebuild for 25b :)

    # Build bundled NVTT
    # nvtt is abandoned upstream and 0ad has forked it and added fixes.
    # Use their copy. bug #768930
    if use nvtt ; then
        cd libraries/source/nvtt || die
        elog "Building bundled NVTT (bug #768930)"
        JOBS="-j$(makeopts_jobs)" ./build.sh || die "Failed to build bundled NVTT"
        cd "${S}" || die
    fi

 

Didn't know compressonator, but gives a good first impression:

  • maintained by AMD
  • has standalone libs for integration.
  • has cli and gui tools for artists.
  • Like 1
Link to comment
Share on other sites

56 minutes ago, Ceres said:

I'm curious: Will performance be better with 64bit on Windows? Only if using more RAM? How much at least?

Performance increases are usually based on program size as far as 64bit goes. So you can work with a lot more data since you can fit it in RAM. There are small increases in other areas, probably not more than 10% speed wise. Of course there is very little difference for the CPU when running a 32bit optimized program on 64bit. The engine would need to be modified to take advantage of most of the benefits that aren't RAM related.

  • Thanks 1
Link to comment
Share on other sites

47 minutes ago, Stan` said:

Judging from that not so much

I meant the native linux build, haven't ever tested the 32 bit msvc build offered for download as I don't have wine32 laying around. Should probably do that some time.

 

47 minutes ago, Ceres said:

Is it worth the effort?

As a rule of thumb, a 64 bit build is faster but requires more ram (compiler can use more instructions, registers etc but types may be larger), how much of a difference it makes only testing can tell. But I don't care about that in the first place. The performance issues can't be fixed even if it's twice as fast as the hogs aren't linear.

The point of a mingw port is a) to build everything from ground up easily and b) cross compile support.

As for the effort, well, it's still reasonable with the roughly 40h I spent on it so far. This includes things like learning premake and lua (which I then abandoned) as well as reading Microsoft documentation.

  • Like 3
Link to comment
Share on other sites

19 minutes ago, hyperion said:

I meant the native linux build, haven't ever tested the 32 bit msvc build offered for download as I don't have wine32 laying around. Should probably do that some time.

Then might be better actually. Because i've seen 40fps difference between the same machine (one was on arch and the other on windows)

Strange I thought wine64 was able to run  32bit software too (No need to check :P)

You might also gain from the fact gcc might be able to write sse3 instructions from code (while msvc cannot (max we use is SSE2))

 

Link to comment
Share on other sites

40 fps sounds substantial.

 

8 minutes ago, Stan` said:

Strange I thought wine64 was able to run  32bit software too (No need to check :P)

IFF built with 32 bit support, which requires 32 bit builds of deps as well, so I disabled it. Not wine's fault :)

Link to comment
Share on other sites

21 minutes ago, Stan` said:

If you could upload the binaries somewhere I could compare on Windows.

Well, the current binary looks in /usr/share for ro data, reusing the native install. Sure can do a build intended for windows, but guess has to wait for tomorrow. Currently hacking in compressor support and about putting things off for the day.

What I have so far:

  • build of core compressor and framework libs (linux only)
  • Some hack of TextureConverter using framework which passes test_TextureManager and actually produces dds.
  • only producing BCn dds, if DXTn or ATI2n are really required somewhere down the road I have to figure out how to get support for those, doesn't look trivial tho. The build system is somewhat of a mess, they claim they are working on it, so things should improve.
  • quite some features in TextureConverter dropped for now.
  • diff is some 450 lines.

So probably 20% there already.

Link to comment
Share on other sites

@Stan` quick tested the mingw build, two issues

  1. I built libcurl with openssl instead of winssl support, so modio didn't work, fixed.
  2. The alt tab issue remains in sdl 2.0.16, can't reproduce in wine

---

So did run the mod packaging with rdb fcollada and compressonator and made some findings.

a) Following files can't be opened in cmp_framework:

  • Microsoft DirectDraw Surface (DDS): 16 x 16, compressed using DXT1
  • Microsoft DirectDraw Surface (DDS): 128 x 128, compressed using DXT3
  • Microsoft DirectDraw Surface (DDS): 512 x 512, compressed using DXT5
  • PNG image data, 1024 x 1024, 16-bit/color RGB, non-interlaced
  • PNG image data, 1024 x 1024, 16-bit grayscale, non-interlaced

Quite some assets in dds format are obviously already compressed. Not sure using nvtt/compressonator to process them doesn't result in loss of quality. dds not yet compressed open fine. Compressonator obviously doesn't like 16 bit png neither.

b) Then a dozen or so *.dds have spaces in their path/file name! Not a real problem, just made writing a script removing the above a bit more awkward. Would still be nice to clean them up.

c) There are png of dimension 1xsomething, they cause segfaults, compressonator probably requires dimension at least 2

d) craps out when generating mipmap for metal_desert_small_a.dds

e) some dae: skeleton structure not recognized with rdb fcollada, do you remember if you changed something? (getting Assertion not satisfied (line 393): failed requirement "recognised skeleton structure")

Assets not matching a)-e) seem fine.

 

Link to comment
Share on other sites

2 hours ago, hyperion said:

The alt tab issue remains in sdl 2.0.16, can't reproduce in wine

Basically the window sticks on top of the desktop while it should go to the background. Relevant issue https://github.com/libsdl-org/SDL/issues/4039

2 hours ago, hyperion said:

b) Then a dozen or so *.dds have spaces in their path/file name! Not a real problem, just made writing a script removing the above a bit more awkward. Would still be nice to clean them up.

c) There are png of dimension 1xsomething, they cause segfaults, compressonator probably requires dimension at least 2

Got a list ?

2 hours ago, hyperion said:

 

e) some dae: skeleton structure not recognized with rdb fcollada, do you remember if you changed something? (getting Assertion not satisfied (line 393): failed requirement "recognised skeleton structure")

What's the DAE? To convert a model from DAE to PSA you need to have the public mod loaded with the skeletons

We changed a lot of stuff in our collada. Most of RDB's fixes are in already.

https://trac.wildfiregames.com/ticket/5907 was created for the remaining two. See also the discussion here https://github.com/rdb/fcollada/pull/4

 

 

 

Link to comment
Share on other sites

22 minutes ago, Stan` said:

Got a list ?

 

Files with spaces:

binaries/data/mods/public/art/terrains/cliff/cliff volcanic coarse.xml
binaries/data/mods/public/art/terrains/cliff/cliff volcanic ground.xml
binaries/data/mods/public/art/terrains/cliff/cliff volcanic light.xml
binaries/data/mods/public/art/terrains/grass/grass b soft dirt 50.xml
binaries/data/mods/public/art/terrains/snow/path a.xml
binaries/data/mods/public/art/terrains/snow/snow 50.xml
binaries/data/mods/public/art/terrains/snow/snow forest.xml
binaries/data/mods/public/art/terrains/snow/snow grass 100.xml
binaries/data/mods/public/art/terrains/snow/snow grass 2.xml
binaries/data/mods/public/art/terrains/snow/snow grass 75.xml
binaries/data/mods/public/art/terrains/snow/snow rocks.xml
binaries/data/mods/public/art/terrains/snow/snow rough.xml
binaries/data/mods/public/art/terrains/special/light blue.xml
binaries/data/mods/public/art/terrains/special/neon green.xml
binaries/data/mods/public/art/textures/skies/sunny 1
binaries/data/mods/public/art/textures/skies/sunny 1/back.dds
binaries/data/mods/public/art/textures/skies/sunny 1/front.dds
binaries/data/mods/public/art/textures/skies/sunny 1/left.dds
binaries/data/mods/public/art/textures/skies/sunny 1/right.dds
binaries/data/mods/public/art/textures/skies/sunny 1/top.dds
binaries/data/mods/public/art/textures/skies/sunset 1
binaries/data/mods/public/art/textures/skies/sunset 1/back.dds
binaries/data/mods/public/art/textures/skies/sunset 1/front.dds
binaries/data/mods/public/art/textures/skies/sunset 1/left.dds
binaries/data/mods/public/art/textures/skies/sunset 1/right.dds
binaries/data/mods/public/art/textures/skies/sunset 1/top.dds
binaries/data/mods/public/art/textures/skies/sunset 2
binaries/data/mods/public/art/textures/skies/sunset 2/back.dds
binaries/data/mods/public/art/textures/skies/sunset 2/front.dds
binaries/data/mods/public/art/textures/skies/sunset 2/left.dds
binaries/data/mods/public/art/textures/skies/sunset 2/right.dds
binaries/data/mods/public/art/textures/skies/sunset 2/top.dds
binaries/data/mods/public/art/textures/skins/props/helmet/hele_phrygian_mask_02_silver - copia.png
binaries/data/mods/public/art/textures/terrain/types/cliff volcanic coarse.dds
binaries/data/mods/public/art/textures/terrain/types/cliff volcanic ground.dds
binaries/data/mods/public/art/textures/terrain/types/cliff volcanic light.dds
binaries/data/mods/public/art/textures/terrain/types/grass b soft dirt 50.dds
binaries/data/mods/public/art/textures/terrain/types/light blue.png
binaries/data/mods/public/art/textures/terrain/types/neon green.png
binaries/data/mods/public/art/textures/terrain/types/path a.dds
binaries/data/mods/public/art/textures/terrain/types/snow 50.dds
binaries/data/mods/public/art/textures/terrain/types/snow forest.dds
binaries/data/mods/public/art/textures/terrain/types/snow grass 100.dds
binaries/data/mods/public/art/textures/terrain/types/snow grass 2.dds
binaries/data/mods/public/art/textures/terrain/types/snow grass 2_norm.png
binaries/data/mods/public/art/textures/terrain/types/snow grass 2_spec.png
binaries/data/mods/public/art/textures/terrain/types/snow grass 75.dds
binaries/data/mods/public/art/textures/terrain/types/snow rocks.dds
binaries/data/mods/public/art/textures/terrain/types/snow rough.dds
binaries/data/mods/public/art/textures/ui/session/icons/mappreview/Barcania (3).png
binaries/data/mods/public/art/textures/ui/session/icons/mappreview/Sandbox - Carthaginians.png
binaries/data/mods/public/art/textures/ui/session/icons/mappreview/Sandbox - Iberians.png
binaries/data/mods/public/art/textures/ui/session/icons/mappreview/The Persian Gates.png
binaries/data/mods/public/art/textures/ui/session/icons/mappreview/Tropical Island.png
libraries/source/fcollada/src/FCollada/Pre-requisites and License.rtf

 

pngs with 1 x dimension

binaries/data/mods/public/art/textures/ui/loading/progressbar/progressbar_middle.png
binaries/data/mods/public/art/textures/ui/session/icons/capture_bar.png
binaries/data/mods/public/art/textures/ui/session/icons/experience_fg.png
binaries/data/mods/public/art/textures/ui/session/icons/health_bg.png
binaries/data/mods/public/art/textures/ui/session/icons/health_fg.png
binaries/data/mods/public/art/textures/ui/session/icons/pack_bg.png
binaries/data/mods/public/art/textures/ui/session/icons/pack_fg.png
binaries/data/mods/public/art/textures/ui/session/icons/stamina_fg.png
binaries/data/mods/public/art/textures/ui/session/icons/supply_bg.png
binaries/data/mods/public/art/textures/ui/session/icons/supply_fg.png
binaries/data/mods/public/art/textures/ui/session/icons/upgrade_bg.png
binaries/data/mods/public/art/textures/ui/session/icons/upgrade_fg.png
binaries/data/mods/public/art/textures/ui/session/panel_shader_top_edge.png

 

50 minutes ago, Stan` said:

What's the DAE? To convert a model from DAE to PSA you need to have the public mod loaded with the skeletons

Collada files. So while generating the public mod (pyrogenesis --archivebuild=public/ --archievebuild-output=public.zip) output like the following is expected?

ERROR: art/animation/biped/rider/cavalry/spearman/idle_shield_relax_shoulder_02.dae: Assertion not satisfied (line 393): failed requirement "recognised skeleton structure"

Haven't looked into it yet. Just noticed those lines scrolling by :)

 

Link to comment
Share on other sites

9 hours ago, hyperion said:

Collada files. So while generating the public mod (pyrogenesis --archivebuild=public/ --archievebuild-output=public.zip) output like the following is expected?

You need to pass -mod=mod and -md=public too else you gonna run into trouble (badly compressed dds for the former som gui texture.xml files are in mod mod and bad animations for the later.

9 hours ago, hyperion said:

pngs with 1 x dimension

Those should be fixable but that's a strange limitation 2^0 is pow of two

9 hours ago, hyperion said:
helmet/hele_phrygian_mask_02_silver - copia.png

This one is a disgrace.

Mappreview can be fixed easily.

Terrain and skies are gonna be a pain though... (Would crash if they are missing iirc)

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

×
×
  • Create New...