Building a 64-bit pyrogenesis (and deps) for Windows

[Stan`]

10 hours ago, nwtour said:

This issue marked as solved in 2.0.16...

I know, right? They found some kind of workaround which obviously works halfway.

Oh btw Windows is totally hating all the mingw binaries. It keeps looking for viruses in those.

[hyperion]

26 minutes ago, Stan` said:

Those should be fixable but that's a strange limitation 2^0 is pow of two

As it segfaults it's a bug anyway, if it's an intended limitation I'd expect an assert. I wouldn't "fix" them yet, might also be due to not being square or some other unexpected criterion. Will dig a bit into it later.

 

9 hours ago, hyperion said:

terrain/types/snow grass 2_norm.png

Is a funny one as well.

 

28 minutes ago, Stan` said:

Terrain and skies are gonna be a pain though

References would need to be updated too ofc. git grep "sunny 1"

binaries/data/mods/public/maps/random/unknown.js:       setSkySet(pickRandom(["cirrus", "cumulus", "sunny", "sunny 1", "mountainous", "stratus"]));
binaries/data/mods/public/maps/scenarios/fast_oasis.xml:                <SkySet>sunny 1</SkySet>
binaries/data/mods/public/maps/skirmishes/corinthian_isthmus_2p.xml:            <SkySet>sunny 1</SkySet>
binaries/data/mods/public/maps/skirmishes/temperate_roadway_2p.xml:             <SkySet>sunny 1</SkySet>

Not hard but a bit of work. Might be an easy "first patch" for someone if you can't be bothered.

[hyperion]

Edit: double post, the forum seems buggy. Anyway let's use this to post for some questions.

 

@Stan` @vladislavbelov

What I'm actually interested is some enlightenment wrt. already compressed dds and DXTn vs BCn

I have little clue about that topic, I read somewhere use BC7 always except BC5 for normal maps today.
Is that reasonable? How new needs hardware be to accept that. What can be used in 0ad?

Also somewhere I picked up closest equivalent DXT1 -> BC1, DXT3 -> BC2, DXT5 -> BC3

What do you want to do with the pre-compressed dds? I read DXT has low quality for size compare to BCn.

 

Edited September 6, 2021 by hyperion

[Stan`]

Ah right! Thought the sky was in the PMP. Mostly worried about the terrain textures references which should be in the PMP files.

[hyperion]

git grep "grass b soft dirt 50"

binaries/data/mods/public/art/terrains/grass/grass b soft dirt 50.xml:<?xml version="1.0" encoding="utf-8"?> <terrain> <textures> <texture name="baseTex" file="types/grass b soft dirt 50.dds"/>  </textures> <material>terrain_base.xml</material> </terrain>
binaries/data/mods/public/maps/random/ngorongoro.js:// ["dirta","savanna_wash_a","savanna_dirt_b","savanna_riparian_bank","savanna_grass_b","grass b soft dirt 50","grass1_spring","grass_field","grass1_spring","savanna_grass_a_wetseason","savanna_grass_b_wetseason","savanna_grass_a","new_savanna_grass_a","new_savanna_grass_b","new_savanna_grass_c","steppe_grass_dirt_66","peat_temp"];

 

[Stan`]

Need to tell grep to look in binary files

[vladislavbelov]

18 minutes ago, hyperion said:

What I'm actually interested is some enlightenment wrt. already compressed dds and DXTn vs BCn

DXT uses BC or it's an alias for BC (as well as S3TC).

19 minutes ago, hyperion said:

I have little clue about that topic, I read somewhere use BC7 always except BC5 for normal maps today.
Is that reasonable? How new needs hardware be to accept that. What can be used in 0ad?

We can't AFAICS. ARB_texture_compression_bptc is supported only by ~83% of players: https://feedback.wildfiregames.com/results/gl/extensions/.

22 minutes ago, hyperion said:

What do you want to do with the pre-compressed dds? I read DXT has low quality for size compare to BCn.

Currently we don't use them for some meaningful reason. But we could. Usually a converter is written for different kinds of images. So we could use some custom algorithm for a specific texture to have less artifacts.

[hyperion]

@vladislavbelov, thanks.

All those formats which are the same are confusing when getting into it at first.

So let's see if I got it right,  bc1-5 only for 0ad for now, dxt1 -> bc1, dxt3 -> bc2, dxt5 ->bc3, ati2 -> bc5

 

Some of my progress:

• Did resize png of size 1xsomething to 2xsomething and no longer see segfaults.
• When starting 0ad it fails at tex_dds.cpp:562, looks like mipmap is problematic.
• Built framework with mingw, only needed minor changes. So no issues from that angle.

[hyperion]

@vladislavbelov

Investigated mipmaps

• framework uses min(w,h) insted of max(w,h) for mipmap min size / count while generating
• framework always sets DDSD_MIPMAPCOUNT != 0 (1 for no mipmap)
• already compressed dds need processing or pyrogenesis will chock due to not supported texture format. Converted them to png so I can feed them to compressonator.

Also tex_dds.cpp only accepts DXT1 DXT3 DXT5, so no BC5 for normal maps, using BC3 instead.

 

@Stan`

Found out what's wrong with metal_desert_small_a.dds, file reports DOS/MBR boot sector, so obviously corrupt dds file. Might want to remove or replace it.

 

Now that I have textures that can be processed with compressonator and pass sanity checks, a few questions wrt packaging:

On 06/09/2021 at 8:29 AM, Stan` said:

You need to pass -mod=mod and -md=public too

The problem with -mod=mod is you need to have mod already installed where pyrogenesis can find it, doing this during build is not possible, as you are only about to install it. Would have to fix archive build.

Can't find the purpose of -md=public in source nor any documentation what it should do, can you elaborate?

 

On 05/09/2021 at 10:07 PM, Stan` said:

To convert a model from DAE to PSA you need to have the public mod loaded with the skeletons

To build the public.zip I need public mod loaded? Typo or real?

[Stan`]

44 minutes ago, hyperion said:

The problem with -mod=mod is you need to have mod already installed where pyrogenesis can find it, doing this during build is not possible, as you are only about to install it. Would have to fix archive build.

Can't find the purpose of -md=public in source nor any documentation what it should do, can you elaborate?

Yes, you need the mod mod, and the public mod, compressed or not in the binaries/data/mods folder.

md is me messing up with my phone, it's obviously -mod=public  

45 minutes ago, hyperion said:

To build the public.zip I need public mod loaded? Typo or real?

Real. Same for all mods using skeletons meshes. 

47 minutes ago, hyperion said:

Found out what's wrong with metal_desert_small_a.dds, file reports DOS/MBR boot sector, so obviously corrupt dds file. Might want to remove or replace it.

Mmmh that file works fine in gimp...

metal_desert_small_a.xcf

 

 

[hyperion]

2 hours ago, Stan` said:

Real. Same for all mods using skeletons meshes. 

Guess I write and archive builder frontend which doesn't expect anything to be installed already. Thanks for clarifying.

 

2 hours ago, Stan` said:

Mmmh that file works fine in gimp...

nvtt didn't chock either. Maybe best effort to recover from broken file headers or so.  Maybe someone else on linux can chime in as to what the file command reports for them for metal_desert_small_a.dds.

Maybe export as dds and I see if it's fixed for me. (Where did you get a working gimp-dds plugin from, thought that was long dead)

[Freagarach]

3 minutes ago, hyperion said:

Maybe someone else on linux can chime in as to what the file command reports for them for metal_desert_small_a.dds.

$ file metal_desert_small_a.dds
metal_desert_small_a.dds: DOS/MBR boot sector; partition 1 : ID=0xff, active 0x8a, start-CHS (0x3ff,254,63), end-CHS (0x3ff,255,63), startsector 3587178495, 3468469193 sectors; partition 4 : ID=0xff, active 0xe2, start-CHS (0x3ff,14,63), end-CHS (0x3ff,255,63), startsector 4294180863, 4278223684 sectors

 

[hyperion]

Phew, not just me

[Stan`]

5 minutes ago, hyperion said:

Guess I write and archive builder frontend which doesn't expect anything to be installed already. Thanks for clarifying.

I suppose we need it for PSA conversion, because it only uses the list of bones specified in that XML file. I once made a packaging mod https://github.com/StanleySweet/package_mod/tree/master/art/skeletons to remove the need from those extra mods. It contains textures.xml files for instructions on what to convert, and the skeletons . of course, it won't work if you have new skeletons e.g packaging hyrule or something;

WSL2: First file is the "corrupted" one.

file metal_desert_small_*
metal_desert_small_a.dds:                      DOS/MBR boot sector; partition 1 : ID=0xff, active 0x8a, start-CHS (0x3ff,254,63), end-CHS (0x3ff,255,63), startsector 3587178495, 3468469193 sectors; partition 4 : ID=0xff, active 0xe2, start-CHS (0x3ff,14,63), end-CHS (0x3ff,255,63), startsector 4294180863, 4278223684 sectors
metal_desert_small_a.dds.b22d835129a068cf.dds: Microsoft DirectDraw Surface (DDS): 128 x 128, compressed using DXT1

Here is a working version, (converted using NVTT)

metal_desert_small_a.dds

 

27 minutes ago, hyperion said:

Where did you get a working gimp-dds plugin from, thought that was long dead)

It's built-in as of 2.0.10 IIRC.

 

[hyperion]

57 minutes ago, Stan` said:

Here is a working version, (converted using NVTT)

Now it's compressed and my script will pick it up and convert to png so compressonator can open it.

The script

• converts 16 bit png to 8 bit depth png
• resizes dimension 1 textures
• converts compressed dds to png

All stuff that needs fixing in compressonator obviously, don't fancy working around this in 0ad proper.

Still the corrupt dds should be replaced in svn, compressed or not.

 

--

 

As for generating the public zip, tried with -mod=mod -mod=public, tried with replacing rdb fcollada with 0ad fcollada, tried with system installed pyrogenesis (pristine a25b, no changes by me), tried with mod-packer binary (source below). I always get those skeleton related errors. What am I missing?

Btw., source/tools/dist/build-archives.sh doesn't add public either, what do you use to create the release tarball?

 

#include "lib/os_path.h"
#include "lib/timer.h"
#include "ps/ArchiveBuilder.h"
#include "ps/GameSetup/CmdLineArgs.h"
#include "ps/XML/Xeromyces.h"

#include <vector>

void RestartEngine() {}
bool g_GameRestarted = false;
void QuitEngine() {}
bool IsQuitRequested() { return false; }
void StartAtlas() {}

int main (int argc, const char* argv[])
{
        timer_Init();

        CmdLineArgs args(argc, argv);

        OsPath mod(args.Get("input"));
        OsPath zip(args.Get("output"));
        OsPath cache(args.Get("cache"));

        CArchiveBuilder builder(mod, cache);

        std::vector<CStr> mods = args.GetMultiple("mod");
        for (size_t i = 0; i < mods.size(); ++i)
                builder.AddBaseMod(OsPath(mods[i]));

        CXeromyces::Startup();

        builder.Build(zip, args.Has("compress"));

        CXeromyces::Terminate();
        return 0;
}

 

[Stan`]

18 minutes ago, hyperion said:

As for generating the public zip, tried with -mod=mod -mod=public, tried with replacing rdb fcollada with 0ad fcollada, tried with system installed pyrogenesis (pristine a25b, no changes by me), tried with mod-packer binary (source below). I always get those skeleton related errors. What am I missing?

Btw., source/tools/dist/build-archives.sh doesn't add public either, what do you use to create the release tarball?

Indeed we seem to use only -mod=mod

We use build-archives.sh you can see the code here https://trac.wildfiregames.com/browser/ps/trunk/build/jenkins/pipelines/macos-all-bundles.Jenkinsfile

 

Is it possible that the executable can't read the xml file for some reason ? Maybe a mingw bug with crlf or something ? I didn't test but maybe the collada.dll file doesn't work (I wouldn't have noticed using the release data)

Where is the public mod located ? Could be a matter of relative paths. At this point I'm just spitballing.

 

 

[maroder]

28 minutes ago, hyperion said:

The script

• converts 16 bit png to 8 bit depth png

Not sure if it matters but there are also some 32 bit png e.g.

https://trac.wildfiregames.com/browser/ps/trunk/binaries/data/mods/public/art/textures/ui/pregame/backgrounds/carthage1_3.png

And just out of curiosity: seems like a bit of an Information loss? Or does this not matter in this case?

[hyperion]

14 minutes ago, Stan` said:

We use build-archives.sh you can see the code here https://trac.wildfiregames.com/browser/ps/trunk/build/jenkins/pipelines/macos-all-bundles.Jenkinsfile

So works on macos, could it be broken on linux only? Will try the update-workspace et al path.

 

20 minutes ago, Stan` said:

Is it possible that the executable can't read the xml file for some reason ? Maybe a mingw bug with crlf or something ? I didn't test but maybe the collada.dll file doesn't work (I wouldn't have noticed using the release data)

Not using mingw for my compressonator experiments. Also 0ad installed by package manager has the same issue.

 

13 minutes ago, maroder said:

Not sure if it matters but there are also some 32 bit png e.g.

Compressonator is fine with those. Also compressonator is working as replacement already. The issue I face is I can't build public.zip due to skeleton xml files not being found / not readable for some reason. Candidates for issue are pyrogenesis, fcollada, libxml first and foremost. But thanks anyway.

[Stan`]

12 minutes ago, hyperion said:

Not using mingw for my compressonator experiments. Also 0ad installed by package manager has the same issue.

It used to be the more reliable platform. I will test on my Ubuntu when I get the chance. But I've never had any issue on any of the three platforms with a clean svn.

[Stan`]

Okay, so first fun fact the script is marked as a POSIX shell script by the shebang (#!/bin/sh) but uses arrays pushd and popd which are not compatible among other features.

Needs to be fixed.

Then run the script noticed the skeleton errors. Other fun fact they are present in the packaging script just nobody noticed them (which ran on macOS)

https://jenkins.wildfiregames.com/blue/rest/organizations/jenkins/pipelines/macos-all-bundles/runs/131/nodes/30/steps/31/log/?start=0

ERROR: art/animation/other/hele_trireme_fore_sail_idle.dae: Assertion not satisfied (line 393): failed requirement "recognised skeleton structure"
ERROR: art/animation/other/hele_trireme_fore_sail_move.dae: Assertion not satisfied (line 393): failed requirement "recognised skeleton structure"
ERROR: art/animation/others/camel_rein_attack_melee.dae: Assertion not satisfied (line 393): failed requirement "recognised skeleton structure"
ERROR: art/animation/others/camel_rein_death.dae: Assertion not satisfied (line 393): failed requirement "recognised skeleton structure"
ERROR: art/animation/others/camel_rein_drop.dae: Assertion not satisfied (line 393): failed requirement "recognised skeleton structure"
ERROR: art/animation/others/camel_rein_idle.dae: Assertion not satisfied (line 393): failed requirement "recognised skeleton structure"
ERROR: art/animation/others/camel_rein_run.dae: Assertion not satisfied (line 393): failed requirement "recognised skeleton structure"
ERROR: art/animation/others/camel_rein_sit_drop.dae: Assertion not satisfied (line 393): failed requirement "recognised skeleton structure"
ERROR: art/animation/others/camel_rein_trot.dae: Assertion not satisfied (line 393): failed requirement "recognised skeleton structure"
ERROR: art/meshes/gaia/tree_apple_top_e.dae: Assertion not satisfied (line 570): failed requirement "mesh has single set of polygons"
ERROR: art/meshes/gaia/tree_apple_top_f.dae: Assertion not satisfied (line 570): failed requirement "mesh has single set of polygons"
ERROR: art/meshes/gaia/tree_apple_top_g.dae: Assertion not satisfied (line 570): failed requirement "mesh has single set of polygons"
ERROR: art/meshes/gaia/tree_apple_top_h.dae: Assertion not satisfied (line 570): failed requirement "mesh has single set of polygons"
WARNING: art/meshes/gaia/tree_apple_trunk_c.dae: FCollada 105: Polygons is empty.
ERROR: art/meshes/gaia/tree_apple_trunk_c.dae: Exception caught while parsing a COLLADA document from file.
WARNING: art/meshes/gaia/tree_apple_trunk_d.dae: FCollada 105: Polygons is empty.
ERROR: art/meshes/gaia/tree_apple_trunk_d.dae: Exception caught while parsing a COLLADA document from file.
ERROR: art/meshes/props/camel_rein.dae: Assertion not satisfied (line 393): failed requirement "recognised skeleton structure"
ERROR: art/meshes/props/hele_trireme_front_sail.dae: Assertion not satisfied (line 393): failed requirement "recognised skeleton structure"
ERROR: art/meshes/skeletal/m_tunic_long.dae: Assertion not satisfied (line 393): failed requirement "recognised skeleton structure"

So I ran it on linux

user@user-Surface-Pro-2  ~/0ad  grep -ir error file.log 
ERROR: art/animation/other/hele_trireme_fore_sail_idle.dae: Assertion not satisfied (line 393): failed requirement "recognised skeleton structure"
ERROR: art/animation/other/hele_trireme_fore_sail_move.dae: Assertion not satisfied (line 393): failed requirement "recognised skeleton structure"
ERROR: art/animation/others/camel_rein_attack_melee.dae: Assertion not satisfied (line 393): failed requirement "recognised skeleton structure"
ERROR: art/animation/others/camel_rein_death.dae: Assertion not satisfied (line 393): failed requirement "recognised skeleton structure"
ERROR: art/animation/others/camel_rein_drop.dae: Assertion not satisfied (line 393): failed requirement "recognised skeleton structure"
ERROR: art/animation/others/camel_rein_idle.dae: Assertion not satisfied (line 393): failed requirement "recognised skeleton structure"
ERROR: art/animation/others/camel_rein_run.dae: Assertion not satisfied (line 393): failed requirement "recognised skeleton structure"
ERROR: art/animation/others/camel_rein_sit_drop.dae: Assertion not satisfied (line 393): failed requirement "recognised skeleton structure"
ERROR: art/animation/others/camel_rein_trot.dae: Assertion not satisfied (line 393): failed requirement "recognised skeleton structure"
ERROR: art/meshes/gaia/tree_apple_top_e.dae: Assertion not satisfied (line 570): failed requirement "mesh has single set of polygons"
ERROR: art/meshes/gaia/tree_apple_top_f.dae: Assertion not satisfied (line 570): failed requirement "mesh has single set of polygons"
ERROR: art/meshes/gaia/tree_apple_top_g.dae: Assertion not satisfied (line 570): failed requirement "mesh has single set of polygons"
ERROR: art/meshes/gaia/tree_apple_top_h.dae: Assertion not satisfied (line 570): failed requirement "mesh has single set of polygons"
ERROR: art/meshes/gaia/tree_apple_trunk_c.dae: Exception caught while parsing a COLLADA document from file.
ERROR: art/meshes/gaia/tree_apple_trunk_d.dae: Exception caught while parsing a COLLADA document from file.
ERROR: art/meshes/props/camel_rein.dae: Assertion not satisfied (line 393): failed requirement "recognised skeleton structure"
ERROR: art/meshes/props/hele_trireme_front_sail.dae: Assertion not satisfied (line 393): failed requirement "recognised skeleton structure"
ERROR: art/meshes/skeletal/m_tunic_long.dae: Assertion not satisfied (line 393): failed requirement "recognised skeleton structure"
 user@user-Surface-Pro-2  ~/0ad  grep -ir warning file.log        
WARNING: art/meshes/gaia/tree_apple_trunk_c.dae: FCollada 105: Polygons is empty.
WARNING: art/meshes/gaia/tree_apple_trunk_d.dae: FCollada 105: Polygons is empty.

I'm voluntarily excluding the iccp profile errors as they have been there for a while and so far have not caused much damage.

But still any test of the Ptolemies camels or a naval battle should have caused errors right? Well, in theory yes... but

perl checkrefs.pl --check-unused 
...
Unused file 'public/art/animation/other/hele_trireme_fore_sail_idle.dae'
Unused file 'public/art/animation/other/hele_trireme_fore_sail_move.dae' 
Unused file 'public/art/animation/others/camel_rein_attack_melee.dae'
Unused file 'public/art/animation/others/camel_rein_death.dae'
Unused file 'public/art/animation/others/camel_rein_drop.dae'
Unused file 'public/art/animation/others/camel_rein_idle.dae'
Unused file 'public/art/animation/others/camel_rein_run.dae'
Unused file 'public/art/animation/others/camel_rein_sit_drop.dae'
Unused file 'public/art/animation/others/camel_rein_trot.dae'
Unused file 'public/art/meshes/gaia/tree_apple_top_e.dae'
Unused file 'public/art/meshes/gaia/tree_apple_top_f.dae'
Unused file 'public/art/meshes/gaia/tree_apple_top_g.dae'
Unused file 'public/art/meshes/gaia/tree_apple_top_h.dae'
Unused file 'public/art/meshes/gaia/tree_apple_trunk_c.dae'
Unused file 'public/art/meshes/gaia/tree_apple_trunk_d.dae'
Unused file 'public/art/meshes/props/camel_rein.dae'
Unused file 'public/art/meshes/props/hele_trireme_front_sail.dae'
Unused file 'public/art/meshes/skeletal/m_tunic_long.dae'

Guess who got lucky as hell?

tldr; Errors are to be expected. They should be fixed.  If you don't end up with a zip or get different ones eventually then there is another issue.

[hyperion]

So a zip (ignoring the errors) was created and I ran 0ad with the "new" textures.

Some font(s) got messed up, some of the textures auto resized to dimension 2 from 1 look like zebras but otherwise the textures seem of better quality, less washed out. Quality setting used 0.5 (0 lowest, 1 highest), maybe 1 looks even better, might just take forever to compress.

 

Stumbled over another issue, some textures have png and dds in tree, the dds should probably be deleted in this case.

binaries/data/mods/public/art/textures/skins/props/cape_hd_black.dds
binaries/data/mods/public/art/textures/skins/props/kart_sail_2.dds
binaries/data/mods/public/art/textures/skins/props/kart_sail_3.dds
binaries/data/mods/public/art/textures/skins/gaia/tree_cypress_a.dds
binaries/data/mods/public/art/textures/skins/props/kart_sail_1.dds
binaries/data/mods/public/art/textures/skins/props/iber_sail_b.dds
binaries/data/mods/public/art/textures/skins/gaia/farming_barley_harvest_a.dds
binaries/data/mods/public/art/textures/skins/gaia/farming_wheat_harvest_a.dds
binaries/data/mods/public/art/textures/skins/gaia/farming_wheat_harvest_b.dds
binaries/data/mods/public/art/textures/terrain/types/desert_forestfloor_palms.dds
binaries/data/mods/public/art/textures/skins/props/celt_prop_1.dds
binaries/data/mods/public/art/textures/terrain/types/road_roman.dds
binaries/data/mods/public/art/textures/skins/structural/rome_ram.dds
binaries/data/mods/public/art/textures/skins/structural/kart_trireme.dds
binaries/data/mods/public/art/textures/skins/structural/iber_struct.dds
binaries/data/mods/public/art/textures/skins/structural/hele_trireme.dds
binaries/data/mods/public/art/textures/skins/structural/hele_struct_b.dds
binaries/data/mods/public/art/textures/skins/skeletal/pers_isp_e_1.dds
binaries/data/mods/public/art/textures/skins/skeletal/rome_ijv_e.dds
binaries/data/mods/public/art/textures/skins/skeletal/mace_bronzeshield_pikeman_a.dds
binaries/data/mods/public/art/textures/skins/props/head/celt_caratacos.dds

 

[Stan`]

Mmh, some of those are different, so will have to be careful  Thanks for digging it up though  

[asterix]

Can we not use something like from this page https://pippin.gimp.org/ in 0ad to replace some dependencies?

[Stan`]

Not really no, seems unrelated.

[hyperion]

28 minutes ago, asterix said:

Can we not use something like from this page https://pippin.gimp.org/ in 0ad to replace some dependencies?

What do you want to replace and why?

 

Anyway, all I needed was to mogrify some assets and run fontbuilder using always colour in render options and have now a working replacement for nvtt, so no longer restricted to releases respectively already packed mods. Also created some tickets with compressonator to see if what we lack we can get in a future release.

I used a somewhat brute method to get compressonator support, I'm sure @vladislavbelov had something more complete in mind For instance I don't offer modders dozen of options on how to treat their textures. For my layman eye they look better on average anyway. I'm not even sure what all those available tuning parameters are about, so everything default.

So now I can build 0ad for windows from ground up using mingw, the only things missing are wseh and stackwalking for dumping crashes. Other than that it's fully functional as far as I can tell. If there are bugs they need first be found.