A new git-based development environment

[Freagarach]

17 hours ago, Itms said:

While we are breaking paths:

I'm not sure wfg/0ad is a good repo path on Gitea. It's inconsistent with github and gitlab (where it's 0ad/0ad). I see that blender made the same decision and called the organization "blender" on their own gitea.

So I'm pondering whether to rename the "wfg" organization to "0ad" (I'll probably keep "Wildfire Games" as the real name).

Does that sound good? I don't like it much but it may objectively be better.

But their foundation is actually called ,,Blender'' so that is something different in my opinion. ^^ I'd stick with ,,wfg'' or any such equivalent. Unless the plan is to have separate organisations for 0 A.D. (e.g. 0ad/empires-ascendent, 0ad/empires-besieged etc.) and pyrogenesis (e.g. pyrogenesis/pyrogenesis; pyrogenesis/tool-do-do-stuff-with-pyro-a etc.).

[Itms]

29 minutes ago, Freagarach said:

But their foundation is actually called ,,Blender'' so that is something different in my opinion. ^^ I'd stick with ,,wfg'' or any such equivalent. Unless the plan is to have separate organisations for 0 A.D. (e.g. 0ad/empires-ascendent, 0ad/empires-besieged etc.) and pyrogenesis (e.g. pyrogenesis/pyrogenesis; pyrogenesis/tool-do-do-stuff-with-pyro-a etc.).

Having a git forge would be a good opportunity to start versioning some of our staff information, so I went ahead and made the change: that way, repositories associated with the development of the game go to 0ad, while we have the wfg organization for actual organizational matters. For instance, infrastructure documentation for the server as asked by Dunedan. SPI-related stuff should also go under wfg/

We should also really look into improving our websites, and thus we should have, when this project starts, a wfg/website repo for wildfiregames.com and a 0ad/website repo for play0ad.com.

Does that sound logical?

[Stan`]

There is some stuff on the Github for the website. Not sure how relevant it still is though.

I wonder wether we should version feedback, docs and other websites we have.

[Dunedan]

47 minutes ago, Itms said:

Having a git forge would be a good opportunity to start versioning some of our staff information, so I went ahead and made the change: that way, repositories associated with the development of the game go to 0ad, while we have the wfg organization for actual organizational matters. For instance, infrastructure documentation for the server as asked by Dunedan. SPI-related stuff should also go under wfg/

We should also really look into improving our websites, and thus we should have, when this project starts, a wfg/website repo for wildfiregames.com and a 0ad/website repo for play0ad.com.

Does that sound logical?

Where would something like https://github.com/0ad/lobby-infrastructure/ fit with that?

[Itms]

11 minutes ago, Dunedan said:

Where would something like https://github.com/0ad/lobby-infrastructure/ fit with that?

Since that is for the lobby of the 0ad game, it would go to 0ad/lobby-infrastructure which would be consistent with GitHub.

[Dunedan]

There is a big topic left I want to discuss and that's the continued use of SVN.

Up until yesterday I was under the impression that the goal of the migration is to replace the use of SVN for all development of 0ad (while keeping read-only SVN access). I obviously should've read the documentation in detail earlier, but after doing so and after a lot of questions answered by @Itms and @Stan` (thanks again for that) I believe I understand the planned setup and its motivation now.

Unfortunately I feel like the migration is one step forward and one step back. Let me explain why: Right now when working on patches and trying out development snapshots I use the git-mirror of 0ad on Github (https://github.com/0ad/0ad). The only time I have to use SVN is when I want to actually commit something.

With the planned setup that'd not be the case anymore and I'd need SVN to even compile 0ad, because the contents of the "source-libs"/"windows-libs" SVN repositories are needed for that. I'd also need SVN to test different locals with a development snapshot, as translations aren't planned to be included in the git repository anymore.

I believe it's fair to say the new setup is much more complex. Previously it was a single SVN repository, with an 1:1 git mirror, now it's one git-repository and three SVN repositories, which share different subsets of code and/or binaries. The only one of these repositories which contains everything (albeit with more coarse-grained commits) is the "nigthly-build" SVN repository. As that's read-only, every developer will have to interact with git and SVN in future.

While one of the big benefits of Gitea is the possibility to use pull requests, that won't be possible for changes to the  "source-libs" and "windows-libs" SVN repositories, as Gitea doesn't support SVN.

Learning about these constraints pulled me down a bit, but let's see if we can get some ideas for improvement out of that.
Going back to square one, what are our expectations for the migration? Mine are:

1. All development for 0ad happens in git afterwards.
2. A comfortable web-frontend is available for browsing git repositories, collecting issues, discussing and merging patches and managing documentation.
3. Build artifacts aren't included in the source code repository anymore.
4. A 1:1 SVN-mirror is available (essentially the equivalent to the git-mirror we had so far).

Based on that, here is what I'd suggest:

• keep the planned 0ad git repository and the use of Gitea
• commit translations and translation credits into the 0ad git repository again (I don't see a good reason why translations should be handled differently from assets or "programming.json" and having them in git makes everything easier)
• integrate the contents of the "source-libs" SVN repository into the 0ad git repository again
• instead of using a "windows-libs" SVN repository, put its contents into a git repository, utilizing Git LFS, and include it in the 0ad git repository as submodule
• add an additional git repository for build artifacts, utilizing Git LFS, and include it in the 0ad git repository as submodule
• make an SVN-mirror available with git-as-svn (https://github.com/git-as-svn/git-as-svn)

By doing all of the above there would be no SVN repository in use anymore, but everything would still be accessible through SVN. The contents of the 0ad git repository and the SVN mirror would be identical and the mirror would always return the most recent content from the git repository. Changes for all repositories could be handled via pull requests in Gitea. While the git repository would be larger than with the currently planned setup (because of the added translations and source-libs), I believe that's warranted given the history of the project and the size of the code base. Partial checkouts using SVN would still be possible and the necessary storage space on the server might even be lower, as git-as-svn is just a bridge between git and SVN.

What do you think about that? What did I miss? Do my ideas even make sense?

[Stan`]

Looking at the scripts here https://svn.itms.ovh/nightly-build/trunk/libraries/ I'm not sure their behavior can be replicated using submodules, but I could be wrong.

1 hour ago, Dunedan said:

Going back to square one, what are our expectations for the migration? Mine are:

One of the expectations that some people including me had at the time was that we would be able to split the engine from the game because a few people wanted to contribute to engine solely without having to clone the big repo, and that's partially solved by LFS, but that hasn't happened. Currently atlas is still heavily depending on the public mod. All the other needed engine resources are in the mod mod now.

Another one was to lower the contribution entry with Pull requests and no longer requiring a third party too like arcanist.

 

[Itms]

3 hours ago, Dunedan said:

What do you think about that? What did I miss? Do my ideas even make sense?

Thanks for the detailed point of view, I'll try to address that from my perspective. (I read on IRC about feeling sorry about giving me more work, don't be! I was aware of that possibility since I worked on my own on this migration. Some parts of it are a bit opinionated: I tried to setup my own ideal setup. That said, I am not against anything in principle, but I'm not going to overwork myself: whenever I am convinced by a proposal, I'll perform it with pleasure, but if I'm not, I'll let someone else do it)

3 hours ago, Dunedan said:

Going back to square one, what are our expectations for the migration?

My number one expectation is: ditch Phabricator (well, they ditched us first, as implodedok put it). Stan hinted at that above. We need to continue having a collaboration platform, and Phabricator was the only big one that provided support for SVN. But apart from that, I have nothing against SVN. It's still a very usable tool, with many upsides, and as long as we had a forge allowing us to collaborate, we could do without branches and PRs. But it's not the case anymore, so we have to switch to git.

You (and others) dream of stopping using SVN entirely, but I also had to convince people who don't want to use git at all. I tried to find a middle ground here and tried to exploit the main advantages of both tools, and I'm quite happy with the result.

3 hours ago, Dunedan said:

The only time I have to use SVN is when I want to actually commit something.

With the planned setup that'd not be the case anymore and I'd need SVN to even compile 0ad

I disagree with that part. You'd need to have subversion installed on your system (just like now), but you wouldn't have to interact with it anymore. You'd just run the libraries/build-source-libs.sh script and be done with it.

3 hours ago, Dunedan said:

I'd also need SVN to test different locals with a development snapshot

No, I added a source/tools/i18n/get-nightly-translations.sh script to export the latest translations into your git clone. It was a valid concern though, bb was the one to bring it to my attention first.

3 hours ago, Dunedan said:

While one of the big benefits of Gitea is the possibility to use pull requests, that won't be possible for changes to the  "source-libs" and "windows-libs" SVN repositories, as Gitea doesn't support SVN.

That is true, but the majority of devs will not have to interact with those repositories. The upgrade of libraries is fundamentally non-collaborative: me and Stan are the only ones who may upgrade Windows libs these days; you can add wraitii and s0600204 to the list for SpiderMonkey upgrades and that's all. Being one of the involved devs, I agree that the new setup is not a huge improvement over what we did on SVN, but it feels way more organized and streamlined to me.

4 hours ago, Dunedan said:

• integrate the contents of the "source-libs" SVN repository into the 0ad git repository again
• instead of using a "windows-libs" SVN repository, put its contents into a git repository, utilizing Git LFS, and include it in the 0ad git repository as submodule

I never used git submodules, so basically: I wouldn't dare trying to setup that I have nothing against it in principle, but I had not thought of submodules before yesterday. If I'm not mistaken, this is a change that could be performed in the future without rewriting history, right? It could be an incremental improvement over the future setup, that could be performed by a knowledgeable contributor. Would that be acceptable to you?

4 hours ago, Dunedan said:

add an additional git repository for build artifacts, utilizing Git LFS, and include it in the 0ad git repository as submodule

same as above, but I believe the directory structure of the repository would make that much harder it would create similar problems to the engine/mods split I think, I'm not sure.

4 hours ago, Dunedan said:

commit translations and translation credits into the 0ad git repository again (I don't see a good reason why translations should be handled differently from assets or "programming.json" and having them in git makes everything easier)

That, on the other hand, I am against. The history of po/pot files in the git repo, when I started my experiment, was HUGE. It was several times larger than the current size (LFS blobs excluded) that I got to. (I can't remember how much, but off the top of my head 600MiB)

And even that is not satisfactory because you only get new translations twice a week. With the nightly-build system, testers obtain everything "as in a release" daily, which is a bit improvement! Of course, committing translations daily to the git repo would be possible, but the size would skyrocket.

Generally speaking, I'm also a bit adverse to letting a bot commit to the source repo. It makes more sense to me to have a git repo managed entirely by humans, and then a repo generated by the CD system, which provides a ready-to-play state to testers. Let me link once again to the glorious diagram I'm really proud of in which it looks like concerns are cleanly separated.

But the main thing that convinced me to strip the translations from the git repo is the SPIR-V shaders for Vulkan. Right now we are blocked with a Vulkan support that depends on a mod which cannot be committed because of its size, the generated nature of its contents, and the huge amount of time/computing needed to generate it (and I believe that's part of the reasons for that stalled A27, as including the mod would complicate the release process). Vulkan cannot be tested out of the box by users without some knowledge.

I want to demonstrate next week (unfortunately I can't show that to support my claim yet) that this can be fixed by my nightly-build approach. The git repo does not have to contain those large and ever-changing shader files. Instead, the nightly build generation can provide those files daily, with the CD system taking care of generating them, and we'd have them ready for releases in advance. I only have to make sure that the generation can be efficient (we mustn't spend 5 hours generating them every night, but only when needed), and that will be my task for the next week.

---

In a nutshell:

1. I don't think SVN is so bad that we should avoid using it entirely.

2. I am open to replacing my source-libs and windows-libs SVN repos with git submodules, but that can be done in the future, and I'd be happy if someone else would do it.

3. I believe that a nightly build for testers and for generating release bundles, generated from a lightweight git repo for the developers, is a superior approach than a 1:1 SVN mirror of a (very?) large git repo.

[Itms]

1 hour ago, Itms said:

The upgrade of libraries is fundamentally non-collaborative: me and Stan are the only ones who may upgrade Windows libs these days; you can add wraitii and s0600204 to the list for SpiderMonkey upgrades and that's all. Being one of the involved devs, I agree that the new setup is not a huge improvement over what we did on SVN, but it feels way more organized and streamlined to me.

By the way, since I also read about that on IRC:

It is very likely that my urgent work after the migration will be a (partial or full) SpiderMonkey upgrade. Having handled several of those now, I am thrilled (that's not an exaggeration) about performing that future upgrade under the new git setup I designed, mostly because CI will finally be able to handle it. I think I'm a good judge of whether the setup is adapted to performing library upgrades.

If I stumble on rough edges, that will be the opportunity to address them, and if I don't, it will be a good example in the repo history of how to perform such an upgrade in the new setup.

To reiterate, maybe it would be even better with submodules, but I don't know enough about them to design a submodule-based setup myself. I'm not against using them in the future.

[Stan`]

On 12/07/2024 at 7:08 PM, Itms said:

But the main thing that convinced me to strip the translations from the git repo is the SPIR-V shaders for Vulkan. Right now we are blocked with a Vulkan support that depends on a mod which cannot be committed because of its size, the generated nature of its contents, and the huge amount of time/computing needed to generate it (and I believe that's part of the reasons for that stalled A27, as including the mod would complicate the release process). Vulkan cannot be tested out of the box by users without some knowledge.

I'd like to add some precisions to this. (@vladislavbelov might correct me)

Vulkan shaders (Also called spir-v shaders) are artifacts produced by compiling the GLSL code (text files) into SPIR-V files (binary files). They only should get rebuilt if new shaders are added or old are modified. That compilation process takes one hour on a fast machine, and about 5 on the CI. The produced output also differs depending on the version of GLSLC (This caused a few disagreements). There is no incremental build.

So you might wonder if we're doing something stupid when compiling them. And actually it's the converse. We're optimizing the number of files we have to generate using these files: https://releases.wildfiregames.com/spir-v/

You see, unlike GLSL which support conditional macro execution using #if and the like, spirv does not and you have to compile every single permutation of those macros. (and there are a lot) Those json files only list the ones that are actually used in order to prevent generating too much useless code. Bruteforcing the generation would add a couple gigabytes of shaders.

When Vulkan was first added the generation script and the relevant rule files were not present anywhere and so we couldn't automate the process. That's one of the things that caused a delay because I didn't want to go the easy route and just dump the spir-v mod as is in the release bundles.

In the future we probably should rebuild the shaders only if something in binaries/data/mods/**/shaders changed, or if the rules file changed (could store a hash of the file or something)

That's still a significant size. The mod itself has the advantage of being compressed

 

[Dunedan]

On 12/07/2024 at 7:08 PM, Itms said:

You (and others) dream of stopping using SVN entirely, but I also had to convince people who don't want to use git at all. I tried to find a middle ground here and tried to exploit the main advantages of both tools, and I'm quite happy with the result.

I don't mind if access via SVN continues to be possible, as long as contributors don't have to use it.

On 12/07/2024 at 7:08 PM, Itms said:

I disagree with that part. You'd need to have subversion installed on your system (just like now), but you wouldn't have to interact with it anymore. You'd just run the libraries/build-source-libs.sh script and be done with it.

Good point, it doesn't apply to contributing to library upgrades though.

On 12/07/2024 at 7:08 PM, Itms said:

No, I added a source/tools/i18n/get-nightly-translations.sh script to export the latest translations into your git clone.

So now the nightly-build SVN isn't a compilation of data from different source anymore, but feeds back into its source again (even though those changes aren't meant to be committed there). IMO cleaner approaches for that would be fetching translations directly from Transifex (which doesn't work, because that requires a Transifex API token not every contributor has), managing them in a separate repository and pulling them from there to the 0ad git repository and the nightly-build SVN repository or just storing them in the 0ad git repository.

On 12/07/2024 at 7:08 PM, Itms said:

That is true, but the majority of devs will not have to interact with those repositories. The upgrade of libraries is fundamentally non-collaborative: me and Stan are the only ones who may upgrade Windows libs these days; you can add wraitii and s0600204 to the list for SpiderMonkey upgrades and that's all.

Wouldn't that foster a culture where certain changes (library upgrades) are done without reviewing them, as there won't be a review tool for SVN anymore. Wouldn't it also make it pretty difficult for contributors without SVN write access to contribute to them?

On 12/07/2024 at 7:08 PM, Itms said:

I never used git submodules, so basically: I wouldn't dare trying to setup that I have nothing against it in principle, but I had not thought of submodules before yesterday. If I'm not mistaken, this is a change that could be performed in the future without rewriting history, right? It could be an incremental improvement over the future setup, that could be performed by a knowledgeable contributor. Would that be acceptable to you?

It should be no problem to do that later on. I'd obviously prefer having it from the start, but later on would work for me too.

On 12/07/2024 at 7:08 PM, Itms said:

same as above, but I believe the directory structure of the repository would make that much harder it would create similar problems to the engine/mods split I think, I'm not sure.

Yes, that probably needs some more consideration.

On 12/07/2024 at 7:08 PM, Itms said:

The history of po/pot files in the git repo, when I started my experiment, was HUGE. It was several times larger than the current size (LFS blobs excluded) that I got to. (I can't remember how much, but off the top of my head 600MiB)

And even that is not satisfactory because you only get new translations twice a week. With the nightly-build system, testers obtain everything "as in a release" daily, which is a bit improvement! Of course, committing translations daily to the git repo would be possible, but the size would skyrocket.

I believe the translations are already even larger than 600MiB. Updating them more often shouldn't increase their size much faster though, because while that would result in more commits, the amount of changes would stay more or less the same (only strings changed more than once between our twice-a-week updates would generate additional changes, but that would be true for the nightly-build SVN setup as well). As git only stores changed data chunks, those additional commits shouldn't do much regarding the repository size. With git, commits are cheap.

When looking at the trade-off between the additional size the translations cause vs. not having them in the git-repository & the added complexity of the nightly-build repository not being a 1:1 mirror of the git repository, I'd choose keeping them in git. People who pay particular attention to the size the data takes up on their hard drive, could still use SVN.

On 12/07/2024 at 7:08 PM, Itms said:

Generally speaking, I'm also a bit adverse to letting a bot commit to the source repo. It makes more sense to me to have a git repo managed entirely by humans, and then a repo generated by the CD system, which provides a ready-to-play state to testers.

What's the reason for you negative stance for bot commits to the source repository? Right now that sounds a bit arbitrary to me.

On 12/07/2024 at 7:08 PM, Itms said:

But the main thing that convinced me to strip the translations from the git repo is the SPIR-V shaders for Vulkan.

I do get the reason for not committing the SPIR-V shaders to the 0ad git repository, but what's the connection to the translations here?

On 12/07/2024 at 7:08 PM, Itms said:

2. I am open to replacing my source-libs and windows-libs SVN repos with git submodules, but that can be done in the future, and I'd be happy if someone else would do it.

Right now you're one of the very few people who can actually change something about it. I'd be happy to support with that though.

On 12/07/2024 at 7:08 PM, Itms said:

3. I believe that a nightly build for testers and for generating release bundles, generated from a lightweight git repo for the developers, is a superior approach than a 1:1 SVN mirror of a (very?) large git repo.

I believe nobody proposed a single large git repository, with large as in "contains everything the current SVN repository includes" and we all agree that removing built artifacts from the 0ad repository is a good idea. Using submodules we could make it appear as it'd be one large repository though, which would remove the need to use SVN during development and would make having a 1:1 SVN mirror easier.

To sum up, I still believe a 0ad git repository (+ submodules for git repositories with compiled artifacts) with a 1:1 SVN mirror would, if possible, be a great win to the reduce the complexity that the new setup introduces.

[Itms]

2 hours ago, Dunedan said:

So now the nightly-build SVN isn't a compilation of data from different source anymore, but feeds back into its source again (even though those changes aren't meant to be committed there).

If the exported translations are not committed, that feedback does not actually happen... Do you think it could create actual issues?

2 hours ago, Dunedan said:

IMO cleaner approaches for that would be fetching translations directly from Transifex (which doesn't work, because that requires a Transifex API token not every contributor has), managing them in a separate repository and pulling them from there to the 0ad git repository and the nightly-build SVN repository or just storing them in the 0ad git repository.

Yes, the first one isn't possible.

The second one I hadn't considered but maybe it would be nice. I am thinking this "translations" repository could even allow us to store the translations of official/semi-official mods. Those would not be pulled into the git and the nightly-build of 0ad, but they could be synced with Transifex, allowing our translator community to translate them. The modders would then pull translations into their mods when they release them, and push pot files to the repository whenever they wish.

For the third one, I answer you further below.

2 hours ago, Dunedan said:

Wouldn't that foster a culture where certain changes (library upgrades) are done without reviewing them, as there won't be a review tool for SVN anymore. Wouldn't it also make it pretty difficult for contributors without SVN write access to contribute to them?

Well it wouldn't be possible to review changes to the libraries themselves, but changes to the 0 A.D. code would still be reviewed and even tested by CI. This is indeed problematic but it is the same situation as today. Take for instance D5002: the contributor cannot publish the SM upgrade itself inside that diff, the diff only consists of the changes in the game. Right now that situation is too complicated for our CI pipelines to handle it (arc patch doesn't work well with binaries) thus the build for those diffs always fail and make much noise.

With git, it would be possible for a contributor to send big binaries in their PR for initial review of the whole. Then we could commit the update to source-libs and have a final round of CI on the final PR that would just update the SVN revision in libraries/ scripts, before committing. So that addresses your concern.

windows-libs changes are currently unreviewable, as they are just me or Stan sending new binaries on SVN. At least with the new system we could send the new binaries, then create a PR that just updates the SVN revision in libraries/ scripts, so that CI can run on it. So it's still a bit unreviewable, but it will actually go through CI before commit.

2 hours ago, Dunedan said:

while that would result in more commits, the amount of changes would stay more or less the same

ah that's right!

2 hours ago, Dunedan said:

What's the reason for you negative stance for bot commits to the source repository? Right now that sounds a bit arbitrary to me.

Well, it is arbitrary, or at least based on feelings (but I think it's important to consider those).

One of my issues is: those commits would lack justification. The history of the repo would contain a lot of commits that do not address an issue. Those would just be commits happening because "translations are updated all Mondays and Fridays".

Moreover, those commits have the big downside of highlighting low development activity. It feels very bad to see that autobuild commits more than the team in times of reduced activity. This is awful for morale. My proposal allows us to have daily updated translations, which is better for testing, but if we keep translations in the git repo, having daily updates would dwarf actual human contributions.

Hence, if my current system really irks you, your second proposal above is the only one I would find acceptable.

2 hours ago, Dunedan said:

I do get the reason for not committing the SPIR-V shaders to the 0ad git repository, but what's the connection to the translations here?

Basically I think that the issue with spir-v and with translations is the same: we have files that are needed for the end user (including nightly testers) but which cannot be committed to the git repo (for translations, replace "cannot be" by "I would rather have not").

I figured that if I generated shaders in the nightly and provided a way to export them into the git repo if developers needed them, I could just have the translations follow suit, and fix two problems with the same consistent solution.

2 hours ago, Dunedan said:

Right now you're one of the very few people who can actually change something about it. I'd be happy to support with that though.

You can contribute PRs to the private dev-migration repo on Gitea. But I agree it's a difficult project to collaborate on Your thoughtful feedback is much appreciated though

2 hours ago, Dunedan said:

I believe nobody proposed a single large git repository, with large as in "contains everything the current SVN repository includes"

By "large" I was thinking about translations (and "very" with shaders on top) but you're right that going from 100MiB to 700MiB would probably not bother users too much (I, on the other hand, would be sad to miss that opportunity to reduce the size by such a factor!) and I hadn't indeed realized that increasing the frequency of translation commits would not increase size that much.

Thanks again for your feedback   I'll try and think about the idea of a cleaner, separate repo for the translations.

[Itms]

On 14/07/2024 at 12:49 PM, Itms said:

You can contribute PRs to the private dev-migration repo on Gitea. But I agree it's a difficult project to collaborate on

@Dunedan Thinking about it, if you want to give a hand, you could try and take a look at creating that translations repo. You should have access to create repositories on Gitea under the 0ad/ organization. Right now I'm focused on shaders and other Jenkins improvements, I'm not going to look at translations until next week.

[Carltonus]

Able to replace some Unicode characters with typical ASCII counterparts on the Main wiki page.

Also made a proof-concept entry for the Han civilization, it just needs some additions based on the Trac counterparts. On a side note, @wowgetoffyourcellphone: what are the Chinese characters used in the term "Wu Wei Yin", and any basis for the Sun Wu Trouble Freeing Forces?

[wowgetoffyourcellphone]

21 minutes ago, Carltonus said:

Able to replace some Unicode characters with typical ASCII counterparts on the Main wiki page.

Also made a proof-concept entry for the Han civilization, it just needs some additions based on the Trac counterparts. On a side note, @wowgetoffyourcellphone: what are the Chinese characters used in the term "Wu Wei Yin", and any basis for the Sun Wu Trouble Freeing Forces?

Much of the naming and such I did not do personally. I merely made the mod suitable to add to the game. If you can come up with better names, then by all means! 

[Stan`]

2 hours ago, Carltonus said:

Able to replace some Unicode characters with typical ASCII counterparts on the Main wiki page.

Also made a proof-concept entry for the Han civilization, it just needs some additions based on the Trac counterparts. On a side note, @wowgetoffyourcellphone: what are the Chinese characters used in the term "Wu Wei Yin", and any basis for the Sun Wu Trouble Freeing Forces?

You do not need to add your name at the top of the page. Since everything is versioned you are credited in the history. If you feel there should be another place for wiki credits, maybe we should have a dedicated page. Or maybe we could pull them @Dunedan @Itms from the wiki and add them to the game...

[Dunedan]

17 hours ago, Stan` said:

If you feel there should be another place for wiki credits, maybe we should have a dedicated page. Or maybe we could pull them @Dunedan @Itms from the wiki and add them to the game...

In my opinion having contributors to the wiki credited via the wiki history is fine.

As the wiki is just another git repository, extracting and showing them in the credits ingame, would be straight-forward as well. I'd worry though, that it might be difficult to exclude spammers from being credited. In the wiki history it's straight forward to see who is a spammer, as changes are associated with a name, but in the game credits there'd only be a name.

[Dunedan]

On 14/07/2024 at 12:49 PM, Itms said:

If the exported translations are not committed, that feedback does not actually happen... Do you think it could create actual issues?

As the PO-files are excluded via .gitignore I think it'd be fine. As you might have already noticed, I'm just a friend of keeping processes lean and simple and such that circular dependency caught my eye.

On 14/07/2024 at 12:49 PM, Itms said:

I am thinking this "translations" repository could even allow us to store the translations of official/semi-official mods. Those would not be pulled into the git and the nightly-build of 0ad, but they could be synced with Transifex, allowing our translator community to translate them. The modders would then pull translations into their mods when they release them, and push pot files to the repository whenever they wish.

Without pulling the translations into the nightly-build, how'd you make such builds available in multiple languages?

While I brought up the idea of a separate git-repository for translations, after thinking about it for a few days, I believe having translations in the 0ad git repository would still the better trade-off. A separate git-repository for translations just adds additional complexity for very little benefit. For example where would all the scripts related to translations life? For some it's obvious: The one to pull translations would fit well into the translations repository. But what about other ones (to credit translators, to lint translations and POT-templates, ...).

One additional thought I had was which translations to pull from Transifex and include. Right now we do that for all languages we have set up in Transifex, no matter if we ship them or not. The volume and rate of changes to translations would be significantly lower, if instead only translations meant to be shipped would be fetched and included in the repository. The obvious downside is of course that testing translations not being shipped yet becomes harder.

On 14/07/2024 at 12:49 PM, Itms said:

With git, it would be possible for a contributor to send big binaries in their PR for initial review of the whole. Then we could commit the update to source-libs and have a final round of CI on the final PR that would just update the SVN revision in libraries/ scripts, before committing. So that addresses your concern.

Just to note: This only works if the source libs are in a git-repository.

What are the cases where binaries are part of source libs though? That should be an exception, shouldn't it?

On 14/07/2024 at 12:49 PM, Itms said:

windows-libs changes are currently unreviewable, as they are just me or Stan sending new binaries on SVN.

Yes, the usual problem with binaries. Reproducible builds would help with that, but that's another can of worms.

On 14/07/2024 at 12:49 PM, Itms said:

One of my issues is: those commits would lack justification. The history of the repo would contain a lot of commits that do not address an issue. Those would just be commits happening because "translations are updated all Mondays and Fridays".

This depends entirely on the definition of justification. I'd argue all of these commits are justified, as they update translations. What's the difference regarding justification from your point of view between updating a translation and a graphical asset?

If somebody doesn't care about translation update commits, it's easy enough to add a custom git alias to not show them in the git history, if they're made with a distinct user just for updating the translations.

On 14/07/2024 at 12:49 PM, Itms said:

Moreover, those commits have the big downside of highlighting low development activity. It feels very bad to see that autobuild commits more than the team in times of reduced activity. This is awful for morale.

Does it though? Is it really better for morale to not have recent commits?

In the end that's no objective measure, but personal preference. I believe though that a complex project structure is more harmful to contributions (especially from new contributors) than whether there are recent commits or not.

On 14/07/2024 at 12:49 PM, Itms said:

You can contribute PRs to the private dev-migration repo on Gitea. But I agree it's a difficult project to collaborate on

Where to find that? I'm not sure if I have access to that right now.

On 14/07/2024 at 12:49 PM, Itms said:

By "large" I was thinking about translations (and "very" with shaders on top) but you're right that going from 100MiB to 700MiB would probably not bother users too much (I, on the other hand, would be sad to miss that opportunity to reduce the size by such a factor!)

I do care more about an easy and accessible project/repository structure and history than optimizing just for size.

IMO having translations and source libs in the 0ad git repository wouldn't hurt anybody, but would notably simplify the whole setup.

[Itms]

19 hours ago, Carltonus said:

Able to replace some Unicode characters with typical ASCII counterparts on the Main wiki page.

Thanks for giving a hand

In other news, I have added SPIR-V generation (still running) to the nightly build, and, through a sprint, I also set up a separate translations repository, to see how it would work in practice.

I'd have to document it on the dev environment page if we go with that, but basically:

• upon each commit to the main repo, pot files are generated and pushed to the translations repo
• for the future: ideally, after that pot update, we should push templates to Transifex instead of letting them autoupdate
• daily, the translation repo pulls updates from Transifex. in the future, we should run the check script and also lint po files, and follow translation issues in that repo. we should be able to eliminate issues such as #4250

This system still has some rough edges but here are my comments after developing that:

• the concerns are nicely separated now. all interaction with Transifex happens in the translations repo. that's a really good thing.
• this separate repo will be the perfect place to collaborate with translators and fix long-standing issues such as #4250
• the base git repo only sends generated pot files using messages.json and updateTemplates.py. I deleted all the other scripts and all the .tx/config files
• the translation repo holds the majority of i18n python scripts, which do not need to know about the structure of the data/ dir anymore, that should make their maintenance simpler
• the checkDiff.py script does not work anymore with git, it was developed specifically for SVN! so the translations git repo gets all po/pot updates even if they are irrelevant. it is not a big deal for that auxiliary repo, but this is another dealbreaker against the idea of having translations in the main git repo.
• this new repo adds some complexity to the migration, as those scripts need to receive changes for that new infrastructure
• the nightly repo (and the git repo) can fetch the translations without using svn export... but git does not have that "export" feature so the new fetch script relies on creating a temporary shallow clone, that's a bit yucky
• the Jenkins setup is much more complex now, with two extra jobs (maybe three if we add a push job to Transifex) on top of the nightly-build job. It is impractical to commit and push to git using Jenkins, this is definitely not a standard use case, and my pipelines may break in the future if Jenkins features evolve. (this reinforces my prejudice against letting bots push to git)
• the Jenkins setup around the translation updates is not covered by Gitea features. it will be impractical to setup post-commit triggers for this.

In a nutshell, I think that this alternative way of handling translations is

• on the pros side: separates issues, addresses some of your concerns, and gives us much more flexibility in our interactions with Transifex, with room for improvement;
• on the cons side: more complex to maintain for Jenkins, would require extra development work for you and other Python devs around the migration

[Itms]

@DunedanAh damnit our messages crossed. Written communication is so frustrating.

40 minutes ago, Dunedan said:

Without pulling the translations into the nightly-build, how'd you make such builds available in multiple languages?

I'm talking about the translations of non-0ad projects in that sentence.

40 minutes ago, Dunedan said:

While I brought up the idea of a separate git-repository for translations, after thinking about it for a few days, I believe having translations in the 0ad git repository would still the better trade-off. A separate git-repository for translations just adds additional complexity for very little benefit.

Most of the complexity I encountered when setting up the separate repo would also exist with translations in the main git repo (many Jenkins jobs, much hacking around git+Jenkins peculiarities, checkDiff.py not compatible with git, ...), so I'm not sure that is correct.

40 minutes ago, Dunedan said:

For example where would all the scripts related to translations life?

I separated them without issue, the only problem is with our bundled modules, but I don't know enough about those Python scripts. Feel free to take a look.

40 minutes ago, Dunedan said:

The obvious downside is of course that testing translations not being shipped yet becomes harder.

Yeah, it is a big downside.... And it adds to the complexity of the release process, as deciding which translations get in would be done much sooner in the cycle.

40 minutes ago, Dunedan said:

Just to note: This only works if the source libs are in a git-repository.

What are the cases where binaries are part of source libs though? That should be an exception, shouldn't it?

No, you can do that all the time since you can patch the build scripts to download stuff from elsewhere. We do that all the time when working on SM upgrades. SpiderMonkey contains the tarball, and all source libs contain windows prebuilt binaries.

40 minutes ago, Dunedan said:

If somebody doesn't care about translation update commits, it's easy enough to add a custom git alias to not show them in the git history, if they're made with a distinct user just for updating the translations.

That's a good point, but it doesn't work for web views.

40 minutes ago, Dunedan said:

In the end that's no objective measure, but personal preference. I believe though that a complex project structure is more harmful to contributions (especially from new contributors) than whether there are recent commits or not.

If it's only personal preference, well I'd go with my own personal preferences, since I'm the one working on this. But I am indeed trying to come up with the best experience for contributors, so I'm listening to your feedback.

At this point I'm just running out of control with this translations issue: I've been dedicating far too much time and energy on that recently, while I dreamt of actually planning the migration And since I'm pondering about how to handle translations since a year and a half now, it's difficult for me to consider all the options fairly: I'm inclined to keep the system I designed.

So I suppose I'll just give up on that issue and accept keeping the translations in the repo. But I have a condition, that you update the checkDiff.py script to work with git! I'm not going to spend extra days on this thing.

40 minutes ago, Dunedan said:

Where to find that? I'm not sure if I have access to that right now.

Whoops I thought you had access, just fixed that.

40 minutes ago, Dunedan said:

I do care more about an easy and accessible project/repository structure and history than optimizing just for size.

Well... I was very proud to have reduced the size of that repository through careful design, and now I'm just throwing that part to the trash. To me it really feels like debasing my work, but I'll have to hope I'm wrong.

40 minutes ago, Dunedan said:

IMO having translations and source libs in the 0ad git repository wouldn't hurt anybody, but would notably simplify the whole setup.

For translations I trust your judgment, but for libraries that's an entirely different problem and keeping them in the repo is not possible. The idea of submodules is (probably) good but it can be done in the future if it actually bothers contributors. Having been maintaining those libs for seven years now, I'm much more confident about that part.

[Dunedan]

1 hour ago, Itms said:

Most of the complexity I encountered when setting up the separate repo would also exist with translations in the main git repo (many Jenkins jobs, much hacking around git+Jenkins peculiarities, checkDiff.py not compatible with git, ...), so I'm not sure that is correct.

I'm not concerned about the complexity of the migration. That's a one-time task. What you and I want to achieve is the best outcome for contributing to the project and managing its infrastructure afterwards. It's just that we seem to prefer different trade-offs.

1 hour ago, Itms said:

I separated them without issue, the only problem is with our bundled modules, but I don't know enough about those Python scripts. Feel free to take a look.

What do you mean by "bundled modules" and what issue exactly did you encounter?

1 hour ago, Itms said:

If it's only personal preference, well I'd go with my own personal preferences, since I'm the one working on this. But I am indeed trying to come up with the best experience for contributors, so I'm listening to your feedback.

At this point I'm just running out of control with this translations issue: I've been dedicating far too much time and energy on that recently, while I dreamt of actually planning the migration And since I'm pondering about how to handle translations since a year and a half now, it's difficult for me to consider all the options fairly: I'm inclined to keep the system I designed.

So I suppose I'll just give up on that issue and accept keeping the translations in the repo. But I have a condition, that you update the checkDiff.py script to work with git! I'm not going to spend extra days on this thing.

As we haven't been able to convince each other which approach is the better one, it's just two personal opinions. In that case I'd suggest moving forward with whatever approach you prefer, as that's the approach you already put time into (and I'm none of the main contributors to the 0ad client application anyway). However, I was hoping to get feedback on the different approaches from other developers as well.

If we need a git-compatible checkDiff.py I'd be happy to implement that.

1 hour ago, Itms said:

Well... I was very proud to have reduced the size of that repository through careful design, and now I'm just throwing that part to the trash. To me it really feels like debasing my work, but I'll have to hope I'm wrong.

You did great work so far and all the late discussion is on me, because I didn't read the details of the planned setup earlier and raised my concerns back then. As mentioned earlier already, I'm really sorry about that.

[Itms]

21 minutes ago, Dunedan said:

I'm not concerned about the complexity of the migration. That's a one-time task. What you and I want to achieve is the best outcome for contributing to the project and managing its infrastructure afterwards. It's just that we seem to prefer different trade-offs.

I agree! But I am still a bit concerned about timing, as my current full-time availability is going to expire It's definitely about trade-offs and there is no ideal solution... maybe we could find one over time but I feel like I don't have that time. I would love to find some middle ground on the translations, I thought the separate repo could be the solution, but it doesn't look like it is.

27 minutes ago, Dunedan said:

What do you mean by "bundled modules" and what issue exactly did you encounter?

Sorry not clear again! I mean the code under i18n_helper, extractors, and also tests, it looks like this code is shared between the pot generation script and the other scripts, so I kept it duplicated, it would require more work to cleanly separate all that. It's probably not worth it.

34 minutes ago, Dunedan said:

However, I was hoping to get feedback on the different approaches from other developers as well.

Hum, me too, to be fair. But I'll post on the staff forums to alert on my schedule and deadlines and hope to receive more input.

36 minutes ago, Dunedan said:

You did great work so far and all the late discussion is on me, because I didn't read the details of the planned setup earlier and raised my concerns back then. As mentioned earlier already, I'm really sorry about that.

Thank you so much for the kind words, and I understand I had a bit forgotten about the hurdles of written communication in such a project and I really appreciate your acknowledgement of my work

[Stan`]

3 hours ago, Dunedan said:

 

One additional thought I had was which translations to pull from Transifex and include. Right now we do that for all languages we have set up in Transifex, no matter if we ship them or not. The volume and rate of changes to translations would be significantly lower, if instead only translations meant to be shipped would be fetched and included in the repository. The obvious downside is of course that testing translations not being shipped yet becomes harder.

The inclusion condition vary from release to release from 85% translated to 90% translated, excluding Bengali and CJK translations. Not sure we can automate this.

3 hours ago, Dunedan said:

What are the cases where binaries are part of source libs though? That should be an exception, shouldn't it?

Windows dynamic link librairies (.dll) are next to the executable. (Macos has dylibs but they are built by users currently).

 

As mentioned on IRC font generation could be part of the nightly until we finally use the currently useless Freetype  dependency.

45 minutes ago, Dunedan said:

However, I was hoping to get feedback on the different approaches from other developers as well.

Sadly there aren't that many left. I haven't heard of @vladislavbelov of late. I suppose @wraitii needs to be asked directly about it.

[Itms]

6 hours ago, Dunedan said:

This depends entirely on the definition of justification. I'd argue all of these commits are justified, as they update translations. What's the difference regarding justification from your point of view between updating a translation and a graphical asset?

I forgot that one. And I am almost certain there is a technical term for what I'm trying to describe, but I don't know it: very frustrating.

What I mean is that translation commits are not a single unit of work. When we commit graphical assets, we don't commit a bunch of new graphics, work-in-progress models, and a few shader patches all together. We try to keep commits atomic, a single commit is the work of one artist, or a small group of artists, working on a specific asset or collection of assets. Whereas a translation update is just a large dump of the current state of our translators work across the entire project. It's not like translators coordinated themselves to publish a consistent update for French, or a multilingual update specifically covering the tutorial. If it was like this, then those commits would be crafted by humans, and I'd keep them in the history; on the contrary, I generally oppose generated commits (even if the generation is based on human work).

A rule of thumb could be: can one write a meaningful commit message? If you can't be more specific than "Updated PO and POT files", the commit is not very meaningful. You wouldn't imagine pushing a "Updated PNG files" commit.

On 14/07/2024 at 12:49 PM, Itms said:

Those would just be commits happening because "translations are updated all Mondays and Fridays".

I'd like to explain myself further about the date-triggered commits. What makes the translation commits especially meaningless is that they would be different if the Jenkins job ran ten minutes later. They depend on the state of Transifex at the moment the script is run. I believe there is a technical term for that but I don't know it either, and I think it's a bad practice to have such contingent commits in large quantity in the history (as I said, those translation commits would take 80% of the size of the git repo if I let them in!).

Obviously I'm proposing such a date-dependent system when I propose nightly builds, but that's a typical Continuous Delivery process where users receive a new version of the game for testing. Since we don't have a choice but to pull Transifex updates regularly, it makes sense to me to include them in that regular delivery. But I don't think automatic continuous delivery is supposed to happen the other way around into the source repository.

[Stan`]

Are you looking for the words cohesive and reproductible ?