A new git-based development environment

[Dunedan]

On 18/07/2024 at 6:53 PM, Itms said:

Sorry not clear again! I mean the code under i18n_helper, extractors, and also tests, it looks like this code is shared between the pot generation script and the other scripts, so I kept it duplicated, it would require more work to cleanly separate all that. It's probably not worth it.

The "extractors" module isn't necessary in "0ad/translations". "i18n_helper" on the other hand contains functionality used by scripts in "0ad/0ad" and "0ad/translations". I don't like the idea of duplicating this code, as it makes changes to it much more annoying. That's part of the reservations I have regarding a separate translations repository.

[vv221]

One week from now, I should finally have time to join this migration effort.

@Itms, my initial plan was to focus on the SpiderMonkey update, but it seems you already have that on you TODO-list? I have to admit it’s good news for me, because I know nothing about SpiderMonkey and would have to learn everything from scratch. My goal was to ensure 0 A.D. relies on the current ESR release, or at least the previous one, I guess yours is similar?

Would you agree that the best way I could help would be to experiment with git submodules ? I have some experience with them, and more generally see myself as a skilled git user (I used it daily for more than 10 years, including in very tricky ways) so that might be the best way to spend my time on this migration.

I have been maintaining git forges for a long time too (mostly self-hosted GitLab), so if you need support on this front feel free to ping me.

[hyperion]

Working with submodules is terribly difficult. A web search will give you plenty of blog post of how people thought it's a good idea for there use case and shot themself in the foot. And there was even an argument for using svn because of difficulty earlier in this thread.

There are situations where submodules make sense but to use it to pretend it's not part of the repo isn't one. If you want to have it linked just put it in directly as was done in the past. Though simply fetching as part of the build process is a far better approach imho. The svn source-libs repo as a stop gap solution I'm fine with, even tho individual git repos for each source lib would be better.

Edited July 21 by hyperion
don't merge unrelated subjects

[hyperion]

NIST banned the use of sha1 and I expect a strong push to migrate to sha256 object format the next couple years. The git object format is no longer labeled experimental and gitea supports it properly for all I know. Suggest to at least consider it.

[vv221]

1 hour ago, hyperion said:

There are situations where submodules make sense

I think our situation of depending on third-party sources using fixed releases is one of these few situations where they make sense.

Not having these sources included in the main repository will make 0 A.D. easier to package for distributions that un-vendor such dependencies, like Debian.

Edited July 21 by vv221

[vv221]

1 hour ago, hyperion said:

Working with submodules is terribly difficult.

That’s right, and I plan to abstract that by providing scripts so the people who want to build 0 A.D. locally will not have to worry about submodules at all.

[hyperion]

20 minutes ago, vv221 said:

That’s right, and I plan to abstract that by providing scripts so the people who want to build 0 A.D. locally will not have to worry about submodules at all.

The point is not how to make submodules less cumbersome for some people but not others but if they offer anything at all in our case.

We already have such a script, it's called update-workspace.sh for Linux, and any dep that can reasonably be un-vendored has a --with-system-* switch. Debian won't gain anything new at all by using submodules instead of plain old fetch everyone understands.

[vv221]

9 minutes ago, hyperion said:

The point is not how to make submodules less cumbersome for some people but not others but if they offer anything at all in our case.

I think they do bring another upside: we need to patch the source of some dependencies, like SpiderMonkey. If we used submodules instead of the curl+tar+patch dance, we could provide an already patched source instead of maintaining patches in the 0ad repository.

I am assuming here that the repository fetched by the submodules is not the upstream one, but something we would host ourselves.

---

To avoid confusion: I am not advocating that we *should* use git submodules. What I mean here is that *if* we decide that they could help us, I am willing to work on them to make their use as painless as possible.

[Stan`]

30 minutes ago, vv221 said:

42 minutes ago, hyperion said:

 

I think they do bring another upside: we need to patch the source of some dependencies, like SpiderMonkey. If we used submodules instead of the curl+tar+patch dance, we could provide an already patched source instead of maintaining patches in the 0ad repository.

That would work better for NVTT and Fcollada which are now solely maintained by us, although we still support distro provided nvtt (but the last version before the github repo being archived is broken)

Also the patching is OS specific...

 

[hyperion]

21 minutes ago, vv221 said:

I think they do bring another upside: we need to patch the source of some dependencies, like SpiderMonkey. If we used submodules instead of the curl+tar+patch dance, we could provide an already patched source instead of maintaining patches in the 0ad repository.

That can be taken as an argument against an svn source-libs repo and using individual git repos if you want.

 

23 minutes ago, vv221 said:

I am assuming here that the repository fetched by the submodules is not the upstream one, but something we would host ourselves.

There is no difference who hosts it, except that one hoster may be down while the other currently isn't. Basically, the use of submodules as described so far is just replacing a repo uri and a hash in "update-workspace.sh" with a submodule and that is definitely not worth the associated pain.

[vv221]

26 minutes ago, hyperion said:

There is no difference who hosts it

The difference would have been the ability to patch the source more easily, but Stan` wrote above that we use OS-specific patches, something that submodules could not help with.

---

EDIT: Actually the OS specificities could still be handled as per-OS git branches, so applying the patches in these repositories instead of doing the patching at build time is an improvement that could still be done.

Improvement that could be done no matter if we chose to go with submodules or tarballs, so this is not really an argument in favour of submodules themselves.

Edited July 21 by vv221

[vv221]

20 minutes ago, hyperion said:

That can be taken as an argument against an svn source-libs repo and using individual git repos if you want.

I do indeed think that per-dependencies repositories would be easier to work with than a common repository for all libs.

Then the way we include these, be it with curl+tar+patch or submodules, is in my opinion not of a big importance. I think the best option is the one the people who will actually maintain the build tools would rather work with.

[Stan`]

I suppose submodules support branches?

[vv221]

3 minutes ago, Stan` said:

I suppose submodules support branches?

It looks like I have been a bit too slow with my EDIT

[Stan`]

I really wish we had a one click build operation like godot does with his scons file.

My scripts take care of that for linux bsd and macos but it still feels like a workaround.

Anyway I think @Itms scripts are a step in the good direction, and their implementation can change.

One big Windows issue is that dlls need to be in the binaries/system folder so they cannot be self contained in the  libraries/** folders. The debug symbols in theory needs not, but without debugger you still need them in the same folder to print a stacktrace.

[hyperion]

8 minutes ago, Stan` said:

I suppose submodules support branches?

Submodules are probably best described as pointers and are at least as hard to work with as pointers in C. The repository pointed at is a normal independent git repo.

[hyperion]

19 minutes ago, vv221 said:

The difference would have been the ability to patch the source more easily, but Stan` wrote above that we use OS-specific patches, something that submodules could not help with.

Well, semantics if you want, but in a dvcs unlike the cvcs there is no real "upstream"

 

[vv221]

6 minutes ago, hyperion said:

Submodules are probably best described as pointers and are at least as hard to work with as pointers in C.

This is a very apt description, and having no problem at all wrapping my head around pointers might explain why I do not see submodules as really confusing.

[Stan`]

Right so you can only point to a specific submodule git repo a specific commit and a specific branch.

But you could have more branches in the containing repo. Anyway sounds like a mess ^^

[Dunedan]

On 18/07/2024 at 4:20 PM, Itms said:

But I have a condition, that you update the checkDiff.py script to work with git! I'm not going to spend extra days on this thing.

In case the translations end up in Git, here is a rewrite of check_diff.py which supports Git instead of Subversion: https://code.wildfiregames.com/D5312

[Itms]

On 21/07/2024 at 12:10 PM, vv221 said:

One week from now, I should finally have time to join this migration effort.

Fantastic, thanks for your interest.

On 21/07/2024 at 12:10 PM, vv221 said:

@Itms, my initial plan was to focus on the SpiderMonkey update, but it seems you already have that on you TODO-list? I have to admit it’s good news for me, because I know nothing about SpiderMonkey and would have to learn everything from scratch. My goal was to ensure 0 A.D. relies on the current ESR release, or at least the previous one, I guess yours is similar?

Indeed we need to get the SM upgrade done, and I have the experience to make it happen, but I will have to focus on the git migration first, so I'll probably be entirely occupied by that for the month of August. So, I can help you, but I cannot entirely take it off your hands for now - and it is a good opportunity to try your hands at it if you are interested.

The plan is to upgrade to a more recent ESR, yes, sometimes it's better to avoid skipping an ESR (especially if the newest one is too new and has undetected issues with embedding), but sometimes there is no need to keep a previous ESR... I'll have to see when I look into this specific one.

On 21/07/2024 at 12:10 PM, vv221 said:

Would you agree that the best way I could help would be to experiment with git submodules ? I have some experience with them, and more generally see myself as a skilled git user (I used it daily for more than 10 years, including in very tricky ways) so that might be the best way to spend my time on this migration.

I have been maintaining git forges for a long time too (mostly self-hosted GitLab), so if you need support on this front feel free to ping me.

It sounds great to have your help and input on the use of submodules! But, considering the work needed, the imminence of the migration, and the unresolved discussion about using them or not, I would like to perform the migration with the source-libs SVN repo, and try and maybe decide using submodules in the near future, after the migration. You can certainly start working on that, but you would design an incremental improvement upon my current work, so expect having to rebase a lot.

On 21/07/2024 at 5:43 PM, Stan` said:

One big Windows issue is that dlls need to be in the binaries/system folder so they cannot be self contained in the  libraries/** folders. The debug symbols in theory needs not, but without debugger you still need them in the same folder to print a stacktrace.

Yes: even with submodules we'd need a wrapper script somewhere. I believe my get/build-libraries script are here to stay, and they could abstract the use of svn-export or git submodules without breaking the user workflow if we switch to submodules.

On 21/07/2024 at 2:55 PM, hyperion said:

NIST banned the use of sha1 and I expect a strong push to migrate to sha256 object format the next couple years. The git object format is no longer labeled experimental and gitea supports it properly for all I know. Suggest to at least consider it.

Thanks for your input! I saw gitea support being launched in the latest release and I didn't know if we could be interested in that. I will look into sha256 support, if we're going to late-switch to git, we may as well use modern features and be future-proof

9 hours ago, Dunedan said:

In case the translations end up in Git, here is a rewrite of check_diff.py which supports Git instead of Subversion: https://code.wildfiregames.com/D5312

Thank you so much for the work I will try and get other developers to give their opinion, but now at least there is no technical blockage against either solution for translations.

[Stan`]

BTW @Itms any reason you kept the Jenkins pipelines private? Is there still a credential leak bug there?

[Itms]

15 minutes ago, Stan` said:

BTW @Itms any reason you kept the Jenkins pipelines private? Is there still a credential leak bug there?

No not at all, I have just been too lazy to set up anonymous access to all the jobs I created, deleted, re-created.... I will open them up in the upcoming days, and I will also be able to considerably simplify the access to Jenkins after the actual migration.

[Stan`]

I'm really not a fan of the new display, but I got accustomed to Blue Ocean after hating it for a while to the point I only used it before it was removed; so maybe i'll get used to it too.

[Itms]

6 minutes ago, Stan` said:

I'm really not a fan of the new display, but I got accustomed to Blue Ocean after hating it for a while to the point I only used it before it was removed; so maybe i'll get used to it too.

Yes, I also liked Blue Ocean for displaying stage output, but unfortunately that project is ended. Blue Ocean will not receive updates and thus will never be able to display a lot of modern Pipeline features I started to put into place.

The officially recommended replacement I used, Pipeline Graph View, is still pretty new and rough around the edges, but I'm sure it will get improvements and get better usability as time goes by