A new git-based development environment

[Itms]

Hi everyone,

For a few years now we have pondered whether git would be the right VCS for the development of 0 A.D. One of the main selling points of git is the ability to create numerous branches for parallel development of features and bugfixes. Several popular platforms (Github, Gitlab.com, ...) leverage this to provide a feature-packed development environment, where people can contribute, discuss, review, and test patches.

Historically, the development of 0 A.D. has been performed using SVN, which is a bit old... but suits our needs in a number of ways, already discussed elsewhere. We are using Phabricator to supercharge SVN with a review and CI/CD platform, but alas, Phabricator has been abandoned by its developers. In any case, a lot of contributors in the past couple of years have expressed disappointment with the peculiarities of Phabricator.

Thus, we have discussed finally moving to git, without any strong urgency or hard opinions on the matter. I have proposed myself to set up a migration path and a new development environment for 0 A.D., and I am proud to present it to you folks today.

I have started working on this "for real" more than a year ago, and started dedicating all my "0 A.D. time" (which has been very scarce) on this project at the end of Spring 2023. It is still, unfortunately, very much a work-in-progress, but now is the time to reveal it and receive feedback.

Ideally, I would like to collect feedback from the community, finish the CI/CD system, and fix bugs, over the course of the spring, with the aim of performing the actual migration, if the community wishes to perform it, during the summer. As always, delays should be expected. It is worth noting that personal changes in my life/work balance will give me more free time after the summer.

Thanks in advance for the feedback you will provide

Presentation of the Proof of Concept

I am self-hosting the services for the duration of the tests, so that I have full control and do not divert resources from WFG. All URLs are under itms.ovh, which should eventually become wildfiregames.com

Email does not work on this Proof of Concept! I have no will to setup email and I certainly don't want to spam anyone with tests.

The services are not updated automatically: the full migration takes around 48h and necessitates some human input. The PoC I am presenting was generated from scratch the past two days. I will update stuff from time to time, when I fix bugs or when I'm bored, but don't expect the PoC to follow the actual development of the WFG services.

Please test as many things as you want and don't hesitate to try to break things! Now is the time to make sure I didn't forget anything. All the data sent to the PoC will be wiped out whenever I re-migrate the services, and of course whenever the actual migration happens. Please feel free to play around all this.

The services (this will be regularly updated)

- gitea.itms.ovh The development platform.

This is an instance of Gitea, a self-hosted lightweight alternative to Gitlab/Github. If, in the far future, we want to stop self-hosting stuff, Gitea is compatible with Github features to allow for easy migration.

The git repository is at wfg/0ad. Commit messages were reformatted, contain the original SVN revision, links to other commits are preserved as much as possible. The HEAD revision of the main "trunk" branch is the HEAD of the SVN repo.

If you wish to use the git repository for development, you must, for now, use the "future" branch, which will be merged to trunk upon the migration.

All Gitea users were imported from Trac. Since email does not work, if you want to login, shoot me a DM and I'll send you a password.

All Trac tickets are imported at wfg/0ad/issues.

The Trac wiki is imported at wfg/0ad/wiki. I tried to self-document the PoC as much as possible, so please read the updated pages, especially BuildInstructionsGettingTheCode and BuildAndDeploymentEnvironment. The FAQ is defaced beyond readability because of its formatting peculiarities, work is needed, my apologies to everyone who worked hard on the visuals of the Trac page...

 

- trac.itms.ovh A read-only copy of Trac, upgraded to the latest release.

This serves as a reference, and can stay online as long as needed. See below for the redirect tool.

- code.itms.ovh A registration-disabled copy of Phabricator, upgraded to the latest version (RIP).

All the inactive accounts were deleted, especially the dormant spam accounts.

We need to keep all the discussions on patches and commits online, however, Phabricator is now a security liability. I disabled the API access as well as the possibility to upload new diffs or files, and, in the future, I might make it read-only. I will consider migrating to Phorge (the community-maintained fork of Phabricator) when they officially support PHP8, which will allow us to stop using a deprecated version of PHP. However, migrating to Phorge does not mean we will start using Phabricator again.

- svn.itms.ovh The old and new SVN repositories.

This holds the current ps repo (which will stop receiving commits after the migration), the audio and art sources (which will keep working as-is), the new libraries repos for precompiled Windows and for bundled libs, and most importantly the new nightly-build repo.

The nightly build can be used by anyone to test the latest development version. Please read NightlyBuild if you want to know more.

- ariadne.itms.ovh Ariadne is a small redirect tool I wrote as a drop-in replacement for Trac or Phabricator when/if we decide to drop them. A lot of links to Trac exist on the Internet. You can follow Trac-like paths on Ariadne to find the relevant content. Each Trac page displays a link to the associated Ariadne path.
For instance :

• A ticket https://ariadne.itms.ovh/ticket/666
• A wiki page https://ariadne.itms.ovh/wiki/Manual_Settings
• A SVN changeset https://ariadne.itms.ovh/changeset/28056
• A Phabricator commit page https://ariadne.itms.ovh/rP28056
• A file in the Trac browser https://ariadne.itms.ovh/browser/ps/trunk/source/test_setup.cpp

Ariadne also provides a nice page for knowing commit correspondence, just use `rXXXX` as path:

• https://ariadne.itms.ovh/r28056
• https://ariadne.itms.ovh/r28000

 

Enjoy! And see you soon for regular updates

Edited July 2 by Itms
Update 02 July

[ShadowOfHassen]

I have been wanting this a long time. I want to give two votes for the project, one with my ShadowOfHassen account and another with my PoppaSmurf account I got on the gitea. (https://gitea.itms.ovh/PoppaSmurf )

I PMED for a password, but I couldn't wait. Account creation worked, and then I created a repository (which I don't think average people should be able to do.) When I tried to upload files using vscodium as a GUI front end to new repo however, I got this error: "

> git pull --tags origin main
fatal: couldn't find remote ref main

"

I also was unable to fork 0 A.D.

 

I really hope this works out! I'll be testing this!

[ShadowOfHassen]

I can confirm that collaboration works. PoppaSmurf has made ShadowOfHassen a collaborator to the new encyclopedia repository. I'd like to figure out why I can't sync changes though.

[ShadowOfHassen]

Would someone do me a favor and say something obnoxously rude in an issue on the encylopedia repo: https://gitea.itms.ovh/PoppaSmurf/historyencylopedia

I want to see if I can delete a post as a moderator.

[ShadowOfHassen]

Also my ShadowOfHassen Account can only have one repo. Poppa Smurf has several but he is still banned from creating another.

 

Which is a shame because I wanted to test private repositories.

Edited April 4 by ShadowOfHassen

[ShadowOfHassen]

For some reason, my commit profile icons don't match my custom icons.

[hyperion]

@Itms

Glad to see progress was made and we might have an improved workflow in the not so distant future. Appreciated.

I wonder if it would make sense to create a subforum for the git migration and keep this topic as a means for informing the community about the general progress. I already see me bringing up various issues which each might warrant a discussion. Putting them into separate threads would help keep some order at least and help avoid duplicate reports to some extent. Or maybe we simply should use trac?

[ShadowOfHassen]

So is the plan to allow us to have multiple repos in the wildfire Git. Clearly you'd need permissions (We ain't hosing no GitLab here) but it would be interesting to be able to have multiple repos (keep the community mod in the gitea perhaps?)

[Gurken Khan]

15 minutes ago, ShadowOfHassen said:

Would someone do me a favor and say something obnoxously rude in an issue on the encylopedia repo: https://gitea.itms.ovh/PoppaSmurf/historyencylopedia

I was willing to comply but didn't see how how I could comment at that URL; clicking around I encountered

Quote

Path readme.md doesn't exist in branch main

(I don't know if this is intended behavior.)

[ShadowOfHassen]

11 minutes ago, Gurken Khan said:

I was willing to comply but didn't see how how I could comment at that URL; clicking around I encountered

(I don't know if this is intended behavior.)

You can make a rude comment in this issue: https://gitea.itms.ovh/PoppaSmurf/historyencylopedia/issues/1

 

The second problem probably is OK because I renamed that file around the same timeframe to Readme.md, so you might have had a glitch around there. (Nothing wrong with @Itms setup, just a qwirk that gitea has.

 

[ShadowOfHassen]

So gitea, doesn't have anything to mark comments off-topic, but I managed to delete @Gurken Khan's obnoxiously rude comment. I, as moderator, could also edit it. So I guess permissions are working.

[Dunedan]

I'm really looking forward for the migration to Gitea, as it makes contributing so much easier and as it bundles different functionality (tickets, wiki, code review, ...) it'll significantly reduce the complexity of infrastructure we need. Great work @Itms.

[Itms]

10 hours ago, ShadowOfHassen said:

Account creation worked, and then I created a repository (which I don't think average people should be able to do.)

9 hours ago, ShadowOfHassen said:

Also my ShadowOfHassen Account can only have one repo. Poppa Smurf has several but he is still banned from creating another.

9 hours ago, ShadowOfHassen said:

So is the plan to allow us to have multiple repos in the wildfire Git. Clearly you'd need permissions (We ain't hosing no GitLab here) but it would be interesting to be able to have multiple repos (keep the community mod in the gitea perhaps?)

Yes, I am hesitating between letting new accounts create garbage repositories, and having to manually approve any new contributor. I chose to set the default repo limit to 1. But it looks like forks don't count? If that's the case, I could set the default to 0 to let new contributors fork and contribute: testing this now.

In any case, I set the limit to 5 on both your accounts.

10 hours ago, ShadowOfHassen said:

When I tried to upload files using vscodium as a GUI front end to new repo however, I got this error

What is the output of git remote -v?

9 hours ago, ShadowOfHassen said:

For some reason, my commit profile icons don't match my custom icons.

This is because the commit author's email does not match the PoppaSmurf account email. You can use secondary email addresses in your account for that. I wrote down that I have to document it.

9 hours ago, hyperion said:

I wonder if it would make sense to create a subforum for the git migration and keep this topic as a means for informing the community about the general progress. I already see me bringing up various issues which each might warrant a discussion. Putting them into separate threads would help keep some order at least and help avoid duplicate reports to some extent. Or maybe we simply should use trac?

No, please post everything here, I'm following. And this is just an unofficial PoC for now, please don't report bugs on Trac.

Edited April 5 by Itms

[Stan`]

11 hours ago, Itms said:

This should stay up as much as possible. We need to keep all the discussions on patches and commits. TODO: I need to find a (possibly ugly) way to disable the upload of new diffs.

Maybe you can remove everyone's permissions ?

@ShadowOfHassen @Itms might correct me here but I don't think it's good to fork 0 A.D. due to its size. Disk space isn't infinite and IMHO it's much better to only have branch. (Last time I checked every fork added 6GB of files...)

Official repos like the community mod and maybe the lobby can go there but having private things sounds bad. Someone ill intentioned could host an entire fork of the game free of charge on the server.

@Itms for the CI I had some attempts on my version. It can build stuff and notifiy gitea through api calls but, extracting the correct branch to build was a bit trickier.

[Itms]

5 minutes ago, Stan` said:

Maybe you can remove everyone's permissions ?

No, I want everyone to be able to keep commenting on open diffs, for instance for posting updates such as "I opened a PR for this diff at the following address..". It is important not to break current work when launching the migration.

I believe my fix will consist of blocking the diff upload URL directly in the web server config, but I'll have to test and see whether that blocks arcanist upload through the API... Well maybe I'll just block the API for security reasons.

9 minutes ago, Stan` said:

might correct me here but I don't think it's good to fork 0 A.D. due to its size. Disk space isn't infinite and IMHO it's much better to only have branch. (Last time I checked every fork added 6GB of files...)

No! That's the beauty of it! (yes I'm going to blow my own horn quite a bit in this thread, sorry). Forks do not duplicate LFS storage. So every fork of 0 A.D. only weighs 100MiB.

13 minutes ago, Stan` said:

Official repos like the community mod and maybe the lobby can go there but having private things sounds bad. Someone ill intentioned could host an entire fork of the game free of charge on the server.

Yes, if forks don't count as we seem to have discovered, I'll keep the limit at 0 repos, it's better. I'm more worried about malicious software than forks... (also we need to stop using the word "fork" in that sense if we start using gitea...

14 minutes ago, Stan` said:

for the CI I had some attempts on my version. It can build stuff and notifiy gitea through api calls but, extracting the correct branch to build was a bit trickier.

I saw that but I think you didn't use the correct tools at all. I'll get back to you when I actually start working on this.

[Stan`]

4 minutes ago, Itms said:

No, I want everyone to be able to keep commenting on open diffs, for instance for posting updates such as "I opened a PR for this diff at the following address..". It is important not to break current work when launching the migration.

So that means we'll have to keep deleting every single Phabricator spam message manually that are now on commits, diffs and sometimes on personnal profiles while maintaining gitea ?

5 minutes ago, Itms said:

No! That's the beauty of it! (yes I'm going to blow my own horn quite a bit in this thread, sorry). Forks do not duplicate LFS storage. So every fork of 0 A.D. only weighs 100MiB.

21 minutes ago, Stan` said:

Oh sweet then. 100MB will make it harder to reach the disk size

Also means the repo won't grow weirdly and exponentially like it did for me

7 minutes ago, Itms said:

Yes, if forks don't count as we seem to have discovered, I'll keep the limit at 0 repos, it's better. I'm more worried about malicious software than forks... (also we need to stop using the word "fork" in that sense if we start using gitea...

True. I also disliked the fact you could make the diff private on Phab because that's kind of anti open source to me and moreover it broke the bot as well. If you're gonna have private code maybe don't share it at all...

9 minutes ago, Itms said:

I saw that but I think you didn't use the correct tools at all. I'll get back to you when I actually start working on this.

X) Well I did my best.

[Itms]

12 minutes ago, Stan` said:

So that means we'll have to keep deleting every single Phabricator spam message manually that are now on commits, diffs and sometimes on personnal profiles while maintaining gitea ?

Reminder: registration is now disabled on Phabricator. Only currently active users can post those updates.

And indeed I forgot to write this in the top message: all the accounts without content were deleted on Phabricator. If you login to code.itms.ovh, you'll see that every possible dormant spam account was deleted. Of course, if the spammer has recently posted, my script will not delete them... But we can get rid of them manually and then we'll be fine.

[hyperion]

1 hour ago, Itms said:

No, please post everything here, I'm following. And this is just an unofficial PoC for now, please don't report bugs on Trac.

Fine, let me add some random findings in short form:

• default branch name trunk -> main maybe

• Pascal case generates a link on trac, maybe camel case as well. The migration script ideally filters such broken links. Example: https://gitea.itms.ovh/wfg/0ad/issues/6796

• https://gitea.itms.ovh/TracUser should probably be renamed, at least if it's used in future for automation

• in commit messages patch by, comments by, reviewed by, svn commit id probably should be a block, ie. no empty lines in between

• might want to rewrap commit messages at width 72, some weren't wrapped at all and some are wrapped poorly after changing svn rev with the much longer commit hash

 

1 hour ago, Itms said:

No! That's the beauty of it! (yes I'm going to blow my own horn quite a bit in this thread, sorry). Forks do not duplicate LFS storage. So every fork of 0 A.D. only weighs 100MiB.

If the underling file system allows sharing extends this should be even less.

 

 

[Itms]

21 minutes ago, hyperion said:

default branch name trunk -> main maybe

I'd like to keep trunk for historical/nostalgic reasons, and also because I'd like us to explicitly follow a trunk-based development model. I am open to discussion about that.

21 minutes ago, hyperion said:

Pascal case generates a link on trac, maybe camel case as well. The migration script ideally filters such broken links. Example: https://gitea.itms.ovh/wfg/0ad/issues/6796

No, that's too difficult. But you can fix your message on Trac by adding a ! before the name to prevent link generation, and thus the migration to gitea will keep it as a non-link.

21 minutes ago, hyperion said:

https://gitea.itms.ovh/TracUser should probably be renamed, at least if it's used in future for automation

TracUser is a Gitea account I created specifically for messages originating from people who have no account on Trac, and thus no account from migration on Gitea. This covers two cases: people who posted once on Trac without registering with an email address; and people who deleted their Trac account. This has no relationship with CI/CD.

21 minutes ago, hyperion said:

in commit messages patch by, comments by, reviewed by, svn commit id probably should be a block, ie. no empty lines in between

Mh I can probably group the Phabricator-related lines indeed, but I'm not sure it makes much of a difference. If others want me to do that, please weigh in.

I think that the reference to the SVN rev must be separated from the rest by a blank line though.

21 minutes ago, hyperion said:

might want to rewrap commit messages at width 72, some weren't wrapped at all and some are wrapped poorly after changing svn rev with the much longer commit hash

Could you come up with a script that performs that rewrap? I am not sure how to perform this in a way that provides a satisfying result for all commit messages.

21 minutes ago, hyperion said:

If the underling file system allows sharing extends this should be even less.

Indeed, it is.

[hyperion]

50 minutes ago, Itms said:

I'd like to keep trunk for historical/nostalgic reasons, and also because I'd like us to explicitly follow a trunk-based development model. I am open to discussion about that.

I don't really mind but @ShadowOfHassen above was puzzled with the outcome of "git pull --tags origin main", so it's safe to assume there is a certain expectation.

 

57 minutes ago, Itms said:

TracUser is a Gitea account I created specifically for messages originating from people who have no account on Trac, and thus no account from migration on Gitea. This covers two cases: people who posted once on Trac without registering with an email address; and people who deleted their Trac account. This has no relationship with CI/CD.

Ok, but then maybe using Ghost like github or similar would be better to indicate the intent of it being a removed/invalid user. Could also be used in future for deleted users without awkwardness.

 

1 hour ago, Itms said:

Mh I can probably group the Phabricator-related lines indeed, but I'm not sure it makes much of a difference. If others want me to do that, please weigh in.

It helps distinguishing metadata from commit message, the linux kernel and many other projects do this as well, guess it would also be good to follow this convention in the future.

 

1 hour ago, Itms said:

Could you come up with a script that performs that rewrap? I am not sure how to perform this in a way that provides a satisfying result for all commit messages.

Is the source of the whole migration available somewhere?

Anyway a first simple solution from the top of my head

:set textwidth=72
:normal gggqG
:wq

save the above as rewrap.vim and run as

vim -es -S rewrap.vim message.txt

Maybe wrapping the first line (subject) is undesirable though. Anyway looking at the result should shed some light on what to improve.

 

Coreutils has commands to similar effect IIRC, would have to dig through man pages tho. But they might fit better into the migration pipeline in the end.

 

 

[Itms]

13 minutes ago, hyperion said:

I don't really mind but @ShadowOfHassen above was puzzled with the outcome of "git pull --tags origin main", so it's safe to assume there is a certain expectation.

Oh I thought that was for his encyclopedia repo test... Using git clone would be more straightforward and would rely less on assumptions.

14 minutes ago, hyperion said:

Ok, but then maybe using Ghost like github or similar would be better to indicate the intent of it being a removed/invalid user. Could also be used in future for deleted users without awkwardness.

It's a good idea to use a more widely recognized name, yes. (However I, for one, didn't know about Ghost)

15 minutes ago, hyperion said:

Is the source of the whole migration available somewhere?

Yes, but in a private repo.

16 minutes ago, hyperion said:

Anyway a first simple solution from the top of my head

Maybe wrapping the first line (subject) is undesirable though. Anyway looking at the result should shed some light on what to improve.

Oh yes, it is undesirable. I thought you were talking about wrapping the first line, and that is difficult to do. The second line of a git commit message must be empty, and I certainly don't want to cut an informative commit message in half. Can you point out a few instances of commit messages where something else than the first line would benefit from being wrapped?

[hyperion]

16 minutes ago, Itms said:

Can you point out a few instances of commit messages where something else than the first line would benefit from being wrapped?

Guess there are many, one with questionable wrapping is fc5518a22b15827c7f4d4410c27dad4c628af8a4 and on that got worse with rev to sha is 9f85b097e11ffac38df23815a62b14bd25925d2d

 

19 minutes ago, Itms said:

rely less on assumptions

you can't change human nature

 

21 minutes ago, Itms said:

Yes, but in a private repo.

Needs to be made public before the actual migration so others can verify the migration, but can wait.

[Stan`]

Probably just me but I liked the idea of naming the repo pyrogenesis instead of 0ad if we ever split things up more so the engine is its own thing. Just thought I'd mention it.

[ShadowOfHassen]

6 hours ago, Itms said:

What is the output of git remote -v?

origin   https://gitea.itms.ovh/PoppaSmurf/historyencylopedia (fetch)
origin   https://gitea.itms.ovh/PoppaSmurf/historyencylopedia (push)

Which should in theory make it work... I think.

 

Yes, @Itmswas right, the problem was with me trying to upload some test files. I'll only do one commit, though, because I don't want to eat a lot of data/ bandwidth. I think most of the general features work, I think it's up to the smart people to try to break it.

1 hour ago, Stan` said:

Probably just me but I liked the idea of naming the repo pyrogenesis instead of 0ad if we ever split things up more so the engine is its own thing. Just thought I'd mention it.

I like this idea. I would love for WFG to make a series of RTSes

[Itms]

1 hour ago, Stan` said:

Probably just me but I liked the idea of naming the repo pyrogenesis instead of 0ad if we ever split things up more so the engine is its own thing. Just thought I'd mention it.

Well I suppose it should happen at the split, not before. Currently this repository is 0ad, since it contains the game assets.

Also it is important to have a memorable and easy to type address. I am already reluctant to have called the repo wfg/0ad instead of 0ad/0ad, but the organization must be named wfg, so I settled with that.

18 minutes ago, ShadowOfHassen said:

Which should in theory make it work... I think.

Don't you need ".git" at the end of the remote url? If not, that's an actual bug I have to take a look at.

20 minutes ago, ShadowOfHassen said:

I'll only do one commit, though, because I don't want to eat a lot of data/ bandwidth.

Please don't restrain yourself, test away. Better detect bottlenecks now.