A new git-based development environment

[plusmid]

Hi @Itms, thank you for your effort to migrate to Git(ea).

I had a discussion on Mastodon about the switch and wanted to bring Forgejo to your attention. It's a fork of Gitea maintained by a non-profit rather than a for-profit corporation.

Switching over should be pretty easy as both projects are still very close. Maybe it's work a try.

Cheers, plusmid

Edited April 26 by plusmid

[Itms]

Hello @plusmid! I know about Forgejo. It's a great project and you're right to promote it.

Basically we want to use and support open-source software, but we'd also like to use widely-used, even if it's commercial, software in order to save energy in maintaining our tools. A lot of members of the community have been advocating for using Github, which everyone uses, and which is "too big to fail". Obviously we're reluctant to use Github/Microsoft closed software, but maybe we'll end up there anyway if it proves too difficult to self-host a forge. Gitlab is too heavy for us to self-host, Gitlab.com is another alternative possibility for the future.

I think Gitea is a very good middle-ground, both open-source, self-hosted and Github-like. It has official maintained support from the Jenkins developers (we use that for CI/CD). We're very happy to know that there is a non-profit alternative in case the Gitea firm starts doing shady stuff, but for now, the only thing we want from upstream is stability and ease-of-use. Forgejo is simply not big enough yet, to put it bluntly.

[plusmid]

A valid concern. I hope Forgejo will grow big enough to one day host 0ad. Thank you for your effort!

[BOB13671]

@ItmsAs someone who is interested in seeing the newest changes and just went through the arduous process of compiling the game from source, a nightly build sounds really cool.

 

Sadly, despite following the nightly build guide, I encountered an issue with TortoiseSVN failing to recognize  https://svn.itms.ovh/nightly-build/trunk. I'm wondering if this may be due to nightly builds not being implemented yet or if the svn link is not working as intended?

 

Edited April 28 by BOB13671

[Stan`]

If you're using tortoise SVN that means you're on Windows right ? Then if you use the current SVN repo you'll get autobuilds

And for linux once the deps are installed you can use a simple shell script to build everything.

https://code.wildfiregames.com/P241

[BOB13671]

The problem I was facing was following the guide for the nightly build (link), and the url for it does not exist. I tried this both on tortoiseGit and the linux terminal.

Example:

~$ svn checkout https://svn.itms.ovh/nightly-build/trunk
svn: E170000: URL 'https://svn.itms.ovh/nightly-build/trunk' doesn't exist

In other news I downloaded the code from the svn repo https://svn.wildfiregames.com/public/ps/trunk/ , and ran the bash script you provided, which it made it extremely easy to get the game running. Thank You!

[Norse_Harold]

I'd like to use my Gitea account to help identify issues with the new system, so please send the password.

What's being used for CI/CD, still Jenkins, or something else? Where in the web-based interface is the CI/CD system? I particularly want to see whether warnings from source/tools/entity/checkrefs.py are being passed to developers effectively, as it's relevant to the changes to build/jenkins/pipelines/docker-differential.Jenkinsfile in D5266.

[Stan`]

2 hours ago, Norse_Harold said:

I'd like to use my Gitea account to help identify issues with the new system, so please send the password.

What's being used for CI/CD, still Jenkins, or something else? Where in the web-based interface is the CI/CD system? I particularly want to see whether warnings from source/tools/entity/checkrefs.py are being passed to developers effectively, as it's relevant to the changes to build/jenkins/pipelines/docker-differential.Jenkinsfile in D5266.

Currently the CI/CD is not implemented, but yeah we'll probably use Jenkins. (There is new stuff like https://www.drone.io/ but everything is we currently have is for jenkins)

For security reasons, I don't think we should run the CI with the Jenkinsfile on the branches (malicious actors might do bad things) If you want to test a new CI script you need access to the Jenkins instance or talk to someone who does.

Regarding linting, it would be indeed interesting to have some better linting that what's currently done, where you have to check the linting on Jenkins. I've redirected error output to the correct stream, so you can technically get errors by filtering output 2. Currently IIRC the linting failing is not breaking the pipeline.

Regarding that change, I'm a bit sad it went from -tax to -actx (cause tax is fun and so is tau (unused), but the rest looks okay)

[Itms]

On 28/04/2024 at 2:44 AM, BOB13671 said:

Sadly, despite following the nightly build guide, I encountered an issue with TortoiseSVN failing to recognize  https://svn.itms.ovh/nightly-build/trunk. I'm wondering if this may be due to nightly builds not being implemented yet or if the svn link is not working as intended?

Not implemented yet, but close.

19 hours ago, Norse_Harold said:

I'd like to use my Gitea account to help identify issues with the new system, so please send the password.

Sent.

19 hours ago, Norse_Harold said:

What's being used for CI/CD, still Jenkins, or something else? Where in the web-based interface is the CI/CD system? I particularly want to see whether warnings from source/tools/entity/checkrefs.py are being passed to developers effectively

I'm almost ready to start CI (hopefully) and I'll use the checkrefs script as my main test avenue. It will allow me to check that Git LFS interacts correctly with the CI.

19 hours ago, Norse_Harold said:

as it's relevant to the changes to build/jenkins/pipelines/docker-differential.Jenkinsfile in D5266.

Once I have set things up, I will be very interested in having those changes in a pull request on gitea.

17 hours ago, Stan` said:

For security reasons, I don't think we should run the CI with the Jenkinsfile on the branches (malicious actors might do bad things) If you want to test a new CI script you need access to the Jenkins instance or talk to someone who does.

This is automatically covered by the Gitea Jenkins plugin. Jenkins uses the Jenkinsfile from the upstream branch when the pull request author is not a recognized contributor. If it works as advertised, this will remove the need to have access to the Jenkins instance.

[Stan`]

2 hours ago, Itms said:

This is automatically covered by the Gitea Jenkins plugin. Jenkins uses the Jenkinsfile from the upstream branch when the pull request author is not a recognized contributor. If it works as advertised, this will remove the need to have access to the Jenkins instance.

Neat!

[ShadowOfHassen]

Any update on this? I know people are busy with their own life, but I'm excited!

[Stan`]

5 minutes ago, ShadowOfHassen said:

Any update on this? I know people are busy with their own life, but I'm excited!

A presentation was given at a small french conference back in May, and work should resume during the summer

Work has been done on the documentation that can be checked there.

 

[Itms]

Hello folks! I have been hard at work on this, and things are starting to look usable.

Here is a (probably incomplete) changelog:

• The CI works! It is still a bit raw and I deactivated it to save space on Jenkins, but I am now working on improving it. It is the last big chunk of work I have to perform, and then we will be ready to migrate.
• The nightly build now exists! Get it via SVN at https://svn.itms.ovh/nightly-build/trunk. It is generated from a month ago but I'll update it through Jenkins in the upcoming days.
• All the documentation was updated and improved during the JDLL event in May where I presented the project and received some feedback. I also improved the Privacy Policy based on feedback. Please head over to the wiki especially BuildAndDeploymentEnvironment.

I need help! The glorious FAQ has been defaced by the conversion to Gitea, which uses Github-flavored Markdown. Is anyone interested in working on manually restoring the appearance of the FAQ, adapting it to its new home on Gitea? Please let me know if you wish to help. Basically it would be necessary to 1) cleanup the structure of the page, fix the tables, the raw HTML and hard links and 2) improve the appearance just like it was done on Trac, but using Markdown features instead of Trac features.

• The git repository now has a script for getting translations straight from the nightly build (the same will be done for Vulkan shaders)
• Links to changesets in the format [25001] on Trac are correctly converted to 04ec75ed7e on Gitea (and I fixed the commit text in that specific revision)
• Important Trac keywords (regression, pathfinding, design, ...) now have a Gitea label, but I can't easily automatically convert keywords to labels, so I will add the labels manually after the migration.

Thanks in advance for your feedback!

[Carltonus]

Am ready to help on this "Gitea" fork, how do I at least begin? Don't want to lose my Trac data since I have contributed much on the wiki and added a handful of tickets...

[Stan`]

4 hours ago, Carltonus said:

Am ready to help on this "Gitea" fork, how do I at least begin? Don't want to lose my Trac data since I have contributed much on the wiki and added a handful of tickets...

Well you can request an account for Itms on gitea.itms.ovh, and look at every nook and cranny if there are broken/missing things.

You can help with the request above your post I suppose.

[Itms]

8 hours ago, Carltonus said:

Am ready to help on this "Gitea" fork, how do I at least begin? Don't want to lose my Trac data since I have contributed much on the wiki and added a handful of tickets...

Thank you for your interest! I sent you a PM with login information and some pointers. Please go through the wiki and tickets, test and update stuff as you wish and let me know if you run into an issue.

[Dunedan]

On 05/04/2024 at 12:20 PM, Itms said:

On 05/04/2024 at 12:02 PM, hyperion said:

Is the source of the whole migration available somewhere?

Yes, but in a private repo.

Not directly related to the migration, but is code or documentation for the set up of the new infrastructure available somewhere? I find being able to do pull requests for infrastructure as well pretty helpful.

On 04/04/2024 at 9:38 PM, Itms said:

Ariadne is a small redirect tool I wrote as a drop-in replacement for Trac or Phabricator when/if we decide to drop them.

I'm in favor of sunsetting Trac and Phab as fast as possible and as long as the links continue to work, by being redirected to Gitea.

On 05/04/2024 at 9:17 AM, Stan` said:

On 05/04/2024 at 9:08 AM, Itms said:

Yes, if forks don't count as we seem to have discovered, I'll keep the limit at 0 repos, it's better. I'm more worried about malicious software than forks... (also we need to stop using the word "fork" in that sense if we start using gitea...

True. I also disliked the fact you could make the diff private on Phab because that's kind of anti open source to me and moreover it broke the bot as well. If you're gonna have private code maybe don't share it at all...

There are valid cases for having stuff with limited visibility at first. I'm thinking about proposed changes and associated discussions which relate to serious security vulnerabilities, where vulnerability details shouldn't be public before a fix is shipped. However, I guess even with everything open on Gitea, we can handle such situation through other channels.

Also mind that limiting repositories for users doesn't help much against spam/malicious stuff, if forking existing repositories is still possible, as one can simply replace the content of the forked repository with something else.

On 05/04/2024 at 10:46 AM, Itms said:

On 05/04/2024 at 10:29 AM, hyperion said:

default branch name trunk -> main maybe

I'd like to keep trunk for historical/nostalgic reasons, and also because I'd like us to explicitly follow a trunk-based development model. I am open to discussion about that.

The vast majority of git repository uses "master" or "main" as their primary branch name. I'd prefer doing so for 0ad as well, as people are used to it. I actually can't remember having ever seen a git repository using a branch named "trunk" as their primary branch.

Just recently I was working on a git repository which used a different name for their primary branch and it was such a PITA, as muscle memory (git checkout ma<TAB>) didn't work.

On 08/04/2024 at 10:21 PM, Itms said:

On 08/04/2024 at 5:14 PM, Stan` said:

EDIT2: I don't have a strong opinion on it, but the fact Gitea is using gravatar might be suprising to some, IIRC the forum used to do that as well.

What do you mean about surprising? In terms of privacy you mean? (if yes, I tried to be precise in the privacy policy, and people who don't use Gravatar won't have their data sent there, so they are not supposed to complain unless I missed something).

For the PoC, getting the profiles from Gravatar allows one to have a sense of the look and feel of the platform and of the community interactions. I wouldn't like to see everyone with generated avatars...

For the actual migration, active users will be able to populate their profile, but IMO it would be a bit of a shame to leave all the old members of the community without a face. I'm very happy to see a profile pic associated with janwas or k776 for instance.

As Gitea is configured right now, for every request where a profile image is meant to be shown, a request for the user whose image is supposed to be shown is made to Gravatar, unless that user uploaded an own image in Gitea. As these requests happen for every visitor, Gravatar is able to track visitors of our Gitea instance, without them being able to consent to this first. I believe this doesn't comply with GDPR.

Upstream Gitea now also ships with OFFLINE_MODE=true by default (https://github.com/go-gitea/gitea/pull/28548), which disables requests to gravatar.com and other external services.

Did Phabricator even support Gravatars or were all profile images there uploaded by the users themselves? Maybe it'd be an alternative to migrate the existing profile images from Phabricator to Gitea instead.

[Stan`]

45 minutes ago, Dunedan said:

 

Did Phabricator even support Gravatars or were all profile images there uploaded by the users themselves? Maybe it'd be an alternative to migrate the existing profile images from Phabricator to Gitea instead.

I don't think it did, it generated them instead. Regarding migrating them, I don't know if that personal data processing is GDPR compliant (not that migrating the account is either, but at least for data minimization) One could also use forums profile pictures in that case. IIRC some users have different account names between Phab, Trac and Gitea

 

[Dunedan]

What I miss in Gitea right now (maybe I haven't just found it yet) is an overview of all (official) public repositories. https://gitea.itms.ovh/ just shows an empty way (aside from the menu and filter options on the right side).

 

  

13 minutes ago, Stan` said:

I don't think it did, it generated them instead.

The generated ones, which just contain the first letter of the name on top of a colored background are something I wouldn't consider migrating. What I was talking about are the ones uploaded manually by the users.

13 minutes ago, Stan` said:

Regarding migrating them, I don't know if that personal data processing is GDPR compliant (not that migrating the account is either, but at least for data minimization)

I'm pretty sure that's compliant, because only the underlying software changes, not the context and purpose the data is used for. With your argument migrating the accounts themselves wouldn't be compliant either.

13 minutes ago, Stan` said:

IIRC some users have different account names between Phab, Trac and Gitea

btw: Gitea also offers an Oauth2 provider with support for OpenID Connect (https://docs.gitea.com/development/oauth2-provider). That means we could use Gitea as central user management tool for all WFG related developer tools. That'd remove the need for different accounts for different services.

[Stan`]

10 minutes ago, Dunedan said:

What I miss in Gitea right now (maybe I haven't just found it yet) is an overview of all (official) public repositories. https://gitea.itms.ovh/ just shows an empty way (aside from the menu and filter options on the right side).

https://gitea.itms.ovh/explore/repos (When you click the explore menu at the top)

 

 

4 minutes ago, Dunedan said:

btw: Gitea also offers an Oauth2 provider with support for OpenID Connect (https://docs.gitea.com/development/oauth2-provider). That means we could use Gitea as central user management tool for all WFG related developer tools. That'd remove the need for different accounts for different services.

Indeed, although in the end, only Gitea should remain But yeah, Ideally I'd have one big account for everything 0 A.D. related one day. Only problem is the number of terms you'd have to agree to.

[Itms]

Thanks for the feedback!

1 hour ago, Dunedan said:

Not directly related to the migration, but is code or documentation for the set up of the new infrastructure available somewhere? I find being able to do pull requests for infrastructure as well pretty helpful.

It's in the staff forums. I am ready to update those threads as soon as changes happen. It would indeed be nice to store them in a repo, for instance a private repo on Gitea after the migration

1 hour ago, Dunedan said:

Just recently I was working on a git repository which used a different name for their primary branch and it was such a PITA, as muscle memory (git checkout ma<TAB>) didn't work.

I do like originality, but this is a very good point. I'll change trunk to main.

1 hour ago, Dunedan said:

As these requests happen for every visitor, Gravatar is able to track visitors of our Gitea instance, without them being able to consent to this first. I believe this doesn't comply with GDPR.

Indeed. I suppose they found a way to comply though, or else the service wouldn't work anymore. I'll disable Gravatar then.

1 hour ago, Dunedan said:

Maybe it'd be an alternative to migrate the existing profile images from Phabricator to Gitea instead.

I'm not going to perform extra work for that. Users will have to fill their profile themselves on Gitea.

6 minutes ago, Dunedan said:

btw: Gitea also offers an Oauth2 provider with support for OpenID Connect (https://docs.gitea.com/development/oauth2-provider). That means we could use Gitea as central user management tool for all WFG related developer tools. That'd remove the need for different accounts for different services.

Yes, that's nice to have! However I would like to avoid adding and maintaining too many devtools in the future If Gitea could become our one and only centralized platform for development that would be great.

[Itms]

While we are breaking paths:

I'm not sure wfg/0ad is a good repo path on Gitea. It's inconsistent with github and gitlab (where it's 0ad/0ad). I see that blender made the same decision and called the organization "blender" on their own gitea.

So I'm pondering whether to rename the "wfg" organization to "0ad" (I'll probably keep "Wildfire Games" as the real name).

Does that sound good? I don't like it much but it may objectively be better.

[Stan`]

9 minutes ago, Itms said:

While we are breaking paths:

I'm not sure wfg/0ad is a good repo path on Gitea. It's inconsistent with github and gitlab (where it's 0ad/0ad). I see that blender made the same decision and called the organization "blender" on their own gitea.

So I'm pondering whether to rename the "wfg" organization to "0ad" (I'll probably keep "Wildfire Games" as the real name).

Does that sound good? I don't like it much but it may objectively be better.

Well the need for those mirrors can be questionned after the migration as well. After all, they were there to offer a git workflow, but you still won't be able to use them to contribute...

Both are fine to me, IIRC I went wildfire-games/pyrogenesis

 

EDIT: Welcome back, blue Itms!

[Stan`]

@Itms Does Gitea has the equivalent of https://code.wildfiregames.com/owners/package/11/ ?

[Itms]

8 minutes ago, Stan&#x60; said:

@Itms Does Gitea has the equivalent of https://code.wildfiregames.com/owners/package/11/ ?

Yes that's one of the things I want to setup next week