Version Control system

[Stan`]

46 minutes ago, bb_ said:

This is an unfair comparison.

It's not if you consider the only problem with git to be disk space. Whether it's on the server or on the client. It's a good comparison to show it doesn't really matter.

46 minutes ago, bb_ said:

Not sure what the official line is (I heard some conflicting info from @Freagarach on irc). Would be good to figure out.

Agreed.

 

[Freagarach]

9 hours ago, bb_ said:

It seems that epriestly (the old maintainer of phab) is just posting some suggestions on what one could do. It doesn't seem to contain any evidence that svn is/will actually be dropped. Not sure what the official line is (I heard some conflicting info from @Freagarach on irc). Would be good to figure out.

I asked on their IRC whether they support SVN, which they confirmed as doing already. No hint about dropping support.
https://wm-bot.wmcloud.org/browser/index.php?start=12%2F08%2F2021&end=12%2F15%2F2021&display=%23phorge

[crazyking]

Sorry for dropping in on the discussion like this but I thought I could give my 2 cents on this. I am not a professional video game developer by any means but committing the art assets to the repo, be it art or sound, is very unusual. I have some experience with web development and I want to give an example: suppose we want to create a social media platform using Git or SVN. We create a folder in the server where we are hosting it and send the pictures that users upload to that folder. PHP, the language used for out project, can store and read the files from the folder, and everything works nicely. If for some reason we want to rollback our site (revert everything to a sate in the past) we will need a cached version of that folder containing all the images that existed in that time. We can even set up a script to create a cache everyday or, if our site isn't that big, every week.

Take what I say here with a grain of salt but I think that what most developers do is setup a script (maybe even a makefile) that downloads the correct folder containing the resources necessary to build a certain version of the game. Something I have not seen mentioned yet is that including the art in the repository will bloat the commit history because some of these resources, specially graphics, get changed frequently.

I also believe that Gitlab will host open source projects without any limitations after going through a "background check" to make sure the project is relevant enough. Given 0AD has existed for over a decade I'm sure Wildfire Games could become one of their partners (it's what they call it, GitLab Open Source Partners). Personally I prefer GitLab because their code review features are better and they don't have questionable initiatives like Github but either will work very well.

Another point is that maybe the issue tracker could be integrated into GitLab/Github but I don't know how complex the tracker being used for this game is. I've seen some other projects not want to migrate because they needed to have a lot of information when filing an issue (GPU, OS, version used, last known good version, etc etc etc) and neither platform were satisfactory in that regard.

[smiley]

(in professional video game development, binary assets are committed along with scripts to the same repository. It's standard procedure and the tools are specifically designed to handle that. Plastic or Perforce are pretty expensive though. There is a consideration that most professional video games are also not as code heavy as they don't usually develop engines.)

[crazyking]

Oh

I'm not sure why you mentioned Plastic and Perforce though, wouldn't Github/Lab be better?

[s0600204]

On 17/12/2021 at 8:40 PM, bb_ said:

On 08/12/2021 at 11:05 AM, Stan` said:

Do note however that they are considering dropping subversion as mentioned by @s0600204 here.

 

In the initial flurry of interest from/discussion between various concerned parties to continue Phabricator in some form, one user proposed dropping support for subversion and mercurial (so as to permit pushing directly to a phabricator-managed git repo with "git push" instead of having to use arcanist). There was some push back from others about losing mercurial support but not a single voice against the proposed loss of support for subversion. (Link to the discussion - requires login.)

Since the fork ("Phorge") has established itself, the only mention to subversion that I can find (in the publicly/guest accessible bits, anyway) is in the list of "projects" for the various components of Phorge. As you might note - a project hasn't been created for subversion, although it is in the list. According to the comments below the list, projects will be created on a "as-needed" basis. So it looks like there's no plan to drop subversion support... but also no-one has stepped forward to express an interest in maintaining it either. (Unlike mercurial, who does have someone.)

[Stan`]

@s0600204< Thanks for chiming in! How do you feel about git, gitlab, splitting repos, basically the things mentioned in the thread?

[s0600204]

My thoughts:

• I like git;
• (Bit controversial, but) I also like Phabricator (and, despite its flaws, arcanist!)
  • (So I'd like to see consideration of migrating across to Phorge - once the Phorge team release a guide as to how to do that!)
• I'd like to see the repository split to differentiate between the engine and the game;
• I'd like to see binaries off-loaded via git-lfs or some other suitable tool;
• I'd like to for us to stop committing Windows-only binaries into the repository, but make them available through other means for those unable (or unwilling) to build themselves:
  • The same process could potentially be used to provide "nightly" builds for macOS and some Linux distros as well;
  • (On a related note, I'd also like to see a build script for Windows like the one we have for macOS - e.g. it also fetches and builds the libraries/dependencies. But that's for another thread.)
• We should continue to be self-hosted where possible;

[smiley]

On 20/12/2021 at 4:40 AM, crazyking said:

I'm not sure why you mentioned Plastic and Perforce though, wouldn't Github/Lab be better?

<offtopic> Game development is rarely done on Git. Plastic and Perforce are the dominant two systems. IIRC, even SVN is more popular than Git. Most major third party engines don't even have integrated support for Git. The biggest difference is that 0AD uses a dedicated engine which makes either option undesirable as there are a lot of code too. </offtopic>

[vladislavbelov]

16 minutes ago, smiley said:

Game development is rarely done on Git. Plastic and Perforce are the dominant two systems. IIRC, even SVN is more popular than Git.

Are there some numbers? (Just wondering, I've met various approaches)

17 minutes ago, smiley said:

Most major third party engines don't even have integrated support for Git.

I suppose it may not tell about Git, but products like Plastic/Perforce. Because they are commercial and companies might pay for their integration.

[smiley]

7 minutes ago, vladislavbelov said:

Are there some numbers? (Just wondering, I've met various approaches)

No concrete numbers as most game devs are proprietary, but there are small scale surveys done on gamedev forums and those two are almost always at the top (Perforce and its not even close).

9 minutes ago, vladislavbelov said:

Because they are commercial and companies might pay for their integration.

Perhaps. I think Plastic paid Unity though. But from anecdotal evidence I have heard, Perforce does a better job with large binaries.

[Stan`]

I really don't think Perforce and Plastic are the solution here. We've got the money for it, but it would be a shame to be dependent on proprietary software. At least they seem to support multiple platforms. I guess it would be a nightmare for maintainers to have to install such tools though.

On 26/12/2021 at 6:27 PM, s0600204 said:

I'd like to see binaries off-loaded via git-lfs or some other suitable tool;

Still not convinced we actually need LFS or that the benefits outweigh the constraints. At the end of the day you'll still have to download binaries. Less of course, but still

Git also has the advantage of being decentralized, where svn isn't which I think mean that if the server ever falls / fails we may lose the history.
Maybe relevant: https://backlog.com/blog/git-vs-svn-version-control-system/

I'm not sure I said it before, but I'd really like to lessen my admin sys workload which is why I proposed to replace Jenkins, Trac, and Phabricator by Gitlab. And hence the reason I want to switch to git.

On 26/12/2021 at 6:27 PM, s0600204 said:

• I'd like to see the repository split to differentiate between the engine and the game;
• I'd like to for us to stop committing Windows-only binaries into the repository, but make them available through other means for those unable (or unwilling) to build themselves:
  • The same process could potentially be used to provide "nightly" builds for macOS and some Linux distros as well;
  • (On a related note, I'd also like to see a build script for Windows like the one we have for macOS - e.g. it also fetches and builds the libraries/dependencies. But that's for another thread.)

I'm also for this.

[vv221]

17 hours ago, Stan&#x60; said:

I'm not sure I said it before, but I'd really like to lessen my admin sys workload which is why I proposed to replace Jenkins, Trac, and Phabricator by Gitlab. And hence the reason I want to switch to git.

This is the main reason I’m all in favour of GitLab, because this is the one I can help with. Including for long term maintenance.

I think that for equivalent features, volunteers for administration/maintenance is what should drive the choice of tools. So if other people feel ready to dedicate some of their time for the maintenance of the development environment, it might be a good idea to make these intentions known

[smiley]

18 hours ago, Stan&#x60; said:

Still not convinced we actually need LFS or that the benefits outweigh the constraints. At the end of the day you'll still have to download binaries. Less of course, but still

It helps with local repository size as blobs are not tracked with all their history. Its not an issue if you shallow clone (you probably should, if you have to blame that far back, just use the web interface).

Build output should not be committed, CI artifacts would do. I am not sure whats the solution for people who want to be up to date with every change while also not downloading all those unchanged DLLs. Perhaps,https://wildfiregames.com/forum/topic/65460-0-ads-focus-on-balance-has-crippled-its-design/?do=findComment&comment=470537 is the solution. Such users are not really interested in the history usually, just need up-to-date binaries.

[Stan`]

15 minutes ago, smiley said:

Build output should not be committed, CI artifacts would do. I am not sure whats the solution for people who want to be up to date with every change while also not downloading all those unchanged DLLs. Perhaps,https://wildfiregames.com/forum/topic/65460-0-ads-focus-on-balance-has-crippled-its-design/?do=findComment&comment=470537 is the solution. Such users are not really interested in the history usually, just need up-to-date binaries.

Initial git migration plan included md5/sha1 checks to prevent useless downloads.

I think if we go the “launcher/updater” route it should be the mod mod, not an external program.

17 minutes ago, smiley said:

It helps with local repository size as blobs are not tracked with all their history. Its not an issue if you shallow clone (you probably should, if you have to blame that far back, just use the web interface).

git lfs import -y seems to work well enough and indeed only the .git contain binaries. If we remove all the file in binaries/system from history I'm not sure 5GB is that much to download, but I live in Europe and internet is cheap.

 

[Stan`]

Here is my current test status to migrating to git.

file.sh/.cmd is to migrate, gitmirror contains a gitignore and gitattributes which are sadly ignored and must be a regexp
authors contains the list of contributors, haven't filled all the emails yet not sure whether I should.

Question is also what do with Phab links (references to #ticket should still work with gitlab) and whether to keep the metadata.

git-migration-test-bed.zip

[wraitii]

It seems to me that most people would agree with an experiment on a self-hosted gitlab instance.
It seems mostly consensual to not store large binaries, including windows libs, in the repo (not the main repo anyways).

I think it'd be great to start experimenting with the above setup. Steps I see:

• Having a script to copy the svn history over to git.
  • I don't think this is vital to get 100% right for the experiment because we can still do the other steps.
• Creating scripts to handle binaries automatically, at the very least on Windows.
  • Vital on windows, probably good on other platforms too.
  • This includes "storing them somewhere"
  • Once removed, we can see how heavy the repo is, and the make a decision on LFS, IMO.
• Set up some CI/CD with this. The existing jenkins script will need to be reworked. That's already quite a bit of work.
  • IMO we need this to work well
• Set up ticketing & issues & stuff like that. Test out the workflow.
• Have a script to download Phabricator metadata (issues, comments) & ideally port-them over to Gitlab.
  • I would prefer to port 100% of the data, but not necessarily keep 100% of the "formatting". If the inline comments aren't inlined properly on gitlab, it's perhaps OK.
  • Again, for experimenting this can be refined later.

If the above steps all work, then we already have a mirror that works, and we can refine the steps for an actual migration.

I can offer help with these steps.