Jump to content

Why not moving to Git?


Recommended Posts

5 hours ago, hyperion said:

man git-worktree ?

As far as I know Jenkins doesn't support worktree out of the box.

So what it does is create one folder per pipeline which in theory leads to 9GB of useless files per pipeline. (My git repo is a bit lighter than the current one, and that's with depth 1)

On linux we use zfs pools and we mount that into dockers. So we have one XGB repo and only the extra binaries in eached jenkins folder.

 

Link to comment
Share on other sites

https://trac.wildfiregames.com/wiki/JenkinsSetup isn't really helpful in understanding the current setup or what you want to do, neither zfs, docker, phab integration, full/partial builds, machines nor test runs are mentioned.

 

Anyway I thought worktree support was added a few years back, whether this might be of use here I don't know

Link to comment
Share on other sites

2 hours ago, hyperion said:

https://trac.wildfiregames.com/wiki/JenkinsSetup isn't really helpful in understanding the current setup or what you want to do, neither zfs, docker, phab integration, full/partial builds, machines nor test runs are mentioned.

 

Anyway I thought worktree support was added a few years back, whether this might be of use here I don't know

I had no idea those pages existed. I wasn't the one to set up the CI, Itms did, so I kind of picked it up when it started failing.

Basically Phabricators and now Gitea trigger webhooks that call Jenkins pipelines. Those run and then pingback the Phabricator server (or don't and then you get stuck builds)

The Jenkins pipelines can be found here pipelines in ps/trunk/build/jenkins – Wildfire Games I made some fixes recently I should commit when I get some free time.

I made the first experiments with the ligther git on gitlab.wildfiregames.com and managed to build the game on macOS and on FreeBSD, because the setup is simpler. 

On linux you've got at least two docker images, gcc7 and clang7 that are mounted when the CI runs. The idea was to not have to install any dependencies on the host VM. 

Jenkins is a VM, the Linux Autobuilder is another VM, the Windows Autobuilder is another one, and the macOS one is running courtesy of MacStadium on their servers.

This script is ran periodically and restores the zpool images docker-rebuild.Jenkinsfile in ps/trunk/build/jenkins/pipelines – Wildfire Games

This script is run at each commit and updates the zpool images docker-svn.Jenkinsfile in ps/trunk/build/jenkins/pipelines – Wildfire Games

This one makes the releases macos-all-bundles.Jenkinsfile in ps/trunk/build/jenkins/pipelines – Wildfire Games

 

 

 

 

Link to comment
Share on other sites

Thanks, I probably have a rough idea now.

on commit to trunk:
    for each builder in freebsd windows-v2015 linux-gcc7 linux-clang7 macos:
        restore snapshot ${builder}:trunk
        do partial build
            on success:
                run tests
                    on failure:
                        go boink
                update snapshot ${builder}:trunk
            on failure:
                go boink
        keep logs

Well, except macos, hurray to MacStadium

Should be possible to use worktrees if careful, but clones are robuster.

Another hack to save another 30+ GB

$ git checkout $(git commit-tree -p HEAD $(git hash-object -t tree /dev/null) < /dev/null)
save/restore snapshot
$ git checkout HEAD^

The combination would shrink your repo to a couple kB and the snapshot to basically only build artifacts.

 

PS: windows-v2015 has c++17 support? ;)

  • Like 1
Link to comment
Share on other sites

  • 1 month later...
  • 2 weeks later...
  • 3 months later...
6 hours ago, Ceres said:

@Stan`
Will there be something like PRs?

Yeah. I have to think about it but basically in my head you'll be able (when granted access for obvious reasons) to create branches on the main repo. (Forks would be nice but it seems it would take too much space on the server)

  • Like 1
Link to comment
Share on other sites

It's most probably EXT4 but from what I've seen gitea copies the whole .git folder. So each time someone makes a patch we could get an extra 5GB and that would quickly be unmanageable. That VM is already 500GB large which is starting to be an issue for backups.

Feel free to investigate :)

Link to comment
Share on other sites

6 hours ago, Ceres said:

Do you own the server/VM and can change the FS?

We do and we can, although not that easily. First we need to migrate everything. The same vm is used for gitea phabricator and trac.

6 hours ago, Ceres said:

Would it be possible to exclude certain assets from forks? If somebody wants to contribute (by forking & PR'ing) to the code but not to SFX or GFX, would that be possible (and save space)?

Not really. You'd need to split the repo into subrepos for that and some team members don't like the idea because it makes contributing to both much harder cause you have to write N PRs where N is the number of repos. Again not an issue with a lot of people much more annoying with little amount of contributors

  • Thanks 1
Link to comment
Share on other sites

On 10/02/2023 at 1:38 PM, Stan&#x60; said:

Don't think you can do cross website pull requests either. 

You can't do official "Pull requests"

But my understanding based on this SO answer: https://stackoverflow.com/a/15057469 by Eevee (CC-BY-SA) is that people could just fork the gh/gitlab repository, do whatever pull request they want over there in separate feature branches (we can even use the free "actions" to run some test) and the we could just merge those manually or automatically back into the gitea one.

The question is more: do we want to accept pr's from there/ does it make sense to split the development discussion in even more places/ is this worth the effort.

 

  • Thanks 1
Link to comment
Share on other sites

19 hours ago, Stan&#x60; said:

It's most probably EXT4 but from what I've seen gitea copies the whole .git folder.

Then a COW filesystem like Btrfs might be best bet as it supports "offline" de-duplication so even if gitea doesn't cp --reflink you'd be fine. https://www.man7.org/linux/man-pages/man2/ioctl_fideduperange.2.html

Added bonus of using Btrfs, backups are easier.

 

Don't know Go but I'll grab the gitea sources, maybe I can figure out what it does.

 

9 hours ago, Stan&#x60; said:

The same vm is used for gitea phabricator and trac.

They don't share subdomain so splitting sounds like a good idea regardless?

  • Thanks 1
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

×
×
  • Create New...