Jump to content

Why not moving to Git?

Recommended Posts

50 minutes ago, andy5995 said:

Reading and replying quickly, but I think understand. Caching is available in github actions, but I've found that sometimes errors happen due to the cache not getting updated correctly.

Due to a long discussion in the team we have to stay away from Github and self host. We sometimes have issues on the CI too, but it's generally working.

51 minutes ago, andy5995 said:

Another way is to create docker image with everything and push it docker hub. Then pull that image and start a container in your CI workflow? But that has disadvantages too, obviously.

Can't really be pushing a 5.1GB image on docker, also only works for linux and not macOS nor windows as they need their specific tools and compilers.

  • Thanks 1
Link to comment
Share on other sites

5 hours ago, hyperion said:

man git-worktree ?

As far as I know Jenkins doesn't support worktree out of the box.

So what it does is create one folder per pipeline which in theory leads to 9GB of useless files per pipeline. (My git repo is a bit lighter than the current one, and that's with depth 1)

On linux we use zfs pools and we mount that into dockers. So we have one XGB repo and only the extra binaries in eached jenkins folder.


Link to comment
Share on other sites

2 hours ago, hyperion said:

https://trac.wildfiregames.com/wiki/JenkinsSetup isn't really helpful in understanding the current setup or what you want to do, neither zfs, docker, phab integration, full/partial builds, machines nor test runs are mentioned.


Anyway I thought worktree support was added a few years back, whether this might be of use here I don't know

I had no idea those pages existed. I wasn't the one to set up the CI, Itms did, so I kind of picked it up when it started failing.

Basically Phabricators and now Gitea trigger webhooks that call Jenkins pipelines. Those run and then pingback the Phabricator server (or don't and then you get stuck builds)

The Jenkins pipelines can be found here pipelines in ps/trunk/build/jenkins – Wildfire Games I made some fixes recently I should commit when I get some free time.

I made the first experiments with the ligther git on gitlab.wildfiregames.com and managed to build the game on macOS and on FreeBSD, because the setup is simpler. 

On linux you've got at least two docker images, gcc7 and clang7 that are mounted when the CI runs. The idea was to not have to install any dependencies on the host VM. 

Jenkins is a VM, the Linux Autobuilder is another VM, the Windows Autobuilder is another one, and the macOS one is running courtesy of MacStadium on their servers.

This script is ran periodically and restores the zpool images docker-rebuild.Jenkinsfile in ps/trunk/build/jenkins/pipelines – Wildfire Games

This script is run at each commit and updates the zpool images docker-svn.Jenkinsfile in ps/trunk/build/jenkins/pipelines – Wildfire Games

This one makes the releases macos-all-bundles.Jenkinsfile in ps/trunk/build/jenkins/pipelines – Wildfire Games





Link to comment
Share on other sites

Thanks, I probably have a rough idea now.

on commit to trunk:
    for each builder in freebsd windows-v2015 linux-gcc7 linux-clang7 macos:
        restore snapshot ${builder}:trunk
        do partial build
            on success:
                run tests
                    on failure:
                        go boink
                update snapshot ${builder}:trunk
            on failure:
                go boink
        keep logs

Well, except macos, hurray to MacStadium

Should be possible to use worktrees if careful, but clones are robuster.

Another hack to save another 30+ GB

$ git checkout $(git commit-tree -p HEAD $(git hash-object -t tree /dev/null) < /dev/null)
save/restore snapshot
$ git checkout HEAD^

The combination would shrink your repo to a couple kB and the snapshot to basically only build artifacts.


PS: windows-v2015 has c++17 support? ;)

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Create New...