Jump to content

Why not moving to Git?

balduin
 Share

Recommended Posts

Stan` Consul

Stan`

Posted
Posted
5 hours ago, hyperion said:

man git-worktree ?

As far as I know Jenkins doesn't support worktree out of the box.

So what it does is create one folder per pipeline which in theory leads to 9GB of useless files per pipeline. (My git repo is a bit lighter than the current one, and that's with depth 1)

On linux we use zfs pools and we mount that into dockers. So we have one XGB repo and only the extra binaries in eached jenkins folder.

 

Link to comment
Share on other sites
  • Replies 113
  • Created
  • Last Reply

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Popular Posts

leper
leper

(I guess I should write a slightly longer answer to this and related questions since this seems to pop up regularly.) TL;DR: SVN works. Why don't you use git, you can do so right now (either one

Stan`
Stan`

A migration to git is planned, also there is an official repo on github. See: http://trac.wildfiregames.com/ticket/1814 http://trac.wildfiregames.com/ticket/1816 http://trac.wildfiregam

feneur
feneur

Moving to Github was not really considered because we want to be in control of things ourselves and not have to depend on another service working or even existing years down the line.

hyperion Primus Pilus

hyperion

Posted
Posted

https://trac.wildfiregames.com/wiki/JenkinsSetup isn't really helpful in understanding the current setup or what you want to do, neither zfs, docker, phab integration, full/partial builds, machines nor test runs are mentioned.

 

Anyway I thought worktree support was added a few years back, whether this might be of use here I don't know

Link to comment
Share on other sites
Stan` Consul

Stan`

Posted
Posted
2 hours ago, hyperion said:

https://trac.wildfiregames.com/wiki/JenkinsSetup isn't really helpful in understanding the current setup or what you want to do, neither zfs, docker, phab integration, full/partial builds, machines nor test runs are mentioned.

 

Anyway I thought worktree support was added a few years back, whether this might be of use here I don't know

I had no idea those pages existed. I wasn't the one to set up the CI, Itms did, so I kind of picked it up when it started failing.

Basically Phabricators and now Gitea trigger webhooks that call Jenkins pipelines. Those run and then pingback the Phabricator server (or don't and then you get stuck builds)

The Jenkins pipelines can be found here pipelines in ps/trunk/build/jenkins – Wildfire Games I made some fixes recently I should commit when I get some free time.

I made the first experiments with the ligther git on gitlab.wildfiregames.com and managed to build the game on macOS and on FreeBSD, because the setup is simpler. 

On linux you've got at least two docker images, gcc7 and clang7 that are mounted when the CI runs. The idea was to not have to install any dependencies on the host VM. 

Jenkins is a VM, the Linux Autobuilder is another VM, the Windows Autobuilder is another one, and the macOS one is running courtesy of MacStadium on their servers.

This script is ran periodically and restores the zpool images docker-rebuild.Jenkinsfile in ps/trunk/build/jenkins/pipelines – Wildfire Games

This script is run at each commit and updates the zpool images docker-svn.Jenkinsfile in ps/trunk/build/jenkins/pipelines – Wildfire Games

This one makes the releases macos-all-bundles.Jenkinsfile in ps/trunk/build/jenkins/pipelines – Wildfire Games

 

 

 

 

Link to comment
Share on other sites
hyperion Primus Pilus

hyperion

Posted
Posted

Thanks, I probably have a rough idea now. 

on commit to trunk:
    for each builder in freebsd windows-v2015 linux-gcc7 linux-clang7 macos:
        restore snapshot ${builder}:trunk
        do partial build
            on success:
                run tests
                    on failure:
                        go boink
                update snapshot ${builder}:trunk
            on failure:
                go boink
        keep logs

Well, except macos, hurray to MacStadium

Should be possible to use worktrees if careful, but clones are robuster.

Another hack to save another 30+ GB 

$ git checkout $(git commit-tree -p HEAD $(git hash-object -t tree /dev/null) < /dev/null)
save/restore snapshot
$ git checkout HEAD^

The combination would shrink your repo to a couple kB and the snapshot to basically only build artifacts.

 

PS: windows-v2015 has c++17 support? ;)

  • Like 1
Link to comment
Share on other sites
  • 1 month later...
  • 2 weeks later...
  • 3 months later...
Stan` Consul

Stan`

Posted
Posted

It's most probably EXT4 but from what I've seen gitea copies the whole .git folder. So each time someone makes a patch we could get an extra 5GB and that would quickly be unmanageable. That VM is already 500GB large which is starting to be an issue for backups.

Feel free to investigate :)

Link to comment
Share on other sites
Stan` Consul

Stan`

Posted
Posted
6 hours ago, Ceres said:

Do you own the server/VM and can change the FS?

We do and we can, although not that easily. First we need to migrate everything. The same vm is used for gitea phabricator and trac.

6 hours ago, Ceres said:

Would it be possible to exclude certain assets from forks? If somebody wants to contribute (by forking & PR'ing) to the code but not to SFX or GFX, would that be possible (and save space)?

Not really. You'd need to split the repo into subrepos for that and some team members don't like the idea because it makes contributing to both much harder cause you have to write N PRs where N is the number of repos. Again not an issue with a lot of people much more annoying with little amount of contributors

  • Thanks 1
Link to comment
Share on other sites
maroder Primus Pilus

maroder

Posted
Posted
On 10/02/2023 at 1:38 PM, Stan&#x60; said:

Don't think you can do cross website pull requests either. 

You can't do official "Pull requests"

But my understanding based on this SO answer: https://stackoverflow.com/a/15057469 by Eevee (CC-BY-SA) is that people could just fork the gh/gitlab repository, do whatever pull request they want over there in separate feature branches (we can even use the free "actions" to run some test) and the we could just merge those manually or automatically back into the gitea one.

The question is more: do we want to accept pr's from there/ does it make sense to split the development discussion in even more places/ is this worth the effort.

 

  • Thanks 1
Link to comment
Share on other sites
hyperion Primus Pilus

hyperion

Posted
Posted
19 hours ago, Stan&#x60; said:

It's most probably EXT4 but from what I've seen gitea copies the whole .git folder.

Then a COW filesystem like Btrfs might be best bet as it supports "offline" de-duplication so even if gitea doesn't cp --reflink you'd be fine. https://www.man7.org/linux/man-pages/man2/ioctl_fideduperange.2.html

Added bonus of using Btrfs, backups are easier.

 

Don't know Go but I'll grab the gitea sources, maybe I can figure out what it does.

 

9 hours ago, Stan&#x60; said:

The same vm is used for gitea phabricator and trac.

They don't share subdomain so splitting sounds like a good idea regardless?

  • Thanks 1
Link to comment
Share on other sites
  • Dunedan unpinned this topic

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
 Share
Go to topic listing
×
×
  • Create New...