Git migration

[zoot]

Well this has been discussed to death, but perhaps a two-repository idea would with with git sub modules: http://git-scm.com/book/en/Git-Tools-Submodules .

• Everything programers use is in one repository. A script clones each new revision of the master branch into a mirror branch of precompiled binaries.
  
• Artists check out an art repository, which bulls the recompiled branch of the engine repository automatically.
  

Wouldn't the binaries branch still be prohibitively big to download?

[Sonarpulse]

I don't know if this works with git sub modules, but there are so called "shallow clones" where git only clones the latest couple revisions. Problem is they come with a number of limitations which may make updating difficult.

[zoot]

I don't know if this works with git sub modules, but there are so called "shallow clones" where git only clones the latest couple revisions. Problem is they come with a number of limitations which may make updating difficult.

Well, technically, no one would be making changes to a shallow clone of the binaries branch itself, am I right? No one needs to manually edit the binaries. So if the shallow cloning is compatible with submodules this might just work...?

[zoot]

Here is my impression of how Sonarpulse's suggestion would work:

The idea is to split the current SVN repository into three Git repositories: Code for the engine code, Data for art and scripts and Binaries for autobuilt executables. Additionally, there will be a Programmers repository which has Code and Data as its submodules and an Artists repository which has Data and a shallow clone of Binaries as its submodules.

Comments?

[Sonarpulse]

You made a very elegant picture of it. Yes it's way more convoluted than the current way of delivering binaries, but I'm pretty sure moving to git (or something similar) will be absolutely essential when games beside 0ad are built on pyrogensis. So might as well make the change at some point ahead of time.

Edit, well I had "Code" and "Binaries" as two different branches of the same repository, and "Data" as being configured to pull either repository, but it doesn't really matter and the picture works either way.

Edited June 21, 2012 by Sonarpulse

[zoot]

Another advantage of Git is that it has no single points of failure. Say you decide to host the main branch on a Git server of your own on git.wildfiregames.com and set it up so it automatically pushes all commits to Github for backup purposes. If the main server on git.wildfiregames.com crashes, no problem - everyone just begin pushing to a branch on the Github mirror while the outage lasts. When the main server comes back up, Git can easily figure out how to merge all the intermediate commits back into the main branch.

The same goes for build automation. Instead of having a single autobuilder set up, any trustworthy person can set up an autobuilder of their own and have it push the new builds to their repository. When the main autobuild is down, people can just begin using one of these secondary autobuilds until the outage is resolved.

Edited August 23, 2012 by zoot

[Wijitmaker]

Does Git have the capability of integrating with trac as SVN does today?

[zoot]

Yes. I just can't speak to the performance, because I haven't tried it. That doesn't imply it's bad, though.

[MoLAoS]

https://git.wiki.kernel.org/index.php/GitSvnComparison

http://blog.webfaction.com/2011/05/trac-and-git-two-new-best-friends/

Note also that GAE uses Trac with GIT with no problems as far as I know, so it definitely should be possible to use it and be just as good as with SVN.

[mackwic]

And why don't just inspire a bit from the cathedral model ?

In the actual workflow, there is two componnents : our local repo and the svn master repo.

Commit are applied with the Trac system. Robust, but a bit heavy.

In the workflow I suggest, there is three: our local, the svn that becomes a "blessed" repo, and the repo of the team.

Contributions follows 3 steps:

...... Step 1 : git svn fetch origin (aKa the bare repo) && git checkout -b {feature, fix}-<subject_of_feature_or_number_of_fix>

We fetch the blessed repo without touching it and immediately create our own branch that's work on the specific goal we want to achieve.

For example, we want to add a nice control for the unit groups, we can our branch "feature-group_controller". If the goal is to fix the bug #D43D5734K, let's create the branche "fix-D43D5734K".

It's part of a common workflow very well explained here : http://nvie.com/post...ranching-model/

The branching model has another advantage that come in step 2.

...... Step 2 : git push {feature, fix}-<name_of_feature_or_fix> team_repo

We push our contribs to the repo of the team in a new branch, not in master. Why ? Because it permits:

• better understanding of the state of the commits : one branch, one name. Easy to follow the progression of the work on a specific stuff (better that the diff viewer)
  

• easier to pack commits : the team make a commit of merging. That commit explains wich bug is related, wich feature is added. If not stable, open a ticket, reverse, fix it. It it's stable, go to step 3
  

...... Step 3 : The team push the new stuff in master to the blessed repo so everyone can fetch it

...... Step 4 : go to step 1

You can use the illustration I make for a better understanding.

The benefits of this model are:

• every developper can use the advantages of git
  
• the team can use the system of light-branching of git to organize the patches
  
• don't touch anything that actually works (it's an important point)
  
• let the artist team commit to the svn repo because that will never break the game (the work is less critical)
  
• less noise on the blessed repo (I think it's also an important point)
  
• improve the quality of the blessed repo code base
  

I have to detail this last point. My point of view, as a programmer, is that the code quality of 0ad is very good and the game is exceptionnaly stable for an Alpha. As a programmer.

But, we have to keep in mind that not only programmers use the svn version but also gamers. Do they all think that 0ad is stable enough ? Adding this step will permit to handle the future with bigger demands and larger audience, where everyone doesn't have in mind that 0ad is in Alpha stage.

What do you think of that ?

workflow_0ad.pdf

[zoot]

In the workflow I suggest, there is three: our local, the svn that becomes a "blessed" repo, and the repo of the team.

Why retain SVN? Git can easily act as a 'blessed' repo.

[mackwic]

Simply because it works for everyone including the programmers. Tools are already configured for that, git offers a full compatibility, so why change ? Why spending manpower in something we don't need ?

[zoot]

For all the reasons that have been listed previously:

the trouble of splitting the codebase into two separate repositories running two different SCM systems

- People seem to be having a lot of trouble making SVN accept patches from their Git repos.

- SVN working copies don't carry their history around like Git working copies do, hence every time someone begins working from an SVN working copy, they break the Git side of the equation.

[Potter]

When using tools like tortoiseSVN or tortoiseGit, normal users don't notice much difference except that they have to do an extra push. So maybe, switching entirely to Git might work out...

Edited August 27, 2012 by Potter

[zoot]

The idea is to split the current SVN repository into three Git repositories: Code for the engine code, Data for art and scripts and Binaries for autobuilt executables. Additionally, there will be a Programmers repository which has Code and Data as its submodules and an Artists repository which has Data and a shallow clone of Binaries as its submodules.

I've made a demo setup to show off the viability of the configuration described above. These are the repositories:

code: https://github.com/zootzoot/demo-code

data: https://github.com/zootzoot/demo-data

binaries: https://github.com/z...t/demo-binaries

programmers: https://github.com/z...emo-programmers

artists: https://github.com/z...ot/demo-artists

The big attraction, of course, is the artists repo. This is how much space an updated binary takes up, uncompressed, in this setup: 255 bytes (compared to a few megabytes in SVN). After ten years of daily builds, that's still just 255 bytes * 365 * 10 = 930750 bytes or 0.88 megabytes, uncompressed.

(But do check my math, lol )

[zoot]

Actually, I missed the 'shallow clone' part in the above. It would probably be necessary to do something like this: http://bugsquash.blogspot.com/2010/11/shallow-submodules-in-git.html?m=1

[zoot]

I don't have much insight to add to this thread except that a migration to Git should be somewhere in the top 5 tasks. Moving to Git allows for a quicker development workflow and opens doors for many other contributors. (Weak points, I know ) I believe progress is stalled due to an auto builder.

Would it perhaps be possible to set up an autobuilder on a server held in SPI's name and fix #1819 in the process?

[zoot]

There doesn't seem to be too much progress on this, though some plans were drawn up. Would anyone be opposed to doing it like this:

Would it perhaps be possible to set up an autobuilder on a server held in SPI's name and fix #1819 in the process?

It seems to me that a server where multiple trusted people have access, under the orderly conditions offered by SPI, would be the best guarantee against it stagnating in someone's basement (or whereever it is these things tends to stagnate ).

[Jeru]

I am the liaison to SPI, I can forward specific requests but I have very little technical knowledge so I will need help phrasing them.

We also pay for hosting already so maybe we can use our existing server.

[zoot]

I am the liaison to SPI, I can forward specific requests but I have very little technical knowledge so I will need help phrasing them.

Sure. If people are supportive of the idea, we'll need to flesh out a more specific plan, which you could then involve them in. I imagine it would involve leasing a virtual server at AWS or somewhere similar.

We also pay for hosting already so maybe we can use our existing server.

Is this the www.wildfiregames.com and/or www.play0ad.com servers? I assumed those were web servers, but I could be mistaken of course. For running things like the autobuilder we would need a full server with shell access.

[alpha123]

Is this the www.wildfiregames.com and/or www.play0ad.com servers? I assumed those were web servers, but I could be mistaken of course. For running things like the autobuilder we would need a full server with shell access.

At least trac.wildfiregames.com runs on Philip's own server, which of course he has complete access to. IIRC he was planning to run the autobuilder on that.

[zoot]

At least trac.wildfiregames.com runs on Philip's own server, which of course he has complete access to. IIRC he was planning to run the autobuilder on that.

I guess that is the process that is stalled though.