Jenkins our butler (build server and automated testing)

[Yves]

Some probably know Jenkins already. It's an Open Source tool for automated building and testing.

The idea is to notice bugs as soon as possible after they are introduced. Let's compare this situation with and without Jenkins to show what I mean (not using Bob and Alice as names for once):

Situation: Yves breaks a test only in debug builds

Currently (without Jenkins): Yves has tested his change in the game and he has launched the tests in release mode. Everything works as expected and he commits.

Two weeks later there's a forum post about a strange problem with the tests. Leper and Sander both can't reproduce it in their first try. On their systems the tests work well. After a few questions in the forum they figure out it only happens in debug builds and after an hour of collaborative analysis they find and fix the problem.

Unfortunately some other commits were based on this change during the last two weeks and more changes need to be done.

Future (with Jenkins): Yves has tested his change in the game and he has launched the tests in release mode. Everything works as expected and he commit.

Jenkins does some automated building and testing with various configurations. One hour later, Yves gets a mail from Jenkins informing him that the tests don't work in debug mode. Yves has worked on the change just an hour ago and everything is still fresh in mind and he can solve the problem quickly because he already knows where to search for it.

That's an optimal case but there are a lot of situations where automated testing can be useful.

Just think about how often someone broke a test in the past which caused a lot of unnecessary noise on IRC when this person could just have fixed it an hour after breaking it and nobody would have noticed.

I you have some time I recommend watching this video, it explains the topic quite well in my optinion (the general part in the beginning at least):

I also recommend this short and general overview of what continuous integration means:

I have started experimenting a bit with Jenkins and figuring out what can be done and what could be useful for us.

Here's what I have so far:

Basic building and testing procedure:

1. Jenkins checks for new svn revisions every 10 minutes. It uses svn update for the checkout instead of doing a fresh checkout everytime.
2. Runs update-workspaces.sh with the newly introduced option "--jenkins-tests"
3. Runs make
4. Runs our test executable and redirects the output to an xml file.
5. Fixes a little incompatibility with the XML schema with sed and then uses the generated XML to display the results of the tests

These steps get exectued for both debug and release configurations.

Jenkins has a web GUI which displays and visualizes all kinds of useful information.

Unfortunately I don't have a DMZ setup at home, so I don't want to make the server accessible to the internet.

But I've made some screenshots to give you an impression of how it looks:

This is the overview showing a trend (currently a dark cloud because many builds failed while I was testing).

On the bottom left corner you see build with two different configurations (debug and release) running.

Here you see the tests listed. They all passed in this run.

There are also nice graphs showing a history of the tests either for all tests or individual tests.

Of course the graph is not very interesting with only two builds and 100% tests passed in both builds:

There's also a build history showing when which tests succeeded or failed:

Last but not least, Jenkins keeps track of commits and automatically adds people who do the commits to its configuration.

We could assign email addresses to people and send mails if one of their commits doesn't pass a build or a test (I heard you can do that, but I haven't tested it yet).

I'll continue improving my Jenkins setup to figure out what can be done and what's useful for us.

At some point we should probably think about configuring a Jenkins server which is accessible through the internet.

At the moment this would be too early IMO. I know that Philip already has a sever, but as far as I know he only uses it for building and hasn't integrated tests or anything like that.

[niektb]

Might do good in case of preventing bugs. Looking forward to it!

[fabio]

Even better it should check every patch posted on the bug system and check it before anyone commits it. No idea if trac support it (github does, see here e.g. https://github.com/residualvm/residualvm/pull/840 ).

Eventually I may provide a server for it.

[historic_bruno]

It's a great initiative, thanks for looking into it It would be best to have at least one Windows server, at least one Linux server, and at least one OS X server. The first two I think are pretty easy, Windows and Linux VMs can be run on many servers, are plentiful and cheap. Not so for the OS X server due to Apple's licensing, there are some virtualization/cloud services for Macs out there, but maybe we can find a community member with a Mac they are willing to setup and "donate" as a server.

Anyway, that's further down the line and just having any Jenkins autobuild/testing server setup would be an awesome first step. Philip's work on this was quite promising and interesting, so I'm glad to see it moving forward again.

[Yves]

Thanks for the input and the support!

At the moment I'm looking into how it works technically and what our possibilities are.

There are a lot of potential designs but in the first step I don't worry about productive development processes or licensing topics.

I mean we could do something like Mozilla with their mozilla-inbound repository where stuff gets committed first before it goes into mozilla-central.

We could also checkout custom github repositories or branches and run our tests there. Maybe we could download and apply smaller patches.

All that not only needs to work technically, but there are security concerns (we are downloading and executing code) and processes need to be defined.

For a solution like the inbound-repository for example, there are a lot of unanswered questions at the moment. Are there manual steps involved? Will committing be queued when one patch is being tested? What happens if a change that comes earlier in the queue breaks the a later patch in the queue etc.

I plan to look into how different systems (Windows, Linux, Mac) can be connected and if data can be aggregated at the same place.

I'll focus on the technical aspect first.

[Josh]

I would be amazing if we could get this working, it's been something we have been wanting far too long now. As far as how to handle failed tests, maybe we could just run it on the main branch and revert failing commits? (That might be a better use case for git though)

[Swyter]

It's a great initiative, thanks for looking into it It would be best to have at least one Windows server, at least one Linux server, and at least one OS X server. The first two I think are pretty easy, Windows and Linux VMs can be run on many servers, are plentiful and cheap. Not so for the OS X server due to Apple's licensing, there are some virtualization/cloud services for Macs out there, but maybe we can find a community member with a Mac they are willing to setup and "donate" as a server.

Anyway, that's further down the line and just having any Jenkins autobuild/testing server setup would be an awesome first step. Philip's work on this was quite promising and interesting, so I'm glad to see it moving forward again.

Have you heard of crosscompilers? You don't have to compile the code in the target platform, MinGW works perfectly fine on Linux for dumping PE blobs which you can run headless on Wine to provide a testsuite or something using the same game/mod data.

One machine to rule them all, Mac OS X is a huge nuisance if you want to do native tests.

But I guess that just having the assurance of not breaking the build for that target is enough.

The game is pretty unstable by itself right now.

[Stan`]

Why does people like Jenkins so much ? My Programmation teacher wants us to upload everything on github to be tested with jenkins. Result : I get 200 PMS on Facebook of my year's promotion, because Jenkins tests screws their working C# programs.

[Swyter]

Why does people like Jenkins so much ? My Programmation teacher wants us to upload everything on github to be tested with jenkins. Result : I get 200 PMS on Facebook of my year's promotion, because Jenkins tests screws their working C# programs.

I like you.

--

Also, there are plenty of free hosted CI solutions for open source teams, I know that this is trendy right now. But please, have a bit of personality and don't do it just because everyone and their cat is doing it!

First of all, you don't seem to have test cases, secondly, seems like the project is having a restructure with all the L10N fuss, Git and other things, so this looks secondary to me at most. Third, have a list of interesting services which may fit:

• Drone.io
• Travis-ci.com

Edited April 25, 2014 by Swyter

[sanderd17]

First of all, you don't seem to have test cases,

Ehem, we do have test cases, never executed the test binary? There are not alot of test cases though, I think mainly because there is no automatic testing. I don't want to waste time on running the tests every time i commit something.

secondly, seems like the project is having a restructure with all the L10N fuss, Git and other things, so this looks secondary to me at most. Third, have a list of interesting services which may fit:

• Drone.io
• Travis-ci.com

CI should be installed together with the move to git. L10n didn't change the project structure or tests at all.

[Stan`]

I'll be interested in MinGW aswell. The biggest reason why is that it would not be microsoft dependant. Moreover I have been working with it and eclipse in the past, and just having to launch msys, make all; make debug; make package; is really really convenient.

[Radagast.]

The game is pretty unstable by itself right now.

I can't see any unstability here unless I didn't break something.

If repository has been updated but not been built & you try to run e.g. Atlas, then it might be unstable because the resources might have changed (just try it when having broken Actors loaded and rotate the character wildly .. it will surely crash). But really that's an edge case of all edge case on this planet. (I wonder how I hit this one.)

Also we are in Alpha. And for Alpha this is a mighty piece of programming & art works.

I don't want to waste time on running the tests every time i commit something.

I share this one.

People even start employing people to write test cases only. At the end they wonder how it can be that they have tests but no program.

Also tests usually need their own tests (as they are human written and contain bugs themselves, we had this all). Do you see the infinite loop? Yet another bug in this system. Do we need a test to test our theories about if we need a test now?

.. really funny who invented this test hype.

Simply throw up a debugger and check the functionality yourself. If you don't find it with the debugger, then how shall your written test cases be correct?

Edited April 25, 2014 by Hephaestion

[Gallaecio]

I can only see benefits in having continuous integration, regardless of the solution that we choose.

[Radagast.]

true. can't be wrong. Personally my master reason for programming is to automate tedious or time-consuming work to the utmost. The secondary reason is to make awesome things happen without/minimal (material) cost.

[historic_bruno]

Have you heard of crosscompilers? You don't have to compile the code in the target platform, MinGW works perfectly fine on Linux for dumping PE blobs which you can run headless on Wine to provide a testsuite or something using the same game/mod data.

Then we would be testing MinGW cross-compilation, not MSVC native compilation, while MSVC is going to be our compiler and IDE of choice on Windows for the foreseeable future. We also wouldn't be testing the MSVC runtime libs, which can have peculiar issues (see r14983 for one of many).

Of course, if someone wants to put the effort into getting the game to build and run with MinGW, that's fine by me I don't think anyone will use it, and it's probably wasted effort, but in that case, we would certainly want to test it as well.

[Echelon9]

This is a really good initiative. Regardless of the ultimate continuous integration testing tool (Jenkins, Travis-CI, others) the approach is a sound one.

Thanks Yves for getting this up on a Jenkins-based system.

[Stan`]

Sorry for the necro, but what is the current state of this ?

Also IIRC we could set up Jenkins for a code convention check, which would allow us to clean some parts of the code.

[Yves]

Currently we have a Jenkins master server and one Windows slave. We just use it for the Windows autobuilds so far, but there are some more ideas how it could be used in the future. Some tasks like installing a Linux slave, building and running the tests is on my todo list, but not quite on top.

[Stan`]

Okay the tess would be really nice also looking for code conventions and stuff could give work like simple tickets