0ad and unit testing

[stilz]

Hello there,

as a professional C++ programmer I do a lot of testing in numerous projects. It was kind of surprise when I discovered that there are no unit tests in 0ad.

I'd like to ask what led to such situation? Is it only matter of lack of time?

Regardless the cause, I'd like to start a discussion on this subject.

What UT frameworks are you experienced in? Which one of them you think would fit to 0ad?

There is also other important thing we should concern - perhaps JavaScript should also be tested. I'm rather seasonal JS programmer, so your opinion would be highly appreciated.

[sanderd17]

I'd check the source code again, there are unit tests for both c++ and js side. They're under simulation/tests in both cases. Only a few components are tested though, and that's because lack of time. But the framework is present.

The tests can be run via the test or test.exe executables.

[stilz]

Yes, there are 64 tests as I can see (round number ;-) - not too much.

[Ykkrosh]

$ ./test Running 286 tests...[...]...OK!

That looks like more than 64 to me

It's still true that's not really a lot, though, and more would be nice. We're using CxxTest which I think works fine, with some custom code to run a few mostly useless tests on simulation component scripts.

I think the main reason for a lack of unit tests is that (as far as I'm aware) we don't have many bugs that unit tests would find - the usual bugs nowadays seem to be undesired gameplay behaviour, or problems with the interactions between gameplay components, and I don't think it's feasible to find them through automated testing. Unit tests aren't an end goal in themselves - the goal is to find existing bugs and future regressions - so there's not much motivation to write tests for code that already seems to work fine.

Another problem is that a lot of the engine was written before we had any interest in unit testing at all, and it was written in a way that makes unit testing hard (e.g. modules that have global state and lots of dependencies on other modules, so it's impossible to test part of it in isolation). Rearchitecting that code to be testable would take a lot of effort and probably introduce plenty of bugs, so it doesn't really seem worthwhile. So we have some unit tests for newer things like simulation2 that were designed with that in mind, but none for older things like the GUI engine since it's all tightly-coupled code.

[01d55]

Another problem is that a lot of the engine was written before we had any interest in unit testing at all, and it was written in a way that makes unit testing hard (e.g. modules that have global state and lots of dependencies on other modules, so it's impossible to test part of it in isolation). Rearchitecting that code to be testable would take a lot of effort and probably introduce plenty of bugs, so it doesn't really seem worthwhile. So we have some unit tests for newer things like simulation2 that were designed with that in mind, but none for older things like the GUI engine since it's all tightly-coupled code.

C & C++ code can be much less tightly coupled than it appearances suggest, thanks to the compilation model. Any source file that can be compiled to an object file is necessarily isolatable as a testable unit.

For example, suppose class B depends on class A, and appears to be tightly coupled (e.g. B composites A by value, or A's methods aren't virtual). However, A and B have been written in the normal way: A.hxx declares class A, A.cxx defines class A's methods, and likewise for B.hxx and B.cxx. It is therefore possible to write a file, A_mock.cxx, which also defines class A's methods, and link A_mock.o instead of A.o in the test executable.

This method of unit testing by compilation unit uses seperate executables for each unit tested. Every symbol referenced but not defined in the compilation unit under test is defined by the test code and any such symbol the test writer overlooks stops the test from being linked.

This doesn't save you from tight linkages when functions are defined in a header, but moving definitions out of headers is much less risky than re-architecting code so that it is isolatable through polymorphism.