iOS: Running Unit Tests with Code Coverage in Jenkins/Hudson

Now that I have a nice unit test environment (see iOS: Setting Up Basic Unit Testing and iOS: Setting Up Advanced Unit Testing) I still need something to run them automatically. I choose Jenkins/Hudson as continuous integration server.

To build a project from Jenkins/Hudson it helps to already have an automatic build. That means you have an XCode project which does not depend on any absolute paths for additional libraries (as Google Toolbox for Mac, OCMock or OCHamcrest). That is the background of the CI Tips to include OCMock and OCHamcrest into the projects source control in my previous articles.

2011/2: see also iOS: Setting Up Advanced Unit Testing Take Two.

Already having an automatic clean build from XCode, it does not take much to run in from Jenkins/Hudson. Quite useful was this article. It shows how to use xcodebuild to run an XCode project without XCode. It also provides a small script to convert the objective-c test output to junit xml output that Jenkins/Hudson will understand.

Based on that blog my “Execute Shell” step looks like this:

xcodebuild -target "UnitTests" -sdk /Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.1.2.sdk/ -configuration "Debug" | /usr/local/bin/ocunit2junit.rb

And the “post build step” “Publish JUnit test result report” like this:

test-reports/*.xml

as “Test report XMLs”.

Great, after following the above article Jenkins/Hudson will watch my repository for changes, check out everything if a new commit hows up, build my “UnitTest” target (automatically running the tests, because of the shell script we added to the unit test project), convert the test output to a junit compatible format and display the test results and statistics in Jenkins/Hudson.

Getting the build running in Jenkins/Hudson was the easy part, making it collect code coverage adds new complications.

Code Coverage: To enable code coverage in the debug unit test target I set the following stuff in the XCode project (GetInfo on the unit test target):

In the “GCC 4.2 Preprocessing” section enable “Generate Test Coverage Files” and “Instrument Program Flow”. Add -lgcov to “Linking” section “other linker flags”. I got that info from here.

Complication No. 4: Unfortunately there are bugs in the XCode (currently 3.2.4 with gcc 4.2.1) that brake the code coverage. In my case link errors and 0% code coverage.

Solution to Complication No. 4: The advice given here helped me to get proper code coverage info. It provides a fixed gcov library (to fix the link errors) and tells you to clear the prefix header (to fix the 0% coverage issue).

Now running the test from Jenkins/Hudson will create the code coverage information but in a format that Jenkins/Hudson does not understand. By using another script called gcovr we can convert the gcov output to Cobertura xml. Cobertura is a Java code coverage tool that Jenkins/Hudson understands. I got this info from here.

Next step is to enable the cobertura “post build step” “Publish Cobertura Coverage Report” and set the “Cobertura xml report pattern” to

**/coverage.xml

in the projects Hudson configuration.

Complication No. 5: Of course this did not work for me out of the box. I had to patch the gcovr script a little bit. Orginally the script will change the current dir and run gcov with the filename of a gcov data file. That does not work. So I modified the script to run gcov with the full path of the data file and that works for me.

Solution to Complication No.5: her is my patch for gcovr:

--- Downloads/gcovr	2010-06-06 18:49:03.000000000 +0200
+++ bin/gcovr	2010-06-08 22:36:21.000000000 +0200
@@ -296,10 +297,13 @@
     (dir,base) = os.path.split(file)
     (name,ext) = os.path.splitext(base)
     prevdir = os.getcwd()
-    os.chdir(dir)
+    #os.chdir(dir)
     (head, tail) = os.path.split(name)
     cmd = gcov_cmd + \
-          " --branch-counts --branch-probabilities --preserve-paths " + tail
+          " --branch-counts --branch-probabilities --preserve-paths " + file \
+          + " -o " + dir
+          #+ tail
     output = subprocess.Popen( cmd.split(" "),
                                stdout=subprocess.PIPE ).communicate()[0]
     #output = subprocess.call(cmd)

CI Tip: add the gcovr script to your repository.

Summary: Finally, after going through all this issues I have an automatic build running in Jenkins/Hudson that runs the unit tests and collects code coverage information.

In the next article iOS: Setting Up Acceptance Testing I will write a little  bit about using Cucumber for acceptance testing with the help of Frank.

This is the 4th article of iOS: Setting Up a Test Environment is a Mess.