iOS: Running Unit Tests with Code Coverage in Jenkins/Hudson

Now that I have a nice unit test environment (see iOS: Setting Up Basic Unit Testing and iOS: Setting Up Advanced Unit Testing) I still need something to run them automatically. I choose Jenkins/Hudson as continuous integration server.

To build a project from Jenkins/Hudson it helps to already have an automatic build. That means you have an XCode project which does not depend on any absolute paths for additional libraries (as Google Toolbox for Mac, OCMock or OCHamcrest). That is the background of the CI Tips to include OCMock and OCHamcrest into the projects source control in my previous articles.

2011/2: see also iOS: Setting Up Advanced Unit Testing Take Two.

Already having an automatic clean build from XCode, it does not take much to run in from Jenkins/Hudson. Quite useful was this article. It shows how to use xcodebuild to run an XCode project without XCode. It also provides a small script to convert the objective-c test output to junit xml output that Jenkins/Hudson will understand.

Based on that blog my “Execute Shell” step looks like this:

xcodebuild -target "UnitTests" -sdk /Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.1.2.sdk/ -configuration "Debug" | /usr/local/bin/ocunit2junit.rb

And the “post build step” “Publish JUnit test result report” like this:

test-reports/*.xml

as “Test report XMLs”.

Great, after following the above article Jenkins/Hudson will watch my repository for changes, check out everything if a new commit hows up, build my “UnitTest” target (automatically running the tests, because of the shell script we added to the unit test project), convert the test output to a junit compatible format and display the test results and statistics in Jenkins/Hudson.

Getting the build running in Jenkins/Hudson was the easy part, making it collect code coverage adds new complications.

Code Coverage: To enable code coverage in the debug unit test target I set the following stuff in the XCode project (GetInfo on the unit test target):

In the “GCC 4.2 Preprocessing” section enable “Generate Test Coverage Files” and “Instrument Program Flow”. Add -lgcov to “Linking” section “other linker flags”. I got that info from here.

Complication No. 4: Unfortunately there are bugs in the XCode (currently 3.2.4 with gcc 4.2.1) that brake the code coverage. In my case link errors and 0% code coverage.

Solution to Complication No. 4: The advice given here helped me to get proper code coverage info. It provides a fixed gcov library (to fix the link errors) and tells you to clear the prefix header (to fix the 0% coverage issue).

Now running the test from Jenkins/Hudson will create the code coverage information but in a format that Jenkins/Hudson does not understand. By using another script called gcovr we can convert the gcov output to Cobertura xml. Cobertura is a Java code coverage tool that Jenkins/Hudson understands. I got this info from here.

Next step is to enable the cobertura “post build step” “Publish Cobertura Coverage Report” and set the “Cobertura xml report pattern” to

**/coverage.xml

in the projects Hudson configuration.

Complication No. 5: Of course this did not work for me out of the box. I had to patch the gcovr script a little bit. Orginally the script will change the current dir and run gcov with the filename of a gcov data file. That does not work. So I modified the script to run gcov with the full path of the data file and that works for me.

Solution to Complication No.5: her is my patch for gcovr:

--- Downloads/gcovr	2010-06-06 18:49:03.000000000 +0200
+++ bin/gcovr	2010-06-08 22:36:21.000000000 +0200
@@ -296,10 +297,13 @@
     (dir,base) = os.path.split(file)
     (name,ext) = os.path.splitext(base)
     prevdir = os.getcwd()
-    os.chdir(dir)
+    #os.chdir(dir)
     (head, tail) = os.path.split(name)
     cmd = gcov_cmd + \
-          " --branch-counts --branch-probabilities --preserve-paths " + tail
+          " --branch-counts --branch-probabilities --preserve-paths " + file \
+          + " -o " + dir
+          #+ tail
     output = subprocess.Popen( cmd.split(" "),
                                stdout=subprocess.PIPE ).communicate()[0]
     #output = subprocess.call(cmd)


CI Tip: add the gcovr script to your repository.

Summary: Finally, after going through all this issues I have an automatic build running in Jenkins/Hudson that runs the unit tests and collects code coverage information.

In the next article iOS: Setting Up Acceptance Testing I will write a little  bit about using Cucumber for acceptance testing with the help of Frank.

This is the 4th article of iOS: Setting Up a Test Environment is a Mess.

Advertisements

17 thoughts on “iOS: Running Unit Tests with Code Coverage in Jenkins/Hudson

  1. i have been trying to run the logical tests as per your blog suggestions but still it shows that build gets “Succeeded” but no test report file is created at location : “weorspace / test-report/*.xml”

    How i found that path is perfect : i myself placed a xml file named “*”
    and it showed that — file already exists that is old,Did test case run?

    Please help me out

    • Try running the given xcodebuild command from the console to check that it creates the test report:

      xcodebuild -target “UnitTests” -sdk /Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.1.2.sdk/ -configuration “Debug” | /usr/local/bin/ocunit2junit.rb

  2. lightforce's tech blog » Blog Archiv » Server Setup for Continuous Integration with iOS Projects

  3. Whops.
    The XML i pasted in didn’t show correctly….hope this works better.
    The Cobertura file contains:
    <sources>
    <source>
    .
    </source>
    </sources>

  4. Hi. Yep, i build with configuration Debug.
    The .gcno files are there, and if i point CoverStory to the directory containing the .gcno files, it displays the source-files fine, coloring the lines red and green. All look good.
    Hudson will display the coverage percentage for each file, but just not show me the source code.

    If I look in the XML file generated by gcovr for Cobertura, it contains the follwing:

    .

    I’m not sure if this i correct, and what the “.” is relative to.

  5. Thank you very much for your post. I now have coverage reports from my iOS project in hudson. However, I am not able to se the source-files in the report. Hudson says:

    Source code is unavailable. Some possible reasons are:
    – This is not the most recent build (to save on disk space, this plugin only keeps the most recent builds source code).
    – Cobertura found the source code but did not provide enough information to locate the source code.
    – Cobertura could not find the source code, so this plugin has no hope of finding it

    Have you managed to make that work?

    • Hi Soren,

      yep, it works for me and I don’t remember any special configure to make it work. Do you build with: -configuration “Debug”?

  6. Hi Chris,

    add another “Execute shell” build step in Hudson, after the xcodebuild step. This will look something like this:

    cd trunk
    gcovr -r . -x -e “.*Test.*” -e “.*GoogleToolbox.*” -e “.*Source Xternal.*” 1>coverage.xml

    The “-e” option excludes the files matching the given pattern. Make sure Hudson does find gcovr, either by putting it on the PATH or adding it to your source control.

    coverage.xml will be a “Cobertura Coverage Report” file. Now you can tell Hudson in “Post Build Actions” to run “Publish Cobertura Coverage Report”. My “Cobertura xml report pattern” is:

    **/coverage.xml

    If all goes well you should see the coverage info.

    That coverage stuff was the hardest to get running because of the Xcode issues and that gcovr did not work for me out of the box.

    Before configuring it in Hudson I made sure that it worked from the command line and produced a proper coverage.xml (simply checked that it contains some real values and not just zeros).

    I hope that clarifies the coverage setup a bit :-).

    Good Luck! :-)

  7. I would like to add coce coverage and that’s where your article gets fuzzy for me. You mention gcovr, but there is no mention on how to add this step to the job. Or am I missing something?

  8. I have been going down the continuous integration with unit test road for almost three weeks now and I am finally up and running with Hudson and I have just started getting into GTM. I am starting to see the light at the end of the tunnel for this thanks to your article.

    • Thanks Chris,

      it took me some time to merge all the loose and fragile pieces, so I’m happy to hear that it helped. :-)

  9. This is really cool and I’m something I’m currently interested in utilizing! Right now we’re using an in house solution for automated building as well as SICCI. Thanks for making this tutorial!

  10. iOS: Setting Up Advanced Unit Testing « Software Noise

  11. iOS: Setting Up a Test Environment is a Mess « Software Noise

  12. iOS: Running Cucumber/Frank with Code Coverage in Hudson « Software Noise

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s