iOS: Running Cucumber/Frank with Code Coverage in Jenkins/Hudson

In the previous article iOS: Setting Up Acceptance Testing I introduced Frank. The next and last step will be to run Cucumber and Frank headless from Jenkins/Hudson.

Frank is still new, and it uses apple script to remote control XCode to run the “frankified” app in the simulator. That’s fine for running the cucumber scenarios manually but it is not so good for running them from Jenkins/Hudson.

To run the cucumber tests from Jenkins/Hudson I need it to run headless, i.e. it should not open any window while running the tests. I don’t want Jenkins/Hudson to interfere with whatever I’m doing right know. It should run invisible in the background.

Luckily it is possible to run an iOS app headless simply by running the app directly like this:

<path to app>/ -RegisterForSystemEvents

By default I would use a cucumber step like the following to run my app from cucumber:

Given /^the app has just started$/ do

with launch_app_in_simulator beeing the frank function that uses apple script  to run the app by controlling XCode.

I have changed that to:

Given /^the app has just started$/ do

with launch_app_headless looking like this:

def launch_app_headless
  @apppid = fork do
    exec(APP, "-RegisterForSystemEvents")


APP is a string variable that contains the path to the frankified app. It is currently set hardcoded in my cucumber Before block:

require 'fileutils'

## adjust app, sdk dir and app dir
APP    = "build/Debug-iphonesimulator/Readme Frankified"
APPDIR = "build/"
SDKDIR = "/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.2.sdk"

USERDIR = "#{APPDIR}/iPhone Simulator User Dir"
PREFDIR = "#{USERDIR}/Library/Preferences"

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "">
<plist version="1.0">

Before do
  # check that pwd contains the "build" dir as we are creating
  # items relative to it.
  Dir["build"].length.should == 1

  # make sure we do start with a clean environment

  pwd     = "#{Dir.pwd}"
  prefdir = "#{pwd}/#{PREFDIR}"
  FileUtils.mkdir_p prefdir"#{PREFDIR}/#{ACCESIBILITY_PLIST}", 'w') do |f|

  ENV['SDKROOT']               = "#{SDKDIR}"
  ENV['DYLD_ROOT_PATH']        = "#{SDKDIR}"
  ENV['TEMP_FILES_DIR']        = "#{pwd}/#{APPDIR}"
  ENV['CFFIXED_USER_HOME']     = "#{pwd}/#{USERDIR}"

The accessibility stuff is required to create a configuration file for the app that enables accessibility. Without it Frank will not find the accessibilityLabels which are used to identify the ui elements.

Now I’m able to run the cucumber test for my app headless in Jenkins/Hudson. I created a second build configuration for the acceptance tests. It uses nearly the same setup as for unit testing. I just have added an additional “Execute Shell” step after building the frankified app that runs cucumber:

cd trunk
cucumber -f junit --o ../cucumber --tags ~@ignore features.frank

Using the -f junit parameter cucmber will create an junit xml file so Jenkins/Hudson can display the test results in the same ways as the unit tests. features.frank is the folder were I keep my cucumber scenarios.

Code coverage is set up the same way as for the unit tests (see iOS: Running Unit Tests with Code Coverage in Jenkins/Hudson). Now here is also the last complication. Frank currently doesn’t shut down the app and the result is that the app doesn’t write the coverage information.

To fix this I simply added a super simple exit command (only calling exit(0)) to Franks embedded http server and a ruby function to call it. I have forked the Frank git repository here

My cucumber After block now shuts down the app so it does write the code coverage info:

After do

with franky_exit beeing:

def frankly_exit
  # calling exit in the app will not return any response
  # so we simply catch the error caused by exiting.
  rescue EOFError

Now the app will shut down with a clean exit, write the code coverage information and the code coverage step in Jenkins/Hudson will properly display the code coverage for my acceptance tests.

Done, finally :-)

Summary: I have reached my goal to have a test environment to create unit tests (using Google Toolbox for MacOCMock & OCHamcrest) and acceptance tests (using Cucumber & Frank) of an iOS  app. Both test categories are running in Jenkins/Hudson as two separate builds and  both builds provide code coverage information. It took a little bit of work, but it is running now.

That is all I have to say regarding testing an iOS app for now! :-)

This is the 6th article of iOS: Setting Up a Test Environment is a Mess.


iOS: Setting Up Acceptance Testing

After setting up Jenkins/Hudson to build and run my unit tests and collecting code coverage in the last article I will give some guidance for setting up acceptance testing with Cucumber based on a new test framework called Frank.

This article will not make much sense if you do not know cucumber. To learn about cucumber check out the cucumber home page here or take a look at the Secret Ninja Cucumber Scrolls.

Frank embeds a http server into the test version of our app which will be run when the app gets started. Via this internal http server the cucumber part of frank will send commands to the app to remote control it (taps, button taps, filling texts and so on) or to look for ui elements I want to see on specific views.

The first step is create a “frankified” version of the app for testing by modifiying a copy of the iOS Apps target configuration in XCode:

  • main gets replaced by a version that start the http server before it creates the usual UIApplicationMain
  • it links to the Frank library
  • and it includes a Frank resource bundle used by the http server

That is described in more detail in the Frank documentation so I will not repeat it here. After having a “frankified” app we can start writing cucumber scenarios.

Frank already comes with a number of usable cucumber steps, making it easier to get started and are also useful as a template for private steps.

Here is a list of the built-in steps:

Then /^I wait to see "([^\"]*)"$/
Then /^I wait to not see "([^\"]*)"$/
Then /^I wait to see a navigation bar titled "([^\"]*)"$/
Then /^I wait to not see a navigation bar titled "([^\"]*)"$/
Then /^I should see a "([^\"]*)" button$/
Then /^I should see "([^\"]*)"$/
Then /^I should not see "([^\"]*)"$/
Then /^I should see the following:/
Then /^I should see a navigation bar titled "([^\"]*)"$/

When /^I type "([^\"]*)" into the "([^\"]*)" text field$/
When /^I fill in "([^\"]*)" with "([^\"]*)"$/
When /^I fill in text fields as follows:$/

And /^I rotate to the "([^\"]*)"$/
When /^I touch "([^\"]*)"$/
Then /I touch the following:/
When /^I touch the button marked "([^\"]*)"$/
When /^I touch the "([^\"]*)" action sheet button$/
When /^I touch the (\d*)(?:st|nd|rd|th)? action sheet button$/

When /^I flip switch "([^\"]*)"$/
Then /^switch "([^\"]*)" should be on$/
Then /^switch "([^\"]*)" should be off$/

Then /^a pop\-over menu is displayed with the following:$/
Then /^I navigate back$/

Having quite a number of example steps it is not very difficult to create additional steps. All the built-in steps are just a few lines and easy to understand.

For example I created a step that looks like this:

Then /^I should see text fields for:$/ do |table|
  table.raw.each do |v|
    check_element_exists(%Q|textField accessibilityLabel:"#{v}"|);

The interesting part is the check_element_exists. It is one of the basic calls used to implement the above steps. It takes care of all the communication with the http server in the “frankified” test app.

By default Frank uses the “Accessibility Label” to locate ui elements in the app using UISpec. The “Accessibility Label”s original use is to give a hint what a control is for to people with visual impairments. Frank uses it to mark ui elements with a “label” that we can use in our cucumber steps to refer to that ui element.

For example the cucumber scenario using my custom step looks like this:

  ask user to login at first startup

  Given the app runs for the first time
  Then I should be on the "Login" screen
  And I should see text fields for:
    | Login    |
    | Password |

The “And” step is using my custom step and it (i.e. UISpec triggered by a call to the embedded http server) will look for two text fields that are marked with the accessibility labels “Login” and “Password”.

I set the labels from Interface Builder by opening the Inspector, switching to the View Identity tab (it is the last tab) and setting the “Label” under the “Accessibility” options.

To make this work in the simulator I first had to enable “Accessibility” feature in MacOSX, i.e. enable “Enable access for assistive devices” in the “Universal Access” system preferences.

Really cool is, that the parameter to check_element_exists is a UISpec script expression.

This make it possible to check for other properties than the accessibilityLabel of an ui element. I can easily check that the “tag” property (it is on the View Attributes tab in the Interface Builder inspector) of an ui element has a specific value using the following expression:

check_element_exists("view tag:1")

Check out UISpec/UIScript to learn more about this.

To run the cucumber scenarios for my app I simply run the cucumber command from bash and watch frank running trough my acceptance tests.

Currently you will have to open XCode as frank uses apple script to run the app in the simulator from XCode. That is not the perfect way to run the cucumber test automatically from Jenkins/Hudson.

Summary: I have reached my next goal, running Cucumber acceptance tests against my iOS app using Frank. Apart from enabling the accessibility stuff, there were no complications to get it running.

In the next article iOS: Running Cucumber/Frank with Code Coverage in Jenkins/Hudson I will describe how I managed to run my “frankified” app headless from Jenkins/Hudson by using just a view lines of additional ruby code.

This is the 5th article of iOS: Setting Up a Test Environment is a Mess.

iOS: Running Unit Tests with Code Coverage in Jenkins/Hudson

Now that I have a nice unit test environment (see iOS: Setting Up Basic Unit Testing and iOS: Setting Up Advanced Unit Testing) I still need something to run them automatically. I choose Jenkins/Hudson as continuous integration server.

To build a project from Jenkins/Hudson it helps to already have an automatic build. That means you have an XCode project which does not depend on any absolute paths for additional libraries (as Google Toolbox for Mac, OCMock or OCHamcrest). That is the background of the CI Tips to include OCMock and OCHamcrest into the projects source control in my previous articles.

2011/2: see also iOS: Setting Up Advanced Unit Testing Take Two.

Already having an automatic clean build from XCode, it does not take much to run in from Jenkins/Hudson. Quite useful was this article. It shows how to use xcodebuild to run an XCode project without XCode. It also provides a small script to convert the objective-c test output to junit xml output that Jenkins/Hudson will understand.

Based on that blog my “Execute Shell” step looks like this:

xcodebuild -target "UnitTests" -sdk /Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.1.2.sdk/ -configuration "Debug" | /usr/local/bin/ocunit2junit.rb

And the “post build step” “Publish JUnit test result report” like this:


as “Test report XMLs”.

Great, after following the above article Jenkins/Hudson will watch my repository for changes, check out everything if a new commit hows up, build my “UnitTest” target (automatically running the tests, because of the shell script we added to the unit test project), convert the test output to a junit compatible format and display the test results and statistics in Jenkins/Hudson.

Getting the build running in Jenkins/Hudson was the easy part, making it collect code coverage adds new complications.

Code Coverage: To enable code coverage in the debug unit test target I set the following stuff in the XCode project (GetInfo on the unit test target):

In the “GCC 4.2 Preprocessing” section enable “Generate Test Coverage Files” and “Instrument Program Flow”. Add -lgcov to “Linking” section “other linker flags”. I got that info from here.

Complication No. 4: Unfortunately there are bugs in the XCode (currently 3.2.4 with gcc 4.2.1) that brake the code coverage. In my case link errors and 0% code coverage.

Solution to Complication No. 4: The advice given here helped me to get proper code coverage info. It provides a fixed gcov library (to fix the link errors) and tells you to clear the prefix header (to fix the 0% coverage issue).

Now running the test from Jenkins/Hudson will create the code coverage information but in a format that Jenkins/Hudson does not understand. By using another script called gcovr we can convert the gcov output to Cobertura xml. Cobertura is a Java code coverage tool that Jenkins/Hudson understands. I got this info from here.

Next step is to enable the cobertura “post build step” “Publish Cobertura Coverage Report” and set the “Cobertura xml report pattern” to


in the projects Hudson configuration.

Complication No. 5: Of course this did not work for me out of the box. I had to patch the gcovr script a little bit. Orginally the script will change the current dir and run gcov with the filename of a gcov data file. That does not work. So I modified the script to run gcov with the full path of the data file and that works for me.

Solution to Complication No.5: her is my patch for gcovr:

--- Downloads/gcovr	2010-06-06 18:49:03.000000000 +0200
+++ bin/gcovr	2010-06-08 22:36:21.000000000 +0200
@@ -296,10 +297,13 @@
     (dir,base) = os.path.split(file)
     (name,ext) = os.path.splitext(base)
     prevdir = os.getcwd()
-    os.chdir(dir)
+    #os.chdir(dir)
     (head, tail) = os.path.split(name)
     cmd = gcov_cmd + \
-          " --branch-counts --branch-probabilities --preserve-paths " + tail
+          " --branch-counts --branch-probabilities --preserve-paths " + file \
+          + " -o " + dir
+          #+ tail
     output = subprocess.Popen( cmd.split(" "),
                                stdout=subprocess.PIPE ).communicate()[0]
     #output =

CI Tip: add the gcovr script to your repository.

Summary: Finally, after going through all this issues I have an automatic build running in Jenkins/Hudson that runs the unit tests and collects code coverage information.

In the next article iOS: Setting Up Acceptance Testing I will write a little  bit about using Cucumber for acceptance testing with the help of Frank.

This is the 4th article of iOS: Setting Up a Test Environment is a Mess.

iOS: Setting Up a Test Environment is a Mess

Really. I have tried. I have done it successfully but it was a bit of work.

There are a number of complications to set up a decent testing environment for iOS. I will describe which tools I choose and what I had to do to get everything running. I will not describe everything in detail but hopefully it will still give you a starting point.

All in all, I think that this should be (a lot) easier, even in this part of the C universe.

So what do I consider a decent testing environment?

I think at first we need support for unit tests and acceptance tests (in the agile meaning). Unit tests to get the low level stuff right (we can also use it for some integration testing) and acceptance tests to do some end-to-end testing.

Running all the tests manually is not a good idea anymore so I will run them on a continuous integration server, which will also take care of collecting code coverage.

My current collection of tools include XCode (xcodebuild, gcc, gcov) with iOS SDK, Google Toolbox for MacOCMock, OCHamcrest, Cucumber, Frank (includes UISpec) and Jenkins/Hudson (with gcovr for xml-ifying gcov).

All tools are free and all I had to do was to make them play together. A number of blogs provided helpful information. I will try to list them for reference.

There is a lot of stuff and to make it an easier read I split in into multiple articles. Here is the complete list:

February 2011: Here is new article regarding setup of OCMock & OCHamcrest:

Ok, let’s start with iOS: Setting Up Basic Unit Testing.