iOS: Running Cucumber/Frank with Code Coverage in Jenkins/Hudson

In the previous article iOS: Setting Up Acceptance Testing I introduced Frank. The next and last step will be to run Cucumber and Frank headless from Jenkins/Hudson.

Frank is still new, and it uses apple script to remote control XCode to run the “frankified” app in the simulator. That’s fine for running the cucumber scenarios manually but it is not so good for running them from Jenkins/Hudson.

To run the cucumber tests from Jenkins/Hudson I need it to run headless, i.e. it should not open any window while running the tests. I don’t want Jenkins/Hudson to interfere with whatever I’m doing right know. It should run invisible in the background.

Luckily it is possible to run an iOS app headless simply by running the app directly like this:

<path to app>/MyApp.app/MyApp -RegisterForSystemEvents

By default I would use a cucumber step like the following to run my app from cucumber:

Given /^the app has just started$/ do
  launch_app_in_simulator
end

with launch_app_in_simulator beeing the frank function that uses apple script  to run the app by controlling XCode.

I have changed that to:

Given /^the app has just started$/ do
  launch_app_headless
end

with launch_app_headless looking like this:

def launch_app_headless
  @apppid = fork do
    exec(APP, "-RegisterForSystemEvents")
  end

  wait_for_frank_to_come_up
end

APP is a string variable that contains the path to the frankified app. It is currently set hardcoded in my cucumber Before block:

require 'fileutils'

##
## adjust app, sdk dir and app dir
##
APP    = "build/Debug-iphonesimulator/Readme Frankified.app/Readme Frankified"
APPDIR = "build/Readme.build/Debug-iphonesimulator/Readme Frankified.build"
SDKDIR = "/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.2.sdk"

USERDIR = "#{APPDIR}/iPhone Simulator User Dir"
PREFDIR = "#{USERDIR}/Library/Preferences"

ACCESIBILITY_PLIST   = "com.apple.Accessibility.plist"
ACCESIBILITY_CONTENT = <<PLIST
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
 <key>ApplicationAccessibilityEnabled</key>
 <true/>
</dict>
</plist>
PLIST

Before do
  # check that pwd contains the "build" dir as we are creating
  # items relative to it.
  Dir["build"].length.should == 1

  # make sure we do start with a clean environment
  FileUtils.remove_dir("#{USERDIR}",true)

  pwd     = "#{Dir.pwd}"
  prefdir = "#{pwd}/#{PREFDIR}"
  FileUtils.mkdir_p prefdir

  File.open("#{PREFDIR}/#{ACCESIBILITY_PLIST}", 'w') do |f|
    f <<ACCESIBILITY_CONTENT
  end

  ENV['SDKROOT']               = "#{SDKDIR}"
  ENV['DYLD_ROOT_PATH']        = "#{SDKDIR}"
  ENV['IPHONE_SIMULATOR_ROOT'] = "#{SDKDIR}"
  ENV['TEMP_FILES_DIR']        = "#{pwd}/#{APPDIR}"
  ENV['CFFIXED_USER_HOME']     = "#{pwd}/#{USERDIR}"
end

The accessibility stuff is required to create a configuration file for the app that enables accessibility. Without it Frank will not find the accessibilityLabels which are used to identify the ui elements.

Now I’m able to run the cucumber test for my app headless in Jenkins/Hudson. I created a second build configuration for the acceptance tests. It uses nearly the same setup as for unit testing. I just have added an additional “Execute Shell” step after building the frankified app that runs cucumber:

cd trunk
cucumber -f junit --o ../cucumber --tags ~@ignore features.frank

Using the -f junit parameter cucmber will create an junit xml file so Jenkins/Hudson can display the test results in the same ways as the unit tests. features.frank is the folder were I keep my cucumber scenarios.

Code coverage is set up the same way as for the unit tests (see iOS: Running Unit Tests with Code Coverage in Jenkins/Hudson). Now here is also the last complication. Frank currently doesn’t shut down the app and the result is that the app doesn’t write the coverage information.

To fix this I simply added a super simple exit command (only calling exit(0)) to Franks embedded http server and a ruby function to call it. I have forked the Frank git repository here http://github.com/hauner/Frank.

My cucumber After block now shuts down the app so it does write the code coverage info:

After do
  frankly_exit
end

with franky_exit beeing:

def frankly_exit
  get_to_uispec_server('exit')
  # calling exit in the app will not return any response
  # so we simply catch the error caused by exiting.
  rescue EOFError
end

Now the app will shut down with a clean exit, write the code coverage information and the code coverage step in Jenkins/Hudson will properly display the code coverage for my acceptance tests.

Done, finally :-)

Summary: I have reached my goal to have a test environment to create unit tests (using Google Toolbox for MacOCMock & OCHamcrest) and acceptance tests (using Cucumber & Frank) of an iOS  app. Both test categories are running in Jenkins/Hudson as two separate builds and  both builds provide code coverage information. It took a little bit of work, but it is running now.

That is all I have to say regarding testing an iOS app for now! :-)

This is the 6th article of iOS: Setting Up a Test Environment is a Mess.

Advertisements

iOS: Setting Up Acceptance Testing

After setting up Jenkins/Hudson to build and run my unit tests and collecting code coverage in the last article I will give some guidance for setting up acceptance testing with Cucumber based on a new test framework called Frank.

This article will not make much sense if you do not know cucumber. To learn about cucumber check out the cucumber home page here or take a look at the Secret Ninja Cucumber Scrolls.

Frank embeds a http server into the test version of our app which will be run when the app gets started. Via this internal http server the cucumber part of frank will send commands to the app to remote control it (taps, button taps, filling texts and so on) or to look for ui elements I want to see on specific views.

The first step is create a “frankified” version of the app for testing by modifiying a copy of the iOS Apps target configuration in XCode:

  • main gets replaced by a version that start the http server before it creates the usual UIApplicationMain
  • it links to the Frank library
  • and it includes a Frank resource bundle used by the http server

That is described in more detail in the Frank documentation so I will not repeat it here. After having a “frankified” app we can start writing cucumber scenarios.

Frank already comes with a number of usable cucumber steps, making it easier to get started and are also useful as a template for private steps.

Here is a list of the built-in steps:

Then /^I wait to see "([^\"]*)"$/
Then /^I wait to not see "([^\"]*)"$/
Then /^I wait to see a navigation bar titled "([^\"]*)"$/
Then /^I wait to not see a navigation bar titled "([^\"]*)"$/
Then /^I should see a "([^\"]*)" button$/
Then /^I should see "([^\"]*)"$/
Then /^I should not see "([^\"]*)"$/
Then /^I should see the following:/
Then /^I should see a navigation bar titled "([^\"]*)"$/

When /^I type "([^\"]*)" into the "([^\"]*)" text field$/
When /^I fill in "([^\"]*)" with "([^\"]*)"$/
When /^I fill in text fields as follows:$/

And /^I rotate to the "([^\"]*)"$/
When /^I touch "([^\"]*)"$/
Then /I touch the following:/
When /^I touch the button marked "([^\"]*)"$/
When /^I touch the "([^\"]*)" action sheet button$/
When /^I touch the (\d*)(?:st|nd|rd|th)? action sheet button$/

When /^I flip switch "([^\"]*)"$/
Then /^switch "([^\"]*)" should be on$/
Then /^switch "([^\"]*)" should be off$/

Then /^a pop\-over menu is displayed with the following:$/
Then /^I navigate back$/

Having quite a number of example steps it is not very difficult to create additional steps. All the built-in steps are just a few lines and easy to understand.

For example I created a step that looks like this:

Then /^I should see text fields for:$/ do |table|
  table.raw.each do |v|
    check_element_exists(%Q|textField accessibilityLabel:"#{v}"|);
  end
end

The interesting part is the check_element_exists. It is one of the basic calls used to implement the above steps. It takes care of all the communication with the http server in the “frankified” test app.

By default Frank uses the “Accessibility Label” to locate ui elements in the app using UISpec. The “Accessibility Label”s original use is to give a hint what a control is for to people with visual impairments. Frank uses it to mark ui elements with a “label” that we can use in our cucumber steps to refer to that ui element.

For example the cucumber scenario using my custom step looks like this:

Scenario:
  ask user to login at first startup

  Given the app runs for the first time
  Then I should be on the "Login" screen
  And I should see text fields for:
    | Login    |
    | Password |

The “And” step is using my custom step and it (i.e. UISpec triggered by a call to the embedded http server) will look for two text fields that are marked with the accessibility labels “Login” and “Password”.

I set the labels from Interface Builder by opening the Inspector, switching to the View Identity tab (it is the last tab) and setting the “Label” under the “Accessibility” options.

Note:
To make this work in the simulator I first had to enable “Accessibility” feature in MacOSX, i.e. enable “Enable access for assistive devices” in the “Universal Access” system preferences.

Really cool is, that the parameter to check_element_exists is a UISpec script expression.

This make it possible to check for other properties than the accessibilityLabel of an ui element. I can easily check that the “tag” property (it is on the View Attributes tab in the Interface Builder inspector) of an ui element has a specific value using the following expression:

check_element_exists("view tag:1")

Check out UISpec/UIScript to learn more about this.

To run the cucumber scenarios for my app I simply run the cucumber command from bash and watch frank running trough my acceptance tests.

Currently you will have to open XCode as frank uses apple script to run the app in the simulator from XCode. That is not the perfect way to run the cucumber test automatically from Jenkins/Hudson.

Summary: I have reached my next goal, running Cucumber acceptance tests against my iOS app using Frank. Apart from enabling the accessibility stuff, there were no complications to get it running.

In the next article iOS: Running Cucumber/Frank with Code Coverage in Jenkins/Hudson I will describe how I managed to run my “frankified” app headless from Jenkins/Hudson by using just a view lines of additional ruby code.

This is the 5th article of iOS: Setting Up a Test Environment is a Mess.

iOS: Setting Up a Test Environment is a Mess

Really. I have tried. I have done it successfully but it was a bit of work.

There are a number of complications to set up a decent testing environment for iOS. I will describe which tools I choose and what I had to do to get everything running. I will not describe everything in detail but hopefully it will still give you a starting point.

All in all, I think that this should be (a lot) easier, even in this part of the C universe.

So what do I consider a decent testing environment?

I think at first we need support for unit tests and acceptance tests (in the agile meaning). Unit tests to get the low level stuff right (we can also use it for some integration testing) and acceptance tests to do some end-to-end testing.

Running all the tests manually is not a good idea anymore so I will run them on a continuous integration server, which will also take care of collecting code coverage.

My current collection of tools include XCode (xcodebuild, gcc, gcov) with iOS SDK, Google Toolbox for MacOCMock, OCHamcrest, Cucumber, Frank (includes UISpec) and Jenkins/Hudson (with gcovr for xml-ifying gcov).

All tools are free and all I had to do was to make them play together. A number of blogs provided helpful information. I will try to list them for reference.

There is a lot of stuff and to make it an easier read I split in into multiple articles. Here is the complete list:

February 2011: Here is new article regarding setup of OCMock & OCHamcrest:

Ok, let’s start with iOS: Setting Up Basic Unit Testing.