In the previous article iOS: Setting Up Acceptance Testing I introduced Frank. The next and last step will be to run Cucumber and Frank headless from Jenkins/Hudson.
Frank is still new, and it uses apple script to remote control XCode to run the “frankified” app in the simulator. That’s fine for running the cucumber scenarios manually but it is not so good for running them from Jenkins/Hudson.
To run the cucumber tests from Jenkins/Hudson I need it to run headless, i.e. it should not open any window while running the tests. I don’t want Jenkins/Hudson to interfere with whatever I’m doing right know. It should run invisible in the background.
Luckily it is possible to run an iOS app headless simply by running the app directly like this:
<path to app>/MyApp.app/MyApp -RegisterForSystemEvents
By default I would use a cucumber step like the following to run my app from cucumber:
Given /^the app has just started$/ do launch_app_in_simulator end
with launch_app_in_simulator beeing the frank function that uses apple script to run the app by controlling XCode.
I have changed that to:
Given /^the app has just started$/ do launch_app_headless end
with launch_app_headless looking like this:
def launch_app_headless
@apppid = fork do
exec(APP, "-RegisterForSystemEvents")
end
wait_for_frank_to_come_up
end
APP is a string variable that contains the path to the frankified app. It is currently set hardcoded in my cucumber Before block:
require 'fileutils'
##
## adjust app, sdk dir and app dir
##
APP = "build/Debug-iphonesimulator/Readme Frankified.app/Readme Frankified"
APPDIR = "build/Readme.build/Debug-iphonesimulator/Readme Frankified.build"
SDKDIR = "/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.2.sdk"
USERDIR = "#{APPDIR}/iPhone Simulator User Dir"
PREFDIR = "#{USERDIR}/Library/Preferences"
ACCESIBILITY_PLIST = "com.apple.Accessibility.plist"
ACCESIBILITY_CONTENT = <<PLIST
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>ApplicationAccessibilityEnabled</key>
<true/>
</dict>
</plist>
PLIST
Before do
# check that pwd contains the "build" dir as we are creating
# items relative to it.
Dir["build"].length.should == 1
# make sure we do start with a clean environment
FileUtils.remove_dir("#{USERDIR}",true)
pwd = "#{Dir.pwd}"
prefdir = "#{pwd}/#{PREFDIR}"
FileUtils.mkdir_p prefdir
File.open("#{PREFDIR}/#{ACCESIBILITY_PLIST}", 'w') do |f|
f <<ACCESIBILITY_CONTENT
end
ENV['SDKROOT'] = "#{SDKDIR}"
ENV['DYLD_ROOT_PATH'] = "#{SDKDIR}"
ENV['IPHONE_SIMULATOR_ROOT'] = "#{SDKDIR}"
ENV['TEMP_FILES_DIR'] = "#{pwd}/#{APPDIR}"
ENV['CFFIXED_USER_HOME'] = "#{pwd}/#{USERDIR}"
end
The accessibility stuff is required to create a configuration file for the app that enables accessibility. Without it Frank will not find the accessibilityLabels which are used to identify the ui elements.
Now I’m able to run the cucumber test for my app headless in Jenkins/Hudson. I created a second build configuration for the acceptance tests. It uses nearly the same setup as for unit testing. I just have added an additional “Execute Shell” step after building the frankified app that runs cucumber:
cd trunk cucumber -f junit --o ../cucumber --tags ~@ignore features.frank
Using the -f junit parameter cucmber will create an junit xml file so Jenkins/Hudson can display the test results in the same ways as the unit tests. features.frank is the folder were I keep my cucumber scenarios.
Code coverage is set up the same way as for the unit tests (see iOS: Running Unit Tests with Code Coverage in Jenkins/Hudson). Now here is also the last complication. Frank currently doesn’t shut down the app and the result is that the app doesn’t write the coverage information.
To fix this I simply added a super simple exit command (only calling exit(0)) to Franks embedded http server and a ruby function to call it. I have forked the Frank git repository here http://github.com/hauner/Frank.
My cucumber After block now shuts down the app so it does write the code coverage info:
After do frankly_exit end
with franky_exit beeing:
def frankly_exit
get_to_uispec_server('exit')
# calling exit in the app will not return any response
# so we simply catch the error caused by exiting.
rescue EOFError
end
Now the app will shut down with a clean exit, write the code coverage information and the code coverage step in Jenkins/Hudson will properly display the code coverage for my acceptance tests.
Done, finally :-)
Summary: I have reached my goal to have a test environment to create unit tests (using Google Toolbox for Mac, OCMock & OCHamcrest) and acceptance tests (using Cucumber & Frank) of an iOS app. Both test categories are running in Jenkins/Hudson as two separate builds and both builds provide code coverage information. It took a little bit of work, but it is running now.
That is all I have to say regarding testing an iOS app for now! :-)
This is the 6th article of iOS: Setting Up a Test Environment is a Mess.
Software Noise 