Interesting is, while I bought most paper books in the past on amazon.de I don’t by software eBooks on amazon.de.

What happend? Not much, the problem is simply that amazon.de just offers kindle/mobi format. Personally, with iBook as my prefered ebook reader, I would like to get them in pub format. Amazon, pub? ehhhh, no.

Second, I also like to have my software ebooks as pdf because they are easier to read on the desktop machine. At home on the Mac and on Windows at work. Amazon, pdf? ehhhh, no again.

So where possible I buy ebooks directly on the publisher website, Manning, PragProg, O’Reilly because they offer all three formats for a single buy. You buy the ebook and you get all three formats.

That looks like a good deal for me. So why should I buy a software ebook on amazon.de? For me, there is no reason to do so.

But not all is lost for amazon. For non software ebooks I don’t care that they are kindle only…

How’s your experience?

Obviously he tried to use it

Based upon his comments I have revisited the code as you can see it below.

First issue is, that the RetainReleaseMock will simply crash when the verify complains. RetainReleaseMock is no test case and so it is missing the method used to report an issue.

Next issue is, that it doesn’t provide very useful messages. You don’t see if it is complaining about the retain or the release count.

To fix the first issue, Michael moved the assertions back to the test method. Although this is a bit longer than the simple verify, it fixes the first issue and second issue in one step.

Ok, I agree, that’s better so I have revisited my code with some renaming (it is no longer a mock but a spy), a special hamcrest matcher and a noise reducing macro which you may use or not.

Here is the test code (without macro and with macro):

@interface CredentialsTest : GTMTestCase {
}
@end

@implementation CredentialsTest
- (void)testShouldRetainAndReleaseUserAndPassword {
  id user = [[[RetainReleaseSpy alloc] init] autorelease];
  id password = [[[RetainReleaseSpy alloc] init] autorelease];

  [[[Credentials alloc] initWithUser:user password:password] release];

  assertThat (user, isRetained (1));
  assertThat (user, isReleased (1));
  assertThat (password, isRetained (1));
  assertThat (password, isReleased (1));
}

@implementation CredentialsTest
- (void)testShouldRetainAndReleaseUserAndPassword {
  id user = RRSpy;
  id password = RRSpy;

  [[[Credentials alloc] initWithUser:user password:password] release];

  assertThat (user, isRetained (1));
  assertThat (user, isReleased (1));
  assertThat (password, isRetained (1));
  assertThat (password, isReleased (1));
}

@end

In this case I intentionally put everything in a single test and violate the “single assertion (concept) per test” rule. It has everything in one spot and I think I can handle “hidden assertion” issue (I wont see the failures of the later assertions when the first fails).

Assertion failures are reported like this:

-[CredentialsTest testShouldRetainAndReleaseUserAndPassword] : Expected release count 1, but was 0

The isRetained and isReleased calls are creating hamcrest matchers to make it a little bit easier to read.

Here is the code of the new RetainReleaseSpy, the matcher code (header and implementation) and the matcher factory methods:

// ios
#import <Foundation/Foundation.h>

// test
#import <OCHamcrest/HCBaseMatcher.h>

@interface RetainReleaseSpy : NSObject {
  int refCount;
}

@property (nonatomic, readonly) int actualRetainCount;
@property (nonatomic, readonly) int actualReleaseCount;

- (id)init;
- (id)retain;
- (void)release;

@end


id<HCMatcher> isRetained (int expectedRetainCount);
id<HCMatcher> isReleased (int expectedReleaseCount);

#define RRSpy [[[RetainReleaseMock alloc] init] autorelease]

#import "RetainReleaseSpy.h"

// test
#import <OCHamcrest/OCHamcrest.h>
#import <OCHamcrest/HCDescription.h>


@implementation RetainReleaseSpy

@synthesize actualRetainCount;
@synthesize actualReleaseCount;

- (id)init {
  if ((self = [super init])) {
    actualRetainCount = 0;
    actualReleaseCount = 0;
    refCount = 1;
  }
  return self;  
}

- (id)retain {
  ++refCount;
  ++actualRetainCount;
  return self;
}

- (void)release {
  ++actualReleaseCount;
  --refCount;
  
  if (refCount == 0) {
    [self dealloc];
  }
}

@end


@interface RetainReleaseMatcher : HCBaseMatcher {
  int expectedCount;

  SEL property;
  NSString* info;
}

- (id) initWithExpectedCount:(int)expectedCount property:(SEL)property info:(NSString*)info;

@end


@implementation RetainReleaseMatcher

- (id) initWithExpectedCount:(int)anExpectedCount property:(SEL)aProperty info:(NSString*)anInfo {
  if ((self = [super init])) {
    expectedCount = anExpectedCount;
    property = aProperty;
    info = [anInfo retain];
  }
  return self;
}

- (BOOL) matches:(id)item {
  int actualCount = (int)[item performSelector:property];
  if (actualCount != expectedCount) {
    return NO;
  }
  return YES;
}

- (void) describeTo:(id<HCDescription>)description {
  [description appendText:[NSString stringWithFormat:@"%@ count %d", info, expectedCount]];
}

- (void) describeMismatchOf:(id)item to:(id<HCDescription>)mismatchDescription {
  int actualCount = (int)[item performSelector:property];
  [mismatchDescription appendText:[NSString stringWithFormat:@"was %d", actualCount]];
}

- (void)dealloc {
  [info release];
  [super dealloc];
}   

@end


id<HCMatcher> isRetained (int expectedRetainCount)
{
  return [[[RetainReleaseMatcher alloc] initWithExpectedCount:expectedRetainCount
    property:@selector(actualRetainCount) info:@"retain"] autorelease];
}

id<HCMatcher> isReleased (int expectedReleaseCount)
{
  return [[[RetainReleaseMatcher alloc] initWithExpectedCount:expectedReleaseCount
    property:@selector(actualReleaseCount) info:@"release"] autorelease];
}

Better than before? I think so.

Objective-C ???? Yes, no joke.

The JetBrains page says:

“appCode is a new IDE based on IntelliJ platform, for developers building apps for Apple devices such as Macs, iPhones & iPads.”

I don’t do daily programming in Java but If I do java programming I use IntelliJ IDEA. At work with the commercial version and at home with the community edition.

My free time interest right know is iOS development and I have written a couple of articles about iOS stuff on my blog. So I was quite curious to take look at appCode.

I downloaded the archive and started to play around with it.

Here are a couple of things I tried:

Loading an Xcode project did work without problem.

Building and running my app did work out of the box. The only thing was that I had to specifically set the device to iPad. It didn’t pick that up by itself.

Using the typical IDEA run/debug configuration dialog I was also able to configure and run my GTM unit tests. appCode recognizes all the targets from the Xcode project and you can modify the SDK, device and so on.

appCode doesn’t seem handle the test results so far. It complained about 65 errors although only 2 failed. Not sure where it got that number from. The failing tests are not appCodes fault. Double clicking the error also didn’t jump to the source.

Debugging works too. I was able to set breakpoints and step through the code. [Update: 11.04.] A bit strange is that it didn’t honor breakpoints in all methods. I’m unable to reproduce the breakpoints that did not work. I retried multiple times and now it just works. I don’t know what made me believe it did not stop on all breakpoints.

The editor itself works great. You are directly at home if you know IntelliJ IDEA. It offers a similar feature set:

IDEAs common navigation shortcuts, auto completion, inspections, live templates and so on.

At the left of the editor you have small icons that tell you that a method overwrites a base class method and clicking it will jump to its declaration. appCode also marks methods with a small pair of arrows icon to jump from declaration to definition and the other way.

Auto completion works really good. I’ve just used it a short time but it looks like it provides better suggestions (less noise) than Xcode.

Inspections. It’s recognizing unused #imports and variables and creates the IDEA typical yellow lines in the editor scroll bar.

Live Templates work too. For example typing alloc<tab> will result in [[ alloc] init] with the cursor before the alloc.

It also comes with a number of refactorings. I tried rename and extract method without problems.

appCode did open a xib file as xml. It didn’t run Interface Builder. Not sure if it is even possible to run Interface Builder now that it is built into Xcode 4.

It also had some problem to find all the #imported files and complained about undefined symbols (in the editor, building works, I guess it simply runs xcodebuild).

I have created a couple of different source folders in my project to organize the sources. In Xcode I have set a couple of additional #import folders and #import header files relative to them.

appCode didn’t pick up the additional #import folders. You can tell appCode about the source root folder and test source root folder but that did not completely fix this issue. I have three parallel source folders: “Source”, “Source Test” and “Source External” and the external folder contains the GTM, OCMock and OCHamcrest code which appCode did not find.

My first impression (after maybe an hour and a half) is quite good. I think I will spend some more time working with it and see how it feels when doing some real work.

What’s your opinion on appCode?

If I strictly follow Uncle Bob’s TDD Rules then yes. I have to write some test code that will make me add the retain & release cycle for an object.

Even if I don’t strictly follow the rules it may make sense… at least it is a reminder to add the release. Which I forget from time to time.

But finally it is your decision if you test it or not. Anyway, I’d like to talk about how would I do it?

First I tried to achieve it with OCMock but that did not work at all. Fiddling with retain and release on objects out of your control is, let’s say, one on the easiest way to make your test code crash

So I created a hand made mock called RetainReleaseMock that tracks the retain & release calls.

Here is a test that demonstrates how I use it. It tests that my Credentials object properly retains & releases the user and password NSStrings passed to the initWithUser:password: method.

@interface CredentialsTest : GTMTestCase {
}
@end

@implementation CredentialsTest

- (void)testShouldRetainAndReleaseUserAndPassword {
 id user = [[[RetainReleaseMock alloc] init] autorelease];
 id password = [[[RetainReleaseMock alloc] init] autorelease];

 [[[Credentials alloc] initWithUser:user password:password] release];

 [user verify];
 [password verify];
}

@end

First I create a RetainReleaseMock for the user and password string. Then I create the Credentials object (passing the two mocks) to trigger the retain and directly release it again to trigger the release.

The final step is to verify that retain and release was called on each mock.

That’s it.

Note that you can use [autorelease] to get the mock properly deallocated.

Here is the code of the RetainReleaseMock:

// ios
#import <Foundation/Foundation.h>

@interface RetainReleaseMock : NSObject {
  int refCount;

  int expectedRetainCount;
  int expectedReleaseCount;
  int actualRetainCount;
  int actualReleaseCount;
}

- (id)init;
- (id)initWithRetain:(int)retainCount release:(int)releaseCount;

- (id)retain;
- (void)release;
- (void)verify;

@end

#import "RetainReleaseMock.h"

// test
#import <OCHamcrest/OCHamcrest.h>

@implementation RetainReleaseMock

- (id)init {
  return [self initWithRetain:1 release:1];
}

- (id)initWithRetain:(int)retainCount release:(int)releaseCount {
  if ((self = [super init])) {
    expectedRetainCount = retainCount;
    expectedReleaseCount = releaseCount;
    actualRetainCount = 0;
    actualReleaseCount = 0;
    refCount = 1;
  }
  return self;
}

- (id)retain {
  ++refCount;
  ++actualRetainCount;
  return self;
}

- (void)release {
  ++actualReleaseCount;
  --refCount;

  if (refCount == 0) {
    [self dealloc];
  }
}

- (void)verify {
  assertThatInt (actualRetainCount, equalToInt (expectedRetainCount));
  assertThatInt (actualReleaseCount, equalToInt (expectedReleaseCount));
}

@end

In case you wonder what the second init method is for:

I have added it to test retain & release on an AppDelegate where I cannot pass the dependencies directly to the AppDelegate (.. because it is created by “iOS”). The AppDelegate calls a (global) factory object that I have stubbed (with OCMock) to return the RetainReleaseMock. The factory stub retains my mock too so I have to adjust the expected retain count.

The test then looks like this:

- (void)testReleasesCoreData {
  // extra retain from appFactory stub.
  id coredata = [[[RetainReleaseMock alloc] initWithRetain:2 release:1] autorelease];

  id appFactory = Stub (AppFactory);
  [[[appFactory stub] andReturn:controller] mainViewController];
  [AppFactory setFactory:appFactory];

  [delegate application:nil didFinishLaunchingWithOptions:nil];
  [delegate release];

  [coredata verify];
  [ReadmeAppFactory reset];
}

If you have a better suggestion to handle this situation, I would like to hear it.

This should makes it a bit easier to setup hamcrest on iOS. Unfortunately there is a new complication. For iOS I now have to use

#import <OCHamcrestIOS/OCHamcrestIOS.h>

instead of

#import <OCHamcrest/OCHamcrest.h>

I don’t like that. I don’t see why it has to use a different name.

My solution is to build my own static lib for OCHamcrest. That is quite easy and it is my recommended solution.

Initially I switched OCMock to a static library because I had some issue compiling OCMock with block support. It didn’t pick up the proper iOS version and I always ended up with a lib missing block support. So finally I copied all the OCMock source files to my project and created a static library.

The OCMock source now live inside my project in a folder called “Source Xternal/OCMock”. To create a static lib I added a new “static library” target to my project (in XCode, right click Targets and Add/New Target and then select static library) with the OCMock source files below “Source Xternal/OCMock”.  Next I added the new lib (libOCMock.a) to my unit test project by adding it to the “Linked Libraries” (Get Info/Linked Libraries on the unit test target). The last step was to add “Source Xternal” to the “Header Search Paths” in the build settings.

Since the source folder is named OCMock the usual #import will work without change, i.e.

#import <OCMock/OCMock.h>

I didn’t need to modify any #import statement. (Now it also picks up the proper settings to compile it with block support ).

I did the same with OCHamcrest. The only difference to the procedure used for OCMock was that I flattened the sources of OCHamcrest into a single folder “Source Xternal/OCHamcrest”. Now all header files are below OCHamcrest and I do not have to touch the hamcrest #imports. (The original source layout uses subfolder to group the files).

This also removed the unwanted “IOS” postfix. So I can simply use the common #import <OCHamcrest/OCHamcrest.h>.

Finally I also moved Google Toolbox for Mac stuff to my “Source Xternal” folder into “Source Xternal/GoogleToolbox” and created another static lib for it.

So my source layout now looks like this:

Source
...
Source Test
...
Source Xternal
Source Xternal/OCMock
Source Xternal/OCHamcrest
Source Xternal/GoogleToolbox

That’s straight forward and also improves the build on my continuous integration server as it now builds all my dependencies as part of my project with the same compiler settings.

Another nice side effect is, that the source of the libraries is not far away and I can easily look into it. From time to time I get strange error messages when compiling and sometimes it helps to take a look into the source of OCHamcrest.

I’m already using this layout for a while now and its definitely my recommendation!

Happy Testing!

Frank is still new, and it uses apple script to remote control XCode to run the “frankified” app in the simulator. That’s fine for running the cucumber scenarios manually but it is not so good for running them from Jenkins/Hudson.

To run the cucumber tests from Jenkins/Hudson I need it to run headless, i.e. it should not open any window while running the tests. I don’t want Jenkins/Hudson to interfere with whatever I’m doing right know. It should run invisible in the background.

Luckily it is possible to run an iOS app headless simply by running the app directly like this:

<path to app>/MyApp.app/MyApp -RegisterForSystemEvents

By default I would use a cucumber step like the following to run my app from cucumber:

Given /^the app has just started$/ do
  launch_app_in_simulator
end

with launch_app_in_simulator beeing the frank function that uses apple script  to run the app by controlling XCode.

I have changed that to:

Given /^the app has just started$/ do
  launch_app_headless
end

with launch_app_headless looking like this:

def launch_app_headless
  @apppid = fork do
    exec(APP, "-RegisterForSystemEvents")
  end

  wait_for_frank_to_come_up
end

APP is a string variable that contains the path to the frankified app. It is currently set hardcoded in my cucumber Before block:

require 'fileutils'

##
## adjust app, sdk dir and app dir
##
APP    = "build/Debug-iphonesimulator/Readme Frankified.app/Readme Frankified"
APPDIR = "build/Readme.build/Debug-iphonesimulator/Readme Frankified.build"
SDKDIR = "/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.2.sdk"

USERDIR = "#{APPDIR}/iPhone Simulator User Dir"
PREFDIR = "#{USERDIR}/Library/Preferences"

ACCESIBILITY_PLIST   = "com.apple.Accessibility.plist"
ACCESIBILITY_CONTENT = <<PLIST
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
 <key>ApplicationAccessibilityEnabled</key>
 <true/>
</dict>
</plist>
PLIST

Before do
  # check that pwd contains the "build" dir as we are creating
  # items relative to it.
  Dir["build"].length.should == 1

  # make sure we do start with a clean environment
  FileUtils.remove_dir("#{USERDIR}",true)

  pwd     = "#{Dir.pwd}"
  prefdir = "#{pwd}/#{PREFDIR}"
  FileUtils.mkdir_p prefdir

  File.open("#{PREFDIR}/#{ACCESIBILITY_PLIST}", 'w') do |f|
    f <<ACCESIBILITY_CONTENT
  end

  ENV['SDKROOT']               = "#{SDKDIR}"
  ENV['DYLD_ROOT_PATH']        = "#{SDKDIR}"
  ENV['IPHONE_SIMULATOR_ROOT'] = "#{SDKDIR}"
  ENV['TEMP_FILES_DIR']        = "#{pwd}/#{APPDIR}"
  ENV['CFFIXED_USER_HOME']     = "#{pwd}/#{USERDIR}"
end

The accessibility stuff is required to create a configuration file for the app that enables accessibility. Without it Frank will not find the accessibilityLabels which are used to identify the ui elements.

Now I’m able to run the cucumber test for my app headless in Jenkins/Hudson. I created a second build configuration for the acceptance tests. It uses nearly the same setup as for unit testing. I just have added an additional “Execute Shell” step after building the frankified app that runs cucumber:

cd trunk
cucumber -f junit --o ../cucumber --tags ~@ignore features.frank

Using the -f junit parameter cucmber will create an junit xml file so Jenkins/Hudson can display the test results in the same ways as the unit tests. features.frank is the folder were I keep my cucumber scenarios.

Code coverage is set up the same way as for the unit tests (see iOS: Running Unit Tests with Code Coverage in Jenkins/Hudson). Now here is also the last complication. Frank currently doesn’t shut down the app and the result is that the app doesn’t write the coverage information.

To fix this I simply added a super simple exit command (only calling exit(0)) to Franks embedded http server and a ruby function to call it. I have forked the Frank git repository here http://github.com/hauner/Frank.

My cucumber After block now shuts down the app so it does write the code coverage info:

After do
  frankly_exit
end

with franky_exit beeing:

def frankly_exit
  get_to_uispec_server('exit')
  # calling exit in the app will not return any response
  # so we simply catch the error caused by exiting.
  rescue EOFError
end

Now the app will shut down with a clean exit, write the code coverage information and the code coverage step in Jenkins/Hudson will properly display the code coverage for my acceptance tests.

Done, finally

Summary: I have reached my goal to have a test environment to create unit tests (using Google Toolbox for Mac,  OCMock & OCHamcrest) and acceptance tests (using Cucumber & Frank) of an iOS  app. Both test categories are running in Jenkins/Hudson as two separate builds and  both builds provide code coverage information. It took a little bit of work, but it is running now.

That is all I have to say regarding testing an iOS app for now!

This is the 6th article of iOS: Setting Up a Test Environment is a Mess.

This article will not make much sense if you do not know cucumber. To learn about cucumber check out the cucumber home page here or take a look at the Secret Ninja Cucumber Scrolls.

Frank embeds a http server into the test version of our app which will be run when the app gets started. Via this internal http server the cucumber part of frank will send commands to the app to remote control it (taps, button taps, filling texts and so on) or to look for ui elements I want to see on specific views.

The first step is create a “frankified” version of the app for testing by modifiying a copy of the iOS Apps target configuration in XCode:

• main gets replaced by a version that start the http server before it creates the usual UIApplicationMain
• it links to the Frank library
• and it includes a Frank resource bundle used by the http server

That is described in more detail in the Frank documentation so I will not repeat it here. After having a “frankified” app we can start writing cucumber scenarios.

Frank already comes with a number of usable cucumber steps, making it easier to get started and are also useful as a template for private steps.

Here is a list of the built-in steps:

Then /^I wait to see "([^\"]*)"$/
Then /^I wait to not see "([^\"]*)"$/
Then /^I wait to see a navigation bar titled "([^\"]*)"$/
Then /^I wait to not see a navigation bar titled "([^\"]*)"$/
Then /^I should see a "([^\"]*)" button$/
Then /^I should see "([^\"]*)"$/
Then /^I should not see "([^\"]*)"$/
Then /^I should see the following:/
Then /^I should see a navigation bar titled "([^\"]*)"$/

When /^I type "([^\"]*)" into the "([^\"]*)" text field$/
When /^I fill in "([^\"]*)" with "([^\"]*)"$/
When /^I fill in text fields as follows:$/

And /^I rotate to the "([^\"]*)"$/
When /^I touch "([^\"]*)"$/
Then /I touch the following:/
When /^I touch the button marked "([^\"]*)"$/
When /^I touch the "([^\"]*)" action sheet button$/
When /^I touch the (\d*)(?:st|nd|rd|th)? action sheet button$/

When /^I flip switch "([^\"]*)"$/
Then /^switch "([^\"]*)" should be on$/
Then /^switch "([^\"]*)" should be off$/

Then /^a pop\-over menu is displayed with the following:$/
Then /^I navigate back$/

Having quite a number of example steps it is not very difficult to create additional steps. All the built-in steps are just a few lines and easy to understand.

For example I created a step that looks like this:

Then /^I should see text fields for:$/ do |table|
  table.raw.each do |v|
    check_element_exists(%Q|textField accessibilityLabel:"#{v}"|);
  end
end

The interesting part is the check_element_exists. It is one of the basic calls used to implement the above steps. It takes care of all the communication with the http server in the “frankified” test app.

By default Frank uses the “Accessibility Label” to locate ui elements in the app using UISpec. The “Accessibility Label”s original use is to give a hint what a control is for to people with visual impairments. Frank uses it to mark ui elements with a “label” that we can use in our cucumber steps to refer to that ui element.

For example the cucumber scenario using my custom step looks like this:

Scenario:
  ask user to login at first startup

  Given the app runs for the first time
  Then I should be on the "Login" screen
  And I should see text fields for:
    | Login    |
    | Password |

The “And” step is using my custom step and it (i.e. UISpec triggered by a call to the embedded http server) will look for two text fields that are marked with the accessibility labels “Login” and “Password”.

I set the labels from Interface Builder by opening the Inspector, switching to the View Identity tab (it is the last tab) and setting the “Label” under the “Accessibility” options.

Note:
To make this work in the simulator I first had to enable “Accessibility” feature in MacOSX, i.e. enable “Enable access for assistive devices” in the “Universal Access” system preferences.

Really cool is, that the parameter to check_element_exists is a UISpec script expression.

This make it possible to check for other properties than the accessibilityLabel of an ui element. I can easily check that the “tag” property (it is on the View Attributes tab in the Interface Builder inspector) of an ui element has a specific value using the following expression:

check_element_exists("view tag:1")

Check out UISpec/UIScript to learn more about this.

To run the cucumber scenarios for my app I simply run the cucumber command from bash and watch frank running trough my acceptance tests.

Currently you will have to open XCode as frank uses apple script to run the app in the simulator from XCode. That is not the perfect way to run the cucumber test automatically from Jenkins/Hudson.

Summary: I have reached my next goal, running Cucumber acceptance tests against my iOS app using Frank. Apart from enabling the accessibility stuff, there were no complications to get it running.

In the next article iOS: Running Cucumber/Frank with Code Coverage in Jenkins/Hudson I will describe how I managed to run my “frankified” app headless from Jenkins/Hudson by using just a view lines of additional ruby code.

This is the 5th article of iOS: Setting Up a Test Environment is a Mess.

To build a project from Jenkins/Hudson it helps to already have an automatic build. That means you have an XCode project which does not depend on any absolute paths for additional libraries (as Google Toolbox for Mac, OCMock or OCHamcrest). That is the background of the CI Tips to include OCMock and OCHamcrest into the projects source control in my previous articles.

2011/2: see also iOS: Setting Up Advanced Unit Testing Take Two.

Already having an automatic clean build from XCode, it does not take much to run in from Jenkins/Hudson. Quite useful was this article. It shows how to use xcodebuild to run an XCode project without XCode. It also provides a small script to convert the objective-c test output to junit xml output that Jenkins/Hudson will understand.

Based on that blog my “Execute Shell” step looks like this:

xcodebuild -target "UnitTests" -sdk /Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.1.2.sdk/ -configuration "Debug" | /usr/local/bin/ocunit2junit.rb

And the “post build step” “Publish JUnit test result report” like this:

test-reports/*.xml

as “Test report XMLs”.

Great, after following the above article Jenkins/Hudson will watch my repository for changes, check out everything if a new commit hows up, build my “UnitTest” target (automatically running the tests, because of the shell script we added to the unit test project), convert the test output to a junit compatible format and display the test results and statistics in Jenkins/Hudson.

Getting the build running in Jenkins/Hudson was the easy part, making it collect code coverage adds new complications.

Code Coverage: To enable code coverage in the debug unit test target I set the following stuff in the XCode project (GetInfo on the unit test target):

In the “GCC 4.2 Preprocessing” section enable “Generate Test Coverage Files” and “Instrument Program Flow”. Add -lgcov to “Linking” section “other linker flags”. I got that info from here.

Complication No. 4: Unfortunately there are bugs in the XCode (currently 3.2.4 with gcc 4.2.1) that brake the code coverage. In my case link errors and 0% code coverage.

Solution to Complication No. 4: The advice given here helped me to get proper code coverage info. It provides a fixed gcov library (to fix the link errors) and tells you to clear the prefix header (to fix the 0% coverage issue).

Now running the test from Jenkins/Hudson will create the code coverage information but in a format that Jenkins/Hudson does not understand. By using another script called gcovr we can convert the gcov output to Cobertura xml. Cobertura is a Java code coverage tool that Jenkins/Hudson understands. I got this info from here.

Next step is to enable the cobertura “post build step” “Publish Cobertura Coverage Report” and set the “Cobertura xml report pattern” to

**/coverage.xml

in the projects Hudson configuration.

Complication No. 5: Of course this did not work for me out of the box. I had to patch the gcovr script a little bit. Orginally the script will change the current dir and run gcov with the filename of a gcov data file. That does not work. So I modified the script to run gcov with the full path of the data file and that works for me.

Solution to Complication No.5: her is my patch for gcovr:

--- Downloads/gcovr	2010-06-06 18:49:03.000000000 +0200
+++ bin/gcovr	2010-06-08 22:36:21.000000000 +0200
@@ -296,10 +297,13 @@
     (dir,base) = os.path.split(file)
     (name,ext) = os.path.splitext(base)
     prevdir = os.getcwd()
-    os.chdir(dir)
+    #os.chdir(dir)
     (head, tail) = os.path.split(name)
     cmd = gcov_cmd + \
-          " --branch-counts --branch-probabilities --preserve-paths " + tail
+          " --branch-counts --branch-probabilities --preserve-paths " + file \
+          + " -o " + dir
+          #+ tail
     output = subprocess.Popen( cmd.split(" "),
                                stdout=subprocess.PIPE ).communicate()[0]
     #output = subprocess.call(cmd)

CI Tip: add the gcovr script to your repository.

Summary: Finally, after going through all this issues I have an automatic build running in Jenkins/Hudson that runs the unit tests and collects code coverage information.

In the next article iOS: Setting Up Acceptance Testing I will write a little  bit about using Cucumber for acceptance testing with the help of Frank.

This is the 4th article of iOS: Setting Up a Test Environment is a Mess.

OCMock: To create test doubles (mock, stubs etc.) in a unit test I like the objective-c mock framework OCMock. Its latest version adds support for iOS (universal (simulator, device) static lib). Take care to check the iOS instructions. You have to add a few compiler switches to avoid obscure linker errors.

Here is another nice article explaining what you can do with OCMock: Making Fun of Things with OCMock.

When I tried it for the first time the iOS support was only available in the svn repository and the linker stuff was not yet covered in the instructions. It is now, so we have one complication less

To use the library I adjusted the “Header Seach Paths” & “Library Search Paths” in the “Search Path” section of the unit test target to include the OCMock paths (OCMock/Debug/Headers, OCMock/Debug, OCMock/Release/Headers, OCMock/Release). The Headers are the same for Debug and Release but currently OCMock creates the above layout and it will be easier to update if it is unchanged.

2011/2: I improved my setup: iOS: Setting Up Advanced Unit Testing Take Two.

CI Tip: I added the debug and release headers/library to my subversion project so I do not have to link to external files when I’m going to setup the Jenkins/Hudson build.

Unit Testing Tip: To get some insight why you should use test doubles watch this presentation by J.B. Rainsberger. Note that he is not really talking about integration tests, he talks about not writing integration tests to test units.  See also his blog where he admits that talking about “integration test” in this context is confusing. Anyway, there is something to learn in this video.

OCHamcrest: Another useful extension is OCHamcrest that allows for more declarative assertions. Nothing that is required but it makes the test more readable.

Complication No. 2, OCHamcrest does not provide iOS support out of the box.

2011/2: The issue described here has been fixed, it it no longer necessary to patch OCHamcrest. See also iOS: Setting Up Advanced Unit Testing Take Two.

Solution to Complication No. 2, There is a project on github that adds an iOS target to the build but it doesn’t create a similar universal library as OCMock. To make it similiar to OCMock I used the original Hamcrest objective-c source and added a static library target for iOS based on the OCMock xcode project.

I only used to the git project to fix the compile errors for the iOS target on the original sources. Luckily it required only two small changes:

Index: Source/Core/HCMatcherAssert.mm
===================================================================
--- Source/Core/HCMatcherAssert.mm	(revision 557)
+++ Source/Core/HCMatcherAssert.mm	(working copy)
@@ -%ld,%ld +%ld,%ld @@
#import "HCMatcher.h"

// Objective-C
-#import
+//#import
+#import

namespace {
Index: Source/Core/HCBaseMatcher.mm
===================================================================
--- Source/Core/HCBaseMatcher.mm	(revision 557)
+++ Source/Core/HCBaseMatcher.mm	(working copy)
@@ -%ld,%ld +%ld,%ld @@
- (void) subclassResponsibility:(SEL)command
{
[NSException raise:NSGenericException
-                format:@"-[%@  %s] not implemented", [self className], command];
+                format:@"-[%@  %s] not implemented", [self class/*Name*/], command];
}

@end

To use the library I adjusted the “Header Seach Paths” & “Library Search Paths” in the “Search Path” section of the unit test target to include the OCHamcrest paths as for OCMock (OCHamcrest/Debug/Headers, OCHamcrest/Debug, OCHamcrest/Release/Headers, OCHamcrest/Release).

Finally I added -lOCHamcrest and -lstdc++ (I added -ObjC and -all-load already for OCMock) to “Other Linker Flags” in the “Linking” Section. Adding HC_SHORTHAND to the “Preprocessor Macros” in the “GCC 4.2 Preprocessing” section allows me to assert without the “ST” prefix. (Based on the description from the github hamcrest)

// useless test example ;-)
- (void) testOCHamcrest {
 assertThatBool (YES, equalToBool (YES));
}

Complication No. 3: Unfortunately if an assertion fails the test runner crashes. This seems to be a bug in the simulator.

2011/2: The issue described here has been fixed, it it no longer necessary to patch OCHamcrest. See also iOS: Setting Up Advanced Unit Testing Take Two.

Solution to Complication No. 3: This can be fixed by patching Hamcrest with a patch provided by the Google Toolbox for Mac project. To apply the patch use something like this:

git apply -p 3 ~/Downloads/hamcrest_ios4_fix.patch

After applying the patch it works a lot better, but it is still not perfect. Hopefully the bug in the simulator will be fixed in a future version of the iOS SDK.

Summary: in this article I wrote about extending the unit testing tool set with OCMock and OCHamcrest.

Advice: if you are unfamiliar with OCMock check it out. You do not really need OCHamcrest if you don’t care about the declarative assertions. If you want to avoid Complication No. 2 and No 3. you can skip OCHamcrest. But I like it

The next topic is iOS: Running Unit Tests with Code Coverage in Jenkins/Hudson.

This is the 3rd article of iOS: Setting Up a Test Environment is a Mess.

Complication No. 1, “the Apple way”: XCode comes with a unit test framwork, OCUnit, but unfortunately its integration is a little bit strange. You create an octest bundle that’s “somehow” (I didn’t try to understand how this is working in detail) loaded into your normal project target and then will run your unit tests.

Usually I would simply create a second binary that contains the test sources and the production sources and links to the same libraries plus the libraries needed for testing.

The only advantage I see in using the bundle approach is that I do not have to create a test target with nearly the same setup for the unit tests (copying all settings, adding all classes you want to test and so on…).

One annoying drawback is setting it up for debugging. Sometimes it is useful to step trough a test to understand where it is going wrong but you will need additional setup to debug your unit tests with breakpoints. That’s inconvenient but possible without to much work for normal MacOSX development (i.e not iOS). For iOS this is getting more complicated. As the article says: “What a mess”.

Another drawback is that you can’t run the tests in the simulator or on the device itself because your test bundle is not an ordinary iOS app.

Solution to Complication No. 1: While looking for iPhone/iOS test resources I found Google Toolbox for Mac (GTM). First I was wondering why I would need it but after being a little bit frustrated by the setup of the standard Apple way above, I gave it a try.

Basically you create a normal standalone iOS app as part of your project. You just have to add a couple of GTM source files to your test target app (a copy of the production app), setup a “run script” build phase (to run the test after they are built) and you are done. This is described in the GTM iPhoneUnitTesting instructions.

Note: I had to remove the MainWindow.xib from the plist file created for my test target. With the xib set I got some complains about the app being unable to load it.

CI Tip: I copied the required files of GTM to my subversion project so I do not have to link to external files when I’m going to setup the Jenkins/Hudson build. 2011/2: See also iOS: Setting Up Advanced Unit Testing Take Two.

The only difference for creating new tests is that you have to use GTMTestCase as base class instead of SenTestCase. But that is not a big issue and GTMTestCase itself is based on SenTestCase. Which is nice because it will automatically play well with OCMock and OCHamcrest (which will be covered in the next article).

A lot simpler than the default way. Nice is that you will not need any extra setup to step into your test code. Just set breakpoints and run the test target. And because it is an ordinary app it will run in the simulator and on the device.

GTM has more stuff to explore (e.g. support for UI testing, additional asserts) but I have not yet played with it.

Summary: with GTM I can easily debug my test cases and run the test in the simulator and on the device. Not bad!

Advice: go with Google Toolbox for Mac for unit testing on iOS.

In iOS: Setting Up Advanced Unit Testing I will pimp the unit testing environment with OCMock and OCHamcrest which has a few new complications.

This is the 2nd article of iOS: Setting Up a Test Environment is a Mess.