iOS: Test Driving Objective-C Retain & Release, Revisited

Michael found a few problems in the code I presented in iOS: Test Driving Objective-C Retain & Release.

Obviously he tried to use it :-)

Based upon his comments I have revisited the code as you can see it below.

First issue is, that the RetainReleaseMock will simply crash when the verify complains. RetainReleaseMock is no test case and so it is missing the method used to report an issue.

Next issue is, that it doesn’t provide very useful messages. You don’t see if it is complaining about the retain or the release count.

To fix the first issue, Michael moved the assertions back to the test method. Although this is a bit longer than the simple verify, it fixes the first issue and second issue in one step.

Ok, I agree, that’s better so I have revisited my code with some renaming (it is no longer a mock but a spy), a special hamcrest matcher and a noise reducing macro which you may use or not.

Here is the test code (without macro and with macro):

@interface CredentialsTest : GTMTestCase {
}
@end

@implementation CredentialsTest
- (void)testShouldRetainAndReleaseUserAndPassword {
  id user = [[[RetainReleaseSpy alloc] init] autorelease];
  id password = [[[RetainReleaseSpy alloc] init] autorelease];

  [[[Credentials alloc] initWithUser:user password:password] release];

  assertThat (user, isRetained (1));
  assertThat (user, isReleased (1));
  assertThat (password, isRetained (1));
  assertThat (password, isReleased (1));
}

@implementation CredentialsTest
- (void)testShouldRetainAndReleaseUserAndPassword {
  id user = RRSpy;
  id password = RRSpy;

  [[[Credentials alloc] initWithUser:user password:password] release];

  assertThat (user, isRetained (1));
  assertThat (user, isReleased (1));
  assertThat (password, isRetained (1));
  assertThat (password, isReleased (1));
}

@end

In this case I intentionally put everything in a single test and violate the “single assertion (concept) per test” rule. It has everything in one spot and I think I can handle “hidden assertion” issue (I wont see the failures of the later assertions when the first fails). ;-)

Assertion failures are reported like this:

-[CredentialsTest testShouldRetainAndReleaseUserAndPassword] : Expected release count 1, but was 0

The isRetained and isReleased calls are creating hamcrest matchers to make it a little bit easier to read.

Here is the code of the new RetainReleaseSpy, the matcher code (header and implementation) and the matcher factory methods:

// ios
#import <Foundation/Foundation.h>

// test
#import <OCHamcrest/HCBaseMatcher.h>

@interface RetainReleaseSpy : NSObject {
  int refCount;
}

@property (nonatomic, readonly) int actualRetainCount;
@property (nonatomic, readonly) int actualReleaseCount;

- (id)init;
- (id)retain;
- (void)release;

@end


id<HCMatcher> isRetained (int expectedRetainCount);
id<HCMatcher> isReleased (int expectedReleaseCount);

#define RRSpy [[[RetainReleaseMock alloc] init] autorelease]
#import "RetainReleaseSpy.h"

// test
#import <OCHamcrest/OCHamcrest.h>
#import <OCHamcrest/HCDescription.h>


@implementation RetainReleaseSpy

@synthesize actualRetainCount;
@synthesize actualReleaseCount;

- (id)init {
  if ((self = [super init])) {
    actualRetainCount = 0;
    actualReleaseCount = 0;
    refCount = 1;
  }
  return self;  
}

- (id)retain {
  ++refCount;
  ++actualRetainCount;
  return self;
}

- (void)release {
  ++actualReleaseCount;
  --refCount;
  
  if (refCount == 0) {
    [self dealloc];
  }
}

@end


@interface RetainReleaseMatcher : HCBaseMatcher {
  int expectedCount;

  SEL property;
  NSString* info;
}

- (id) initWithExpectedCount:(int)expectedCount property:(SEL)property info:(NSString*)info;

@end


@implementation RetainReleaseMatcher

- (id) initWithExpectedCount:(int)anExpectedCount property:(SEL)aProperty info:(NSString*)anInfo {
  if ((self = [super init])) {
    expectedCount = anExpectedCount;
    property = aProperty;
    info = [anInfo retain];
  }
  return self;
}

- (BOOL) matches:(id)item {
  int actualCount = (int)[item performSelector:property];
  if (actualCount != expectedCount) {
    return NO;
  }
  return YES;
}

- (void) describeTo:(id<HCDescription>)description {
  [description appendText:[NSString stringWithFormat:@"%@ count %d", info, expectedCount]];
}

- (void) describeMismatchOf:(id)item to:(id<HCDescription>)mismatchDescription {
  int actualCount = (int)[item performSelector:property];
  [mismatchDescription appendText:[NSString stringWithFormat:@"was %d", actualCount]];
}

- (void)dealloc {
  [info release];
  [super dealloc];
}   

@end


id<HCMatcher> isRetained (int expectedRetainCount)
{
  return [[[RetainReleaseMatcher alloc] initWithExpectedCount:expectedRetainCount
    property:@selector(actualRetainCount) info:@"retain"] autorelease];
}

id<HCMatcher> isReleased (int expectedReleaseCount)
{
  return [[[RetainReleaseMatcher alloc] initWithExpectedCount:expectedReleaseCount
    property:@selector(actualReleaseCount) info:@"release"] autorelease];
}

Better than before? I think so.

iOS: Test Driving Objective-C Retain & Release

Does it make sense to write unit test for retain & release for my classes?

If I strictly follow Uncle Bob’s TDD Rules then yes. I have to write some test code that will make me add the retain & release cycle for an object.

Even if I don’t strictly follow the rules it may make sense… at least it is a reminder to add the release. Which I forget from time to time.

But finally it is your decision if you test it or not. Anyway, I’d like to talk about how would I do it?

First I tried to achieve it with OCMock but that did not work at all. Fiddling with retain and release on objects out of your control is, let’s say, one on the easiest way to make your test code crash ;-)

So I created a hand made mock called RetainReleaseMock that tracks the retain & release calls.

Here is a test that demonstrates how I use it. It tests that my Credentials object properly retains & releases the user and password NSStrings passed to the initWithUser:password: method.

@interface CredentialsTest : GTMTestCase {
}
@end

@implementation CredentialsTest

- (void)testShouldRetainAndReleaseUserAndPassword {
 id user = [[[RetainReleaseMock alloc] init] autorelease];
 id password = [[[RetainReleaseMock alloc] init] autorelease];

 [[[Credentials alloc] initWithUser:user password:password] release];

 [user verify];
 [password verify];
}

@end

First I create a RetainReleaseMock for the user and password string. Then I create the Credentials object (passing the two mocks) to trigger the retain and directly release it again to trigger the release.

The final step is to verify that retain and release was called on each mock.

That’s it.

Note that you can use [autorelease] to get the mock properly deallocated.

Here is the code of the RetainReleaseMock:

// ios
#import <Foundation/Foundation.h>

@interface RetainReleaseMock : NSObject {
  int refCount;

  int expectedRetainCount;
  int expectedReleaseCount;
  int actualRetainCount;
  int actualReleaseCount;
}

- (id)init;
- (id)initWithRetain:(int)retainCount release:(int)releaseCount;

- (id)retain;
- (void)release;
- (void)verify;

@end

#import "RetainReleaseMock.h"

// test
#import <OCHamcrest/OCHamcrest.h>

@implementation RetainReleaseMock

- (id)init {
  return [self initWithRetain:1 release:1];
}

- (id)initWithRetain:(int)retainCount release:(int)releaseCount {
  if ((self = [super init])) {
    expectedRetainCount = retainCount;
    expectedReleaseCount = releaseCount;
    actualRetainCount = 0;
    actualReleaseCount = 0;
    refCount = 1;
  }
  return self;
}

- (id)retain {
  ++refCount;
  ++actualRetainCount;
  return self;
}

- (void)release {
  ++actualReleaseCount;
  --refCount;

  if (refCount == 0) {
    [self dealloc];
  }
}

- (void)verify {
  assertThatInt (actualRetainCount, equalToInt (expectedRetainCount));
  assertThatInt (actualReleaseCount, equalToInt (expectedReleaseCount));
}

@end

In case you wonder what the second init method is for:

I have added it to test retain & release on an AppDelegate where I cannot pass the dependencies directly to the AppDelegate (.. because it is created by “iOS”). The AppDelegate calls a (global) factory object that I have stubbed (with OCMock) to return the RetainReleaseMock. The factory stub retains my mock too so I have to adjust the expected retain count.

The test then looks like this:

- (void)testReleasesCoreData {
  // extra retain from appFactory stub.
  id coredata = [[[RetainReleaseMock alloc] initWithRetain:2 release:1] autorelease];

  id appFactory = Stub (AppFactory);
  [[[appFactory stub] andReturn:controller] mainViewController];
  [AppFactory setFactory:appFactory];

  [delegate application:nil didFinishLaunchingWithOptions:nil];
  [delegate release];

  [coredata verify];
  [ReadmeAppFactory reset];
}

If you have a better suggestion to handle this situation, I would like to hear it. :-)

iOS: Setting Up Advanced Unit Testing – Take Two

As Jon Reid mentioned on my “Setting Up Advanced Unit Testing” article on iOS testing, OCHamcrest now has  better support for iOS. It creates an OCHamcrest framework for IOS that includes the patches needed for iOS. I can now simply add the framework to my project and add a few additional linker options (which are the same as for OCMock) to make it work.

This should makes it a bit easier to setup hamcrest on iOS. Unfortunately there is a new complication. For iOS I now have to use

#import <OCHamcrestIOS/OCHamcrestIOS.h>

instead of

#import <OCHamcrest/OCHamcrest.h>

I don’t like that. I don’t see why it has to use a different name.

My solution is to build my own static lib for OCHamcrest. That is quite easy and it is my recommended solution.

Initially I switched OCMock to a static library because I had some issue compiling OCMock with block support. It didn’t pick up the proper iOS version and I always ended up with a lib missing block support. So finally I copied all the OCMock source files to my project and created a static library.

The OCMock source now live inside my project in a folder called “Source Xternal/OCMock”. To create a static lib I added a new “static library” target to my project (in XCode, right click Targets and Add/New Target and then select static library) with the OCMock source files below “Source Xternal/OCMock”.  Next I added the new lib (libOCMock.a) to my unit test project by adding it to the “Linked Libraries” (Get Info/Linked Libraries on the unit test target). The last step was to add “Source Xternal” to the “Header Search Paths” in the build settings.

Since the source folder is named OCMock the usual #import will work without change, i.e.

#import <OCMock/OCMock.h>

I didn’t need to modify any #import statement. (Now it also picks up the proper settings to compile it with block support :-)).

I did the same with OCHamcrest. The only difference to the procedure used for OCMock was that I flattened the sources of OCHamcrest into a single folder “Source Xternal/OCHamcrest”. Now all header files are below OCHamcrest and I do not have to touch the hamcrest #imports. (The original source layout uses subfolder to group the files).

This also removed the unwanted “IOS” postfix. So I can simply use the common #import <OCHamcrest/OCHamcrest.h>.

Finally I also moved Google Toolbox for Mac stuff to my “Source Xternal” folder into “Source Xternal/GoogleToolbox” and created another static lib for it.

So my source layout now looks like this:

Source
...
Source Test
...
Source Xternal
Source Xternal/OCMock
Source Xternal/OCHamcrest
Source Xternal/GoogleToolbox

That’s straight forward and also improves the build on my continuous integration server as it now builds all my dependencies as part of my project with the same compiler settings.

Another nice side effect is, that the source of the libraries is not far away and I can easily look into it. From time to time I get strange error messages when compiling and sometimes it helps to take a look into the source of OCHamcrest.

I’m already using this layout for a while now and its definitely my recommendation!

Happy Testing!

iOS: Running Cucumber/Frank with Code Coverage in Jenkins/Hudson

In the previous article iOS: Setting Up Acceptance Testing I introduced Frank. The next and last step will be to run Cucumber and Frank headless from Jenkins/Hudson.

Frank is still new, and it uses apple script to remote control XCode to run the “frankified” app in the simulator. That’s fine for running the cucumber scenarios manually but it is not so good for running them from Jenkins/Hudson.

To run the cucumber tests from Jenkins/Hudson I need it to run headless, i.e. it should not open any window while running the tests. I don’t want Jenkins/Hudson to interfere with whatever I’m doing right know. It should run invisible in the background.

Luckily it is possible to run an iOS app headless simply by running the app directly like this:

<path to app>/MyApp.app/MyApp -RegisterForSystemEvents

By default I would use a cucumber step like the following to run my app from cucumber:

Given /^the app has just started$/ do
  launch_app_in_simulator
end

with launch_app_in_simulator beeing the frank function that uses apple script  to run the app by controlling XCode.

I have changed that to:

Given /^the app has just started$/ do
  launch_app_headless
end

with launch_app_headless looking like this:

def launch_app_headless
  @apppid = fork do
    exec(APP, "-RegisterForSystemEvents")
  end

  wait_for_frank_to_come_up
end

APP is a string variable that contains the path to the frankified app. It is currently set hardcoded in my cucumber Before block:

require 'fileutils'

##
## adjust app, sdk dir and app dir
##
APP    = "build/Debug-iphonesimulator/Readme Frankified.app/Readme Frankified"
APPDIR = "build/Readme.build/Debug-iphonesimulator/Readme Frankified.build"
SDKDIR = "/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.2.sdk"

USERDIR = "#{APPDIR}/iPhone Simulator User Dir"
PREFDIR = "#{USERDIR}/Library/Preferences"

ACCESIBILITY_PLIST   = "com.apple.Accessibility.plist"
ACCESIBILITY_CONTENT = <<PLIST
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
 <key>ApplicationAccessibilityEnabled</key>
 <true/>
</dict>
</plist>
PLIST

Before do
  # check that pwd contains the "build" dir as we are creating
  # items relative to it.
  Dir["build"].length.should == 1

  # make sure we do start with a clean environment
  FileUtils.remove_dir("#{USERDIR}",true)

  pwd     = "#{Dir.pwd}"
  prefdir = "#{pwd}/#{PREFDIR}"
  FileUtils.mkdir_p prefdir

  File.open("#{PREFDIR}/#{ACCESIBILITY_PLIST}", 'w') do |f|
    f <<ACCESIBILITY_CONTENT
  end

  ENV['SDKROOT']               = "#{SDKDIR}"
  ENV['DYLD_ROOT_PATH']        = "#{SDKDIR}"
  ENV['IPHONE_SIMULATOR_ROOT'] = "#{SDKDIR}"
  ENV['TEMP_FILES_DIR']        = "#{pwd}/#{APPDIR}"
  ENV['CFFIXED_USER_HOME']     = "#{pwd}/#{USERDIR}"
end

The accessibility stuff is required to create a configuration file for the app that enables accessibility. Without it Frank will not find the accessibilityLabels which are used to identify the ui elements.

Now I’m able to run the cucumber test for my app headless in Jenkins/Hudson. I created a second build configuration for the acceptance tests. It uses nearly the same setup as for unit testing. I just have added an additional “Execute Shell” step after building the frankified app that runs cucumber:

cd trunk
cucumber -f junit --o ../cucumber --tags ~@ignore features.frank

Using the -f junit parameter cucmber will create an junit xml file so Jenkins/Hudson can display the test results in the same ways as the unit tests. features.frank is the folder were I keep my cucumber scenarios.

Code coverage is set up the same way as for the unit tests (see iOS: Running Unit Tests with Code Coverage in Jenkins/Hudson). Now here is also the last complication. Frank currently doesn’t shut down the app and the result is that the app doesn’t write the coverage information.

To fix this I simply added a super simple exit command (only calling exit(0)) to Franks embedded http server and a ruby function to call it. I have forked the Frank git repository here http://github.com/hauner/Frank.

My cucumber After block now shuts down the app so it does write the code coverage info:

After do
  frankly_exit
end

with franky_exit beeing:

def frankly_exit
  get_to_uispec_server('exit')
  # calling exit in the app will not return any response
  # so we simply catch the error caused by exiting.
  rescue EOFError
end

Now the app will shut down with a clean exit, write the code coverage information and the code coverage step in Jenkins/Hudson will properly display the code coverage for my acceptance tests.

Done, finally :-)

Summary: I have reached my goal to have a test environment to create unit tests (using Google Toolbox for MacOCMock & OCHamcrest) and acceptance tests (using Cucumber & Frank) of an iOS  app. Both test categories are running in Jenkins/Hudson as two separate builds and  both builds provide code coverage information. It took a little bit of work, but it is running now.

That is all I have to say regarding testing an iOS app for now! :-)

This is the 6th article of iOS: Setting Up a Test Environment is a Mess.

iOS: Setting Up Acceptance Testing

After setting up Jenkins/Hudson to build and run my unit tests and collecting code coverage in the last article I will give some guidance for setting up acceptance testing with Cucumber based on a new test framework called Frank.

This article will not make much sense if you do not know cucumber. To learn about cucumber check out the cucumber home page here or take a look at the Secret Ninja Cucumber Scrolls.

Frank embeds a http server into the test version of our app which will be run when the app gets started. Via this internal http server the cucumber part of frank will send commands to the app to remote control it (taps, button taps, filling texts and so on) or to look for ui elements I want to see on specific views.

The first step is create a “frankified” version of the app for testing by modifiying a copy of the iOS Apps target configuration in XCode:

  • main gets replaced by a version that start the http server before it creates the usual UIApplicationMain
  • it links to the Frank library
  • and it includes a Frank resource bundle used by the http server

That is described in more detail in the Frank documentation so I will not repeat it here. After having a “frankified” app we can start writing cucumber scenarios.

Frank already comes with a number of usable cucumber steps, making it easier to get started and are also useful as a template for private steps.

Here is a list of the built-in steps:

Then /^I wait to see "([^\"]*)"$/
Then /^I wait to not see "([^\"]*)"$/
Then /^I wait to see a navigation bar titled "([^\"]*)"$/
Then /^I wait to not see a navigation bar titled "([^\"]*)"$/
Then /^I should see a "([^\"]*)" button$/
Then /^I should see "([^\"]*)"$/
Then /^I should not see "([^\"]*)"$/
Then /^I should see the following:/
Then /^I should see a navigation bar titled "([^\"]*)"$/

When /^I type "([^\"]*)" into the "([^\"]*)" text field$/
When /^I fill in "([^\"]*)" with "([^\"]*)"$/
When /^I fill in text fields as follows:$/

And /^I rotate to the "([^\"]*)"$/
When /^I touch "([^\"]*)"$/
Then /I touch the following:/
When /^I touch the button marked "([^\"]*)"$/
When /^I touch the "([^\"]*)" action sheet button$/
When /^I touch the (\d*)(?:st|nd|rd|th)? action sheet button$/

When /^I flip switch "([^\"]*)"$/
Then /^switch "([^\"]*)" should be on$/
Then /^switch "([^\"]*)" should be off$/

Then /^a pop\-over menu is displayed with the following:$/
Then /^I navigate back$/

Having quite a number of example steps it is not very difficult to create additional steps. All the built-in steps are just a few lines and easy to understand.

For example I created a step that looks like this:

Then /^I should see text fields for:$/ do |table|
  table.raw.each do |v|
    check_element_exists(%Q|textField accessibilityLabel:"#{v}"|);
  end
end

The interesting part is the check_element_exists. It is one of the basic calls used to implement the above steps. It takes care of all the communication with the http server in the “frankified” test app.

By default Frank uses the “Accessibility Label” to locate ui elements in the app using UISpec. The “Accessibility Label”s original use is to give a hint what a control is for to people with visual impairments. Frank uses it to mark ui elements with a “label” that we can use in our cucumber steps to refer to that ui element.

For example the cucumber scenario using my custom step looks like this:

Scenario:
  ask user to login at first startup

  Given the app runs for the first time
  Then I should be on the "Login" screen
  And I should see text fields for:
    | Login    |
    | Password |

The “And” step is using my custom step and it (i.e. UISpec triggered by a call to the embedded http server) will look for two text fields that are marked with the accessibility labels “Login” and “Password”.

I set the labels from Interface Builder by opening the Inspector, switching to the View Identity tab (it is the last tab) and setting the “Label” under the “Accessibility” options.

Note:
To make this work in the simulator I first had to enable “Accessibility” feature in MacOSX, i.e. enable “Enable access for assistive devices” in the “Universal Access” system preferences.

Really cool is, that the parameter to check_element_exists is a UISpec script expression.

This make it possible to check for other properties than the accessibilityLabel of an ui element. I can easily check that the “tag” property (it is on the View Attributes tab in the Interface Builder inspector) of an ui element has a specific value using the following expression:

check_element_exists("view tag:1")

Check out UISpec/UIScript to learn more about this.

To run the cucumber scenarios for my app I simply run the cucumber command from bash and watch frank running trough my acceptance tests.

Currently you will have to open XCode as frank uses apple script to run the app in the simulator from XCode. That is not the perfect way to run the cucumber test automatically from Jenkins/Hudson.

Summary: I have reached my next goal, running Cucumber acceptance tests against my iOS app using Frank. Apart from enabling the accessibility stuff, there were no complications to get it running.

In the next article iOS: Running Cucumber/Frank with Code Coverage in Jenkins/Hudson I will describe how I managed to run my “frankified” app headless from Jenkins/Hudson by using just a view lines of additional ruby code.

This is the 5th article of iOS: Setting Up a Test Environment is a Mess.

iOS: Running Unit Tests with Code Coverage in Jenkins/Hudson

Now that I have a nice unit test environment (see iOS: Setting Up Basic Unit Testing and iOS: Setting Up Advanced Unit Testing) I still need something to run them automatically. I choose Jenkins/Hudson as continuous integration server.

To build a project from Jenkins/Hudson it helps to already have an automatic build. That means you have an XCode project which does not depend on any absolute paths for additional libraries (as Google Toolbox for Mac, OCMock or OCHamcrest). That is the background of the CI Tips to include OCMock and OCHamcrest into the projects source control in my previous articles.

2011/2: see also iOS: Setting Up Advanced Unit Testing Take Two.

Already having an automatic clean build from XCode, it does not take much to run in from Jenkins/Hudson. Quite useful was this article. It shows how to use xcodebuild to run an XCode project without XCode. It also provides a small script to convert the objective-c test output to junit xml output that Jenkins/Hudson will understand.

Based on that blog my “Execute Shell” step looks like this:

xcodebuild -target "UnitTests" -sdk /Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.1.2.sdk/ -configuration "Debug" | /usr/local/bin/ocunit2junit.rb

And the “post build step” “Publish JUnit test result report” like this:

test-reports/*.xml

as “Test report XMLs”.

Great, after following the above article Jenkins/Hudson will watch my repository for changes, check out everything if a new commit hows up, build my “UnitTest” target (automatically running the tests, because of the shell script we added to the unit test project), convert the test output to a junit compatible format and display the test results and statistics in Jenkins/Hudson.

Getting the build running in Jenkins/Hudson was the easy part, making it collect code coverage adds new complications.

Code Coverage: To enable code coverage in the debug unit test target I set the following stuff in the XCode project (GetInfo on the unit test target):

In the “GCC 4.2 Preprocessing” section enable “Generate Test Coverage Files” and “Instrument Program Flow”. Add -lgcov to “Linking” section “other linker flags”. I got that info from here.

Complication No. 4: Unfortunately there are bugs in the XCode (currently 3.2.4 with gcc 4.2.1) that brake the code coverage. In my case link errors and 0% code coverage.

Solution to Complication No. 4: The advice given here helped me to get proper code coverage info. It provides a fixed gcov library (to fix the link errors) and tells you to clear the prefix header (to fix the 0% coverage issue).

Now running the test from Jenkins/Hudson will create the code coverage information but in a format that Jenkins/Hudson does not understand. By using another script called gcovr we can convert the gcov output to Cobertura xml. Cobertura is a Java code coverage tool that Jenkins/Hudson understands. I got this info from here.

Next step is to enable the cobertura “post build step” “Publish Cobertura Coverage Report” and set the “Cobertura xml report pattern” to

**/coverage.xml

in the projects Hudson configuration.

Complication No. 5: Of course this did not work for me out of the box. I had to patch the gcovr script a little bit. Orginally the script will change the current dir and run gcov with the filename of a gcov data file. That does not work. So I modified the script to run gcov with the full path of the data file and that works for me.

Solution to Complication No.5: her is my patch for gcovr:

--- Downloads/gcovr	2010-06-06 18:49:03.000000000 +0200
+++ bin/gcovr	2010-06-08 22:36:21.000000000 +0200
@@ -296,10 +297,13 @@
     (dir,base) = os.path.split(file)
     (name,ext) = os.path.splitext(base)
     prevdir = os.getcwd()
-    os.chdir(dir)
+    #os.chdir(dir)
     (head, tail) = os.path.split(name)
     cmd = gcov_cmd + \
-          " --branch-counts --branch-probabilities --preserve-paths " + tail
+          " --branch-counts --branch-probabilities --preserve-paths " + file \
+          + " -o " + dir
+          #+ tail
     output = subprocess.Popen( cmd.split(" "),
                                stdout=subprocess.PIPE ).communicate()[0]
     #output = subprocess.call(cmd)


CI Tip: add the gcovr script to your repository.

Summary: Finally, after going through all this issues I have an automatic build running in Jenkins/Hudson that runs the unit tests and collects code coverage information.

In the next article iOS: Setting Up Acceptance Testing I will write a little  bit about using Cucumber for acceptance testing with the help of Frank.

This is the 4th article of iOS: Setting Up a Test Environment is a Mess.

iOS: Setting Up Advanced Unit Testing

In the previous article iOS: Setting Up Basic Unit Testing I choose Google Toolbox for Mac as my unit testing framework. Now I will add OCMock and OCHamcrest.

OCMock: To create test doubles (mock, stubs etc.) in a unit test I like the objective-c mock framework OCMock. Its latest version adds support for iOS (universal (simulator, device) static lib). Take care to check the iOS instructions. You have to add a few compiler switches to avoid obscure linker errors.

Here is another nice article explaining what you can do with OCMock: Making Fun of Things with OCMock.

When I tried it for the first time the iOS support was only available in the svn repository and the linker stuff was not yet covered in the instructions. It is now, so we have one complication less :-)

To use the library I adjusted the “Header Seach Paths” & “Library Search Paths” in the “Search Path” section of the unit test target to include the OCMock paths (OCMock/Debug/Headers, OCMock/Debug, OCMock/Release/Headers, OCMock/Release). The Headers are the same for Debug and Release but currently OCMock creates the above layout and it will be easier to update if it is unchanged.

2011/2: I improved my setup: iOS: Setting Up Advanced Unit Testing Take Two.

CI Tip: I added the debug and release headers/library to my subversion project so I do not have to link to external files when I’m going to setup the Jenkins/Hudson build.

Unit Testing Tip: To get some insight why you should use test doubles watch this presentation by J.B. Rainsberger. Note that he is not really talking about integration tests, he talks about not writing integration tests to test units.  See also his blog where he admits that talking about “integration test” in this context is confusing. Anyway, there is something to learn in this video.

OCHamcrest: Another useful extension is OCHamcrest that allows for more declarative assertions. Nothing that is required but it makes the test more readable.

Complication No. 2, OCHamcrest does not provide iOS support out of the box.

2011/2: The issue described here has been fixed, it it no longer necessary to patch OCHamcrest. See also iOS: Setting Up Advanced Unit Testing Take Two.

Solution to Complication No. 2, There is a project on github that adds an iOS target to the build but it doesn’t create a similar universal library as OCMock. To make it similiar to OCMock I used the original Hamcrest objective-c source and added a static library target for iOS based on the OCMock xcode project.

I only used to the git project to fix the compile errors for the iOS target on the original sources. Luckily it required only two small changes:

Index: Source/Core/HCMatcherAssert.mm
===================================================================
--- Source/Core/HCMatcherAssert.mm	(revision 557)
+++ Source/Core/HCMatcherAssert.mm	(working copy)
@@ -%ld,%ld +%ld,%ld @@
#import "HCMatcher.h"

// Objective-C
-#import
+//#import
+#import

namespace {
Index: Source/Core/HCBaseMatcher.mm
===================================================================
--- Source/Core/HCBaseMatcher.mm	(revision 557)
+++ Source/Core/HCBaseMatcher.mm	(working copy)
@@ -%ld,%ld +%ld,%ld @@
- (void) subclassResponsibility:(SEL)command
{
[NSException raise:NSGenericException
-                format:@"-[%@  %s] not implemented", [self className], command];
+                format:@"-[%@  %s] not implemented", [self class/*Name*/], command];
}

@end

To use the library I adjusted the “Header Seach Paths” & “Library Search Paths” in the “Search Path” section of the unit test target to include the OCHamcrest paths as for OCMock (OCHamcrest/Debug/Headers, OCHamcrest/Debug, OCHamcrest/Release/Headers, OCHamcrest/Release).

Finally I added -lOCHamcrest and -lstdc++ (I added -ObjC and -all-load already for OCMock) to “Other Linker Flags” in the “Linking” Section. Adding HC_SHORTHAND to the “Preprocessor Macros” in the “GCC 4.2 Preprocessing” section allows me to assert without the “ST” prefix. (Based on the description from the github hamcrest)

// useless test example ;-)
- (void) testOCHamcrest {
 assertThatBool (YES, equalToBool (YES));
}


Complication No. 3: Unfortunately if an assertion fails the test runner crashes. This seems to be a bug in the simulator.

2011/2: The issue described here has been fixed, it it no longer necessary to patch OCHamcrest. See also iOS: Setting Up Advanced Unit Testing Take Two.

Solution to Complication No. 3: This can be fixed by patching Hamcrest with a patch provided by the Google Toolbox for Mac project. To apply the patch use something like this:

git apply -p 3 ~/Downloads/hamcrest_ios4_fix.patch

After applying the patch it works a lot better, but it is still not perfect. Hopefully the bug in the simulator will be fixed in a future version of the iOS SDK.

Summary: in this article I wrote about extending the unit testing tool set with OCMock and OCHamcrest.

Advice: if you are unfamiliar with OCMock check it out. You do not really need OCHamcrest if you don’t care about the declarative assertions. If you want to avoid Complication No. 2 and No 3. you can skip OCHamcrest. But I like it :-)

The next topic is iOS: Running Unit Tests with Code Coverage in Jenkins/Hudson.

This is the 3rd article of iOS: Setting Up a Test Environment is a Mess.