Cucumber & Grails: Transaction Rollback

… the short version is: “the cucumber grails plugin will not provide automatic transaction rollback”.

After adding automatic rollback code in the cucumber grails plugin (version 0.5) and trying to use it I had to learn, that the automatic part is flawed and that it does not work in all cases.

I will try to explain why it does not work and how the cucumber plugin will deal with it. If you happen to know more about how Grails/Spring transactions work, please correct me where I am wrong. If you have to add something please do! :-)

First some background how Grails handles database transactions by default.

runtime

If we use Grails default settings, it will wrap any service call into its own transaction. If we make two service calls in a single controller action, each one will have its own transaction. We will take a look at a simple example to demonstrate the convention. See also the Services chapter in the grails documentation.

You can change the defaults, but changing them will not improve anything. It will just make the situation more complicated and therefore it does only supports the statement that we will not get automatic transaction rollback for functional testing. :-)

Reusing the plugins Book example and running grails (run-app) we can easily check this behaviour.

package books

class Book {
    String author
    String title

    static constraints = {
        title unique: true
    }
}

The service looks like this:

package books

class BookService {
    Book add (boolean error, Map params) {
        def newBook = new Book (params)
        newBook.save (failOnError: true)
        if (error) {
            throw new RuntimeException ("rollback!")
        }
        newBook
    }
}

To easily get an error when we want to, I have added a parameter to the add() method that tells the function to error.

The following controller code uses the service with code you should not write in real code. It is written like this to demonstrate grails transaction handling. You should use a single service call to make the action run with a single transaction.

Now when we run the multipleTransactions() action in the following controller:

package books

class BookController {
    def bookService

    def multipleTransactions () {
        params.author = "A1"
        params.title = "T1"
        bookService.add (false, params)

        params.author = "A2"
        params.title = "T2"
        bookService.add (true, params)
    }
}

… and look at the database (you can use the dbconsole to look into the h2 database) we will see a single row:

ID VERSION AUTHOR TITLE
1 0 A10 T10

After reading Services this should not be a big surprise. Both add() calls to the service get their own transaction. Because of the true parameter in the second call, it throws and error an grails will automatically rollback the transaction. But only the transaction of the second call. The transaction of the first call is already committed.

integration test

In integration tests grails does automatically wrap each test in a transaction and triggers a rollback when the test is finished (which by the way is the code I called to get the rollback in the plugin).

It just does not work in all cases.

It fails when we exercise the application with http requests. I assume because grails session/request code will run (which is not run in an integration test when we simply new the controller) using its own transaction code that is ignoring the transaction wrapped around the test (guessing here, I don’t yet understand what grails does in detail).

It also fails if we start to run code using withTransaction().

package books

class BookService {
    Book add (Map params) {
        def newBook
        Book.withNewTransaction {
            newBook = new Book (params)
            newBook.save (failOnError: true)
        }
        newBook     
    }
}

.. with the controller:

package books

class BookController {
    def bookService

    def transaction () {
        params.author = "A1"
        params.title = "T1"
        bookService.add (params)
    }
}

… and when we run two cucumber scenarios calling BookController.transaction () the second call will fail because of the unique constraint on the title. The withTransaction() part will not be rollbacked.

conclusion

So where does this information place us regarding rollback of database transactions in context of functional testing with cucumber? Although it is a play example it shows that transaction handling can already get complex in such a simple example. Now consider multiple request, service annotations that change the default behaviour or “manual” withTransaction() calls. It is not as easy as it looks.

Maybe it would work if we were able to intercept the creation of any transaction created by grails, always return the same transaction and rollback it when the test is finished.

Appart from not finding any information how this would work (maybe I didn’t use the correct search words) it does not look like a good idea. Although it sounds like a solution in the first moment, it will make grails behave differently in the test enviroment and the production environment. How big the difference is, is hard to say. It depends on how much you change grails default behaviour.

Anyway, it will behave differently and I think transactions are important enough that we do not want this when doing functional testing. When we run end to end, the transactions should be identical to production.

My conclusion is, that the transaction handling is complicated enough that the plugin can not handle this for us. Therefore it will no longer run grails transaction code automatically. We will have to take care of cleaning up the database by ourself. Either by completely dropping and re-creating the database schema before each test or by running some statements that will restore the database to its original state (I will write another article showing an example for both ways).

The plugin will still provide some simple configuration to enable the integration test rollback support. You can use it to get the automatic rollback when you build functional integration tests, e.g. you do not exercise the app using http requests and you don’t use withTransaction().

IntelliJ IDEA & Spock: Groovy Pointless Arithmetic

Maybe this is just plain stupid but there is a simple workaround for the warning 0 * report.addError can be replaced by 0 in the following test (in IDEA):

def "does not report step error or failure when it succeeds" () {
  Result result = Mock (Result)
  result.error >> null

  when:
    uat.step (stepStub ())
    uat.result (result)

  then:
    0 * report.addError (_)
    0 * report.addFailure (_)
}

My solution so far was to add an @SuppressWarnings(["GroovyPointlessArithmetic"]) to the test class. But there is a simpler solution without disabling the warning. I can simply write

def "does not report step error or failure when it succeeds" () {
  Result result = Mock (Result)
  result.error >> null

  when:
    uat.step (stepStub ())
    uat.result (result)

  then:
    (0) * report.addError (_)
    (0) * report.addFailure (_)
}

to remove the warning. See the difference? I have replaced the 0 with (0) and the warning goes away. :-)

Automating Specification with Cucumber & Grails

Specification by Example is an interesting approach to build a specification in collaboration with a customer. One part of it is to automate the specification without changing it and to build a living documentation system that is always in sync with the software.

One possibility to automate the specification is to describe it using Cucumbers Given, When & Then style and writing some glue code to connect the steps with the system we build.

I will introduce you to a new cucumber grails plugin (grails.org/plugins, documentation) that make it a lot easier to automate cucumber specifications of a Grails application.

The plugin is a functional test plugin for grails and it is only available in grails functional test phase. It is interesting because it runs cucumber inside a running grails application which allows us to use the grails api in the cucumber step implementations. We can call GORM stuff like dynamic finders, services and controllers.

To make the magic work, the plugin is based on Cucumber-JVM the native JVM implementation of cucumber. Cucumber-JVM does support many JVM languages. Groovy is one of them which makes it a perfect fit for grails.

As we will see in the following introduction, the plugin integrates into grails standard test infrastructure which makes it quite easy to use. The full source code of the example and the plugin itself is online in my github repository (example source code, plugin source code).

We will walk through a very simple example that will use cucumber to test two scenarios against a grails web application. We will implement the steps under the skin of the application, i.e. below the ui.
I have written another article (here) that runs the same example against the ui using Geb in the steps to remote control a web browser (Note: because of my latest changes in the plugin the article is no longer 100% up to date).

Let us start with the setup…

setup

First, create a grails app with: grails create-app Books (I am using grails 2.0.3). Next, add the plugin dependency to grails-app/conf/BuildConfig.groovy in the plugins section:

plugins {
   test ":cucumber:0.4.0"
}

Done, setup complete.

.feature files

By convention, the plugin expects all cucumber files (features and steps) in test/functional (you can change it in the configuration, see the documentation).

Create the following two .feature files in test/functional which describe the functionality of the example application:

NewBook.feature:

Feature: new book entry
    As a book owner
    I want to add books I own to the book tracker
    so that I do not have to remember them by myself

Scenario: new book
   Given I open the book tracker
    When I add "Specification by Example"
    Then I see "Specification by Example"s details

ListBooks.feature:

Feature: list owned books
    As a book owner
    I want to list my books
    so that I can look up which books I own

@ignore
Scenario: list existing books
   Given I have already added "Specification by Example"
    When I view the book list
    Then my book list contains "Specification by Example"

Specification by Example: Website, amazon.com

running the features

I already mentioned that the plugin properly integrates into grails and that it is a functional test plugin. To run the cucumber features only we can use the usual grails test-app command using one of the following variations:

grails test-app functional:cucumber
grails test-app :cucumber
grails test-app functional:

Running it we will see the typical cucumber output for missing step implementations:

You can implement missing steps with the snippets below:

Given(~"^I have already added \"([^\"]*)\"$") { String arg1 ->
    // Express the Regexp above with the code you wish you had
    throw new PendingException()
}
When(~"^I view the book list$") { ->
    // Express the Regexp above with the code you wish you had
    throw new PendingException()
}
Then(~"^my book list contains \"([^\"]*)\"$") { String arg1 ->
    // Express the Regexp above with the code you wish you had
    throw new PendingException()
}
Given(~"^I open the book tracker$") { ->
    // Express the Regexp above with the code you wish you had
    throw new PendingException()
}
When(~"^I add \"([^\"]*)\"$") { String arg1 ->
    // Express the Regexp above with the code you wish you had
    throw new PendingException()
}
Then(~"^I see \"([^\"]*)\"s details$") { String arg1 ->
    // Express the Regexp above with the code you wish you had
    throw new PendingException()
}

tagging features

To concentrate on implementing the steps for the NewBook feature we will disable the second one by adding an @ignore tag to it and by telling the plugin to ignore the scenarios tagged with @ignore:

ListBooks.feature:

Feature: list owned books
    As a book owner
    I want to list my books
    so that I can look up which books I own

@ignore
Scenario: list existing books
   Given I have already added "Specification by Example"
    When I view the book list
    Then my book list contains "Specification by Example"

The plugin will pick up a couple of configuration options from grails-app/conf/CucumberConfig.groovy, so we create it and add a tags configuration like this:

cucumber {
    tags = ["~@ignore"]
}

tags is list of strings and each item corresponds to a standard cucumber --tags option.

Running grails test-app :cucumber again, we will only get the missing steps message for the NewBook feature:

You can implement missing steps with the snippets below:

Given(~"^I open the book tracker$") { ->
    // Express the Regexp above with the code you wish you had
    throw new PendingException()
}
When(~"^I add \"([^\"]*)\"$") { String arg1 ->
    // Express the Regexp above with the code you wish you had
    throw new PendingException()
}
Then(~"^I see \"([^\"]*)\"s details$") { String arg1 ->
    // Express the Regexp above with the code you wish you had
    throw new PendingException()
}

implementing steps, part one

Create a new file test/functional/steps/Book_steps.groovy and

  • copy the step templates into it
  • add an import for PendingException
  • mixin the EN language of Gherkin
  • escape the $ at the end of the regular expression because in groovy it is a special character in GStrings. Alternativly you can replace the double quotes with single quotes.

Finally it should look like this:

import cucumber.runtime.PendingException

this.metaClass.mixin (cucumber.runtime.groovy.EN)


Given (~"^I open the book tracker\$") { ->
    // Express the Regexp above with the code you wish you had
    throw new PendingException()
}

When (~"^I add \"([^\"]*)\"\$") { String arg1 ->
    // Express the Regexp above with the code you wish you had
    throw new PendingException()
}

Then (~"^I see \"([^\"]*)\"s details\$") { String arg1 ->
    // Express the Regexp above with the code you wish you had
    throw new PendingException()
}

Running again will throw a PendingException. To implement the steps we will need a domain class, a service and a controller. That is what we will do next, step by step…

Given (~‘^I open the book tracker$’)

This step is easy. We want to test under the skin ignoring the ui, so we simply do nothing in this step. In a real project we would probably check that the controller has an action that returns the entry page of our application.

Given (~"^I open the book tracker\$") { ->
    // nop
}

When (~‘^I add “([^"]*)”$’)

In this step we want to add a book. We need a controller with an add action that we will call in the step implementation.

Create a BookController with an add action method. It will get the book details we enter in the ui form and it will return the new Book as JSON that will be shown in the ui. We don’t care what the ui does with it, we will only check that we get the correct JSON.

As is usual in grails the controller will just delegate the work to a service and prepare the result for the ui. In this case rendering the Book as JSON. Here is the controller code:

package books

import grails.converters.JSON

class BookController {
    def bookService

    def add () {
        render bookService.add (params) as JSON
    }
}

The service:

package books

class BookService {
    Book add (Map params) {
        def newBook = new Book (params)
        newBook.save ()
        newBook
    }
}

And finally the Book domain class:

package books

class Book {
    String author
    String title
}

After creating the grails artifacts we can finally implement the step like this:

import books.BookController
import data.Data

// Scenario State
BookController bookController

When (~"^I add \"([^\"]*)\"\$") { String bookTitle ->
    bookController = new BookController ()
    bookController.params << Data.findByTitle (bookTitle)
    bookController.add ()
}

Note the additional imports and the bookController variable we will need to call the controller we create in the Then step.

If you are familiar with Grails you will probably understand most of it. We are simply using grails standard integration test logic for controllers.

Data is a small helper class that will lookup all the properties of a book identified by the bookTitle. This has the advantage that we do not have to list all properties of a book in the .feature file and the feature is more robust against changes like adding an additional property. That the additional property gets handled properly would be tested on the integration or unit test level.

Here is the source for test/functional/data/Data.groovy:

package data

import books.Book


class Data {
    static def books = [
        [title: "Specification by Example", author: "Gojko Adzic"]
    ]

    static public def findByTitle (String title) {
        books.find { book ->
            book.title == title
        }
    }

    static void clearBooks () {
        Book.findAll()*.delete (flush: true)
    }
}

Running the features now will still fail with a rather obscure java.lang.IllegalStateException exception (I guess at least for most of us, count me in). We are calling the controller outside of a real http request and to fix this we have to add a few simple lines of setup and tear down code.

Before & After

If we are building normal integration tests for a controller, grails will take care of setting up the test environment in a way that we will not get the IllegalStateException, that we have a mock request and response etc. and that we can set the request parameters with bookController.params = ... .

When we run cucumber we have to take care of this ourself. Let’s create a new file test/functional/hooks/env.grooy with the following content:

import org.codehaus.groovy.grails.test.support.GrailsTestRequestEnvironmentInterceptor

this.metaClass.mixin (cucumber.runtime.groovy.Hooks)


GrailsTestRequestEnvironmentInterceptor scenarioInterceptor

Before () {
    scenarioInterceptor = new GrailsTestRequestEnvironmentInterceptor (appCtx)
    scenarioInterceptor.init ()
}

After () {
    scenarioInterceptor.destroy ()
}

Running cucumber now will fail with a PendingException from our last step in this scenario. We still have to check that the result is correct to finish our first scenario.

Then (~‘^I see “([^"]*)”s details$’)

We need to assert that the returned JSON has an id (so we know it was save()d) and author and title with the same values we passed to the BookController:

Then (~"^I see \"([^\"]*)\"s details\$") { String bookTitle ->
    def expected = Data.findByTitle (bookTitle)
    def actual = bookController.response.json

    assert actual.id
    assert actual.title  == expected.title
    assert actual.author == expected.author
}

Next run and we see:

> grails test-app :cucumber --stacktrace
| Server running. Browse to http://localhost:8080/Books
| Running 1 cucumber test...
| Completed 1 cucumber test, 0 failed in 1623ms
| Server stopped
| Tests PASSED - view reports in target/test-reports

Great, the scenario passed! Now let’s finish our example application by implementing the steps for the ListBooks feature. There is a little bit more :-)

implementing steps, part two

First remove the @ignore tag from the list existing books scenario and run it again to get the templates for the missing steps and copy them to the BookSteps.groovy file (don’t forget to escape the ‘$’):

Given(~"^I have already added \"([^\"]*)\"$") { String arg1 ->
    // Express the Regexp above with the code you wish you had
    throw new PendingException()
}
When(~"^I view the book list$") { ->
    // Express the Regexp above with the code you wish you had
    throw new PendingException()
}
Then(~"^my book list contains \"([^\"]*)\"$") { String arg1 ->
    // Express the Regexp above with the code you wish you had
    throw new PendingException()
}

Given (~‘^I have already added “([^"]*)”$’)

We want to pre-populate our database. We will reuse our BookService to create a new Book in the database. Reusing our business code to setup test data looks like a good idea to make sure that the we get proper data into the database without duplicating the code for our tests.

Here is the code:

Given (~"^I have already added \"([^\"]*)\"\$") { String bookTitle ->
    def bookService = appCtx.getBean ("bookService")
    bookService.add (Data.findByTitle (bookTitle))
}

Now that we have two features running and interacting with the database we should also take care of cleaning up the database after each scenario.

more Before & After

Here is the additional code for test/functional/hooks/env.groovy:

import data.Data

Before () {
    ....
    Data.clearBooks ()
}

After () {
    ....
}

You can also add the the cleanup to the After hook. I have choosen to add it to the setup because if a scenario fails we can still look at its database state. Maybe it will help to diagnose the problem.

When (~‘^I view the book list$’)

Not much new here, we have to deliver the full list of books. We get it by adding a new all() action method to the controller that will return the list as JSON.

Here is the step code:

When (~"^I view the book list\$") { ->
    bookController = new BookController ()
    bookController.all ()
}

Here the new methods for the controller:

....    
class BookController {
    ....    
    def all () {
        def books = bookService.all ()
        render books as JSON
    }
}

.. and the service:

class BookService {
    ....
    List<Book> all () {
        Book.findAll ()
    }
}

Running again fails with the now common PendingException in the last step.

Then (~‘^my book list contains “([^"]*)”$’)

We check that the returned list contains the book we have pre-populated to the database in the Given step.

Here is the code:

Then (~"^my book list contains \"([^\"]*)\"\$") { String bookTitle ->
    def expected = Data.findByTitle (bookTitle)
    def all = bookController.response.json
    actual = all.getJSONObject (0)

    assert actual.id
    assert actual.title  == expected.title
    assert actual.author == expected.author
}

It is nearly the same as the previous Then check in the first feature, we just have to extract the object from the list first. We should probably refactor the assertions to a method and use that in both Then steps. We will keep that as an exercise for you ;-)

Running the test will report two passed cucumber features:

> grails test-app :cucumber --stacktrace
| Server running. Browse to http://localhost:8080/Books
| Running 2 cucumber tests...
| Completed 2 cucumber tests, 0 failed in 1612ms
| Server stopped
| Tests PASSED - view reports in target/test-reports

In case you have not noticed, we are done! ;-)

Summary

Congratulations, you have succesfully implemented two features for our example application! :-)

We have implemented a couple of cucumber steps using a grails domain class, a service, a controller and a few lines of code for the before and after hooks. We have seen were to put the feature files and the step implementations and how we can configure @tags.

This should cover the basics to get you started with cucumber and grails. :-)

Happy Cuking!

iOS: Test Driving Objective-C Retain & Release, Revisited

Michael found a few problems in the code I presented in iOS: Test Driving Objective-C Retain & Release.

Obviously he tried to use it :-)

Based upon his comments I have revisited the code as you can see it below.

First issue is, that the RetainReleaseMock will simply crash when the verify complains. RetainReleaseMock is no test case and so it is missing the method used to report an issue.

Next issue is, that it doesn’t provide very useful messages. You don’t see if it is complaining about the retain or the release count.

To fix the first issue, Michael moved the assertions back to the test method. Although this is a bit longer than the simple verify, it fixes the first issue and second issue in one step.

Ok, I agree, that’s better so I have revisited my code with some renaming (it is no longer a mock but a spy), a special hamcrest matcher and a noise reducing macro which you may use or not.

Here is the test code (without macro and with macro):

@interface CredentialsTest : GTMTestCase {
}
@end

@implementation CredentialsTest
- (void)testShouldRetainAndReleaseUserAndPassword {
  id user = [[[RetainReleaseSpy alloc] init] autorelease];
  id password = [[[RetainReleaseSpy alloc] init] autorelease];

  [[[Credentials alloc] initWithUser:user password:password] release];

  assertThat (user, isRetained (1));
  assertThat (user, isReleased (1));
  assertThat (password, isRetained (1));
  assertThat (password, isReleased (1));
}

@implementation CredentialsTest
- (void)testShouldRetainAndReleaseUserAndPassword {
  id user = RRSpy;
  id password = RRSpy;

  [[[Credentials alloc] initWithUser:user password:password] release];

  assertThat (user, isRetained (1));
  assertThat (user, isReleased (1));
  assertThat (password, isRetained (1));
  assertThat (password, isReleased (1));
}

@end

In this case I intentionally put everything in a single test and violate the “single assertion (concept) per test” rule. It has everything in one spot and I think I can handle “hidden assertion” issue (I wont see the failures of the later assertions when the first fails). ;-)

Assertion failures are reported like this:

-[CredentialsTest testShouldRetainAndReleaseUserAndPassword] : Expected release count 1, but was 0

The isRetained and isReleased calls are creating hamcrest matchers to make it a little bit easier to read.

Here is the code of the new RetainReleaseSpy, the matcher code (header and implementation) and the matcher factory methods:

// ios
#import <Foundation/Foundation.h>

// test
#import <OCHamcrest/HCBaseMatcher.h>

@interface RetainReleaseSpy : NSObject {
  int refCount;
}

@property (nonatomic, readonly) int actualRetainCount;
@property (nonatomic, readonly) int actualReleaseCount;

- (id)init;
- (id)retain;
- (void)release;

@end


id<HCMatcher> isRetained (int expectedRetainCount);
id<HCMatcher> isReleased (int expectedReleaseCount);

#define RRSpy [[[RetainReleaseMock alloc] init] autorelease]
#import "RetainReleaseSpy.h"

// test
#import <OCHamcrest/OCHamcrest.h>
#import <OCHamcrest/HCDescription.h>


@implementation RetainReleaseSpy

@synthesize actualRetainCount;
@synthesize actualReleaseCount;

- (id)init {
  if ((self = [super init])) {
    actualRetainCount = 0;
    actualReleaseCount = 0;
    refCount = 1;
  }
  return self;  
}

- (id)retain {
  ++refCount;
  ++actualRetainCount;
  return self;
}

- (void)release {
  ++actualReleaseCount;
  --refCount;
  
  if (refCount == 0) {
    [self dealloc];
  }
}

@end


@interface RetainReleaseMatcher : HCBaseMatcher {
  int expectedCount;

  SEL property;
  NSString* info;
}

- (id) initWithExpectedCount:(int)expectedCount property:(SEL)property info:(NSString*)info;

@end


@implementation RetainReleaseMatcher

- (id) initWithExpectedCount:(int)anExpectedCount property:(SEL)aProperty info:(NSString*)anInfo {
  if ((self = [super init])) {
    expectedCount = anExpectedCount;
    property = aProperty;
    info = [anInfo retain];
  }
  return self;
}

- (BOOL) matches:(id)item {
  int actualCount = (int)[item performSelector:property];
  if (actualCount != expectedCount) {
    return NO;
  }
  return YES;
}

- (void) describeTo:(id<HCDescription>)description {
  [description appendText:[NSString stringWithFormat:@"%@ count %d", info, expectedCount]];
}

- (void) describeMismatchOf:(id)item to:(id<HCDescription>)mismatchDescription {
  int actualCount = (int)[item performSelector:property];
  [mismatchDescription appendText:[NSString stringWithFormat:@"was %d", actualCount]];
}

- (void)dealloc {
  [info release];
  [super dealloc];
}   

@end


id<HCMatcher> isRetained (int expectedRetainCount)
{
  return [[[RetainReleaseMatcher alloc] initWithExpectedCount:expectedRetainCount
    property:@selector(actualRetainCount) info:@"retain"] autorelease];
}

id<HCMatcher> isReleased (int expectedReleaseCount)
{
  return [[[RetainReleaseMatcher alloc] initWithExpectedCount:expectedReleaseCount
    property:@selector(actualReleaseCount) info:@"release"] autorelease];
}

Better than before? I think so.

iOS: Test Driving Objective-C Retain & Release

Does it make sense to write unit test for retain & release for my classes?

If I strictly follow Uncle Bob’s TDD Rules then yes. I have to write some test code that will make me add the retain & release cycle for an object.

Even if I don’t strictly follow the rules it may make sense… at least it is a reminder to add the release. Which I forget from time to time.

But finally it is your decision if you test it or not. Anyway, I’d like to talk about how would I do it?

First I tried to achieve it with OCMock but that did not work at all. Fiddling with retain and release on objects out of your control is, let’s say, one on the easiest way to make your test code crash ;-)

So I created a hand made mock called RetainReleaseMock that tracks the retain & release calls.

Here is a test that demonstrates how I use it. It tests that my Credentials object properly retains & releases the user and password NSStrings passed to the initWithUser:password: method.

@interface CredentialsTest : GTMTestCase {
}
@end

@implementation CredentialsTest

- (void)testShouldRetainAndReleaseUserAndPassword {
 id user = [[[RetainReleaseMock alloc] init] autorelease];
 id password = [[[RetainReleaseMock alloc] init] autorelease];

 [[[Credentials alloc] initWithUser:user password:password] release];

 [user verify];
 [password verify];
}

@end

First I create a RetainReleaseMock for the user and password string. Then I create the Credentials object (passing the two mocks) to trigger the retain and directly release it again to trigger the release.

The final step is to verify that retain and release was called on each mock.

That’s it.

Note that you can use [autorelease] to get the mock properly deallocated.

Here is the code of the RetainReleaseMock:

// ios
#import <Foundation/Foundation.h>

@interface RetainReleaseMock : NSObject {
  int refCount;

  int expectedRetainCount;
  int expectedReleaseCount;
  int actualRetainCount;
  int actualReleaseCount;
}

- (id)init;
- (id)initWithRetain:(int)retainCount release:(int)releaseCount;

- (id)retain;
- (void)release;
- (void)verify;

@end

#import "RetainReleaseMock.h"

// test
#import <OCHamcrest/OCHamcrest.h>

@implementation RetainReleaseMock

- (id)init {
  return [self initWithRetain:1 release:1];
}

- (id)initWithRetain:(int)retainCount release:(int)releaseCount {
  if ((self = [super init])) {
    expectedRetainCount = retainCount;
    expectedReleaseCount = releaseCount;
    actualRetainCount = 0;
    actualReleaseCount = 0;
    refCount = 1;
  }
  return self;
}

- (id)retain {
  ++refCount;
  ++actualRetainCount;
  return self;
}

- (void)release {
  ++actualReleaseCount;
  --refCount;

  if (refCount == 0) {
    [self dealloc];
  }
}

- (void)verify {
  assertThatInt (actualRetainCount, equalToInt (expectedRetainCount));
  assertThatInt (actualReleaseCount, equalToInt (expectedReleaseCount));
}

@end

In case you wonder what the second init method is for:

I have added it to test retain & release on an AppDelegate where I cannot pass the dependencies directly to the AppDelegate (.. because it is created by “iOS”). The AppDelegate calls a (global) factory object that I have stubbed (with OCMock) to return the RetainReleaseMock. The factory stub retains my mock too so I have to adjust the expected retain count.

The test then looks like this:

- (void)testReleasesCoreData {
  // extra retain from appFactory stub.
  id coredata = [[[RetainReleaseMock alloc] initWithRetain:2 release:1] autorelease];

  id appFactory = Stub (AppFactory);
  [[[appFactory stub] andReturn:controller] mainViewController];
  [AppFactory setFactory:appFactory];

  [delegate application:nil didFinishLaunchingWithOptions:nil];
  [delegate release];

  [coredata verify];
  [ReadmeAppFactory reset];
}

If you have a better suggestion to handle this situation, I would like to hear it. :-)

iOS: Setting Up Advanced Unit Testing – Take Two

As Jon Reid mentioned on my “Setting Up Advanced Unit Testing” article on iOS testing, OCHamcrest now has  better support for iOS. It creates an OCHamcrest framework for IOS that includes the patches needed for iOS. I can now simply add the framework to my project and add a few additional linker options (which are the same as for OCMock) to make it work.

This should makes it a bit easier to setup hamcrest on iOS. Unfortunately there is a new complication. For iOS I now have to use

#import <OCHamcrestIOS/OCHamcrestIOS.h>

instead of

#import <OCHamcrest/OCHamcrest.h>

I don’t like that. I don’t see why it has to use a different name.

My solution is to build my own static lib for OCHamcrest. That is quite easy and it is my recommended solution.

Initially I switched OCMock to a static library because I had some issue compiling OCMock with block support. It didn’t pick up the proper iOS version and I always ended up with a lib missing block support. So finally I copied all the OCMock source files to my project and created a static library.

The OCMock source now live inside my project in a folder called “Source Xternal/OCMock”. To create a static lib I added a new “static library” target to my project (in XCode, right click Targets and Add/New Target and then select static library) with the OCMock source files below “Source Xternal/OCMock”.  Next I added the new lib (libOCMock.a) to my unit test project by adding it to the “Linked Libraries” (Get Info/Linked Libraries on the unit test target). The last step was to add “Source Xternal” to the “Header Search Paths” in the build settings.

Since the source folder is named OCMock the usual #import will work without change, i.e.

#import <OCMock/OCMock.h>

I didn’t need to modify any #import statement. (Now it also picks up the proper settings to compile it with block support :-)).

I did the same with OCHamcrest. The only difference to the procedure used for OCMock was that I flattened the sources of OCHamcrest into a single folder “Source Xternal/OCHamcrest”. Now all header files are below OCHamcrest and I do not have to touch the hamcrest #imports. (The original source layout uses subfolder to group the files).

This also removed the unwanted “IOS” postfix. So I can simply use the common #import <OCHamcrest/OCHamcrest.h>.

Finally I also moved Google Toolbox for Mac stuff to my “Source Xternal” folder into “Source Xternal/GoogleToolbox” and created another static lib for it.

So my source layout now looks like this:

Source
...
Source Test
...
Source Xternal
Source Xternal/OCMock
Source Xternal/OCHamcrest
Source Xternal/GoogleToolbox

That’s straight forward and also improves the build on my continuous integration server as it now builds all my dependencies as part of my project with the same compiler settings.

Another nice side effect is, that the source of the libraries is not far away and I can easily look into it. From time to time I get strange error messages when compiling and sometimes it helps to take a look into the source of OCHamcrest.

I’m already using this layout for a while now and its definitely my recommendation!

Happy Testing!

iOS: Running Cucumber/Frank with Code Coverage in Jenkins/Hudson

In the previous article iOS: Setting Up Acceptance Testing I introduced Frank. The next and last step will be to run Cucumber and Frank headless from Jenkins/Hudson.

Frank is still new, and it uses apple script to remote control XCode to run the “frankified” app in the simulator. That’s fine for running the cucumber scenarios manually but it is not so good for running them from Jenkins/Hudson.

To run the cucumber tests from Jenkins/Hudson I need it to run headless, i.e. it should not open any window while running the tests. I don’t want Jenkins/Hudson to interfere with whatever I’m doing right know. It should run invisible in the background.

Luckily it is possible to run an iOS app headless simply by running the app directly like this:

<path to app>/MyApp.app/MyApp -RegisterForSystemEvents

By default I would use a cucumber step like the following to run my app from cucumber:

Given /^the app has just started$/ do
  launch_app_in_simulator
end

with launch_app_in_simulator beeing the frank function that uses apple script  to run the app by controlling XCode.

I have changed that to:

Given /^the app has just started$/ do
  launch_app_headless
end

with launch_app_headless looking like this:

def launch_app_headless
  @apppid = fork do
    exec(APP, "-RegisterForSystemEvents")
  end

  wait_for_frank_to_come_up
end

APP is a string variable that contains the path to the frankified app. It is currently set hardcoded in my cucumber Before block:

require 'fileutils'

##
## adjust app, sdk dir and app dir
##
APP    = "build/Debug-iphonesimulator/Readme Frankified.app/Readme Frankified"
APPDIR = "build/Readme.build/Debug-iphonesimulator/Readme Frankified.build"
SDKDIR = "/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.2.sdk"

USERDIR = "#{APPDIR}/iPhone Simulator User Dir"
PREFDIR = "#{USERDIR}/Library/Preferences"

ACCESIBILITY_PLIST   = "com.apple.Accessibility.plist"
ACCESIBILITY_CONTENT = <<PLIST
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
 <key>ApplicationAccessibilityEnabled</key>
 <true/>
</dict>
</plist>
PLIST

Before do
  # check that pwd contains the "build" dir as we are creating
  # items relative to it.
  Dir["build"].length.should == 1

  # make sure we do start with a clean environment
  FileUtils.remove_dir("#{USERDIR}",true)

  pwd     = "#{Dir.pwd}"
  prefdir = "#{pwd}/#{PREFDIR}"
  FileUtils.mkdir_p prefdir

  File.open("#{PREFDIR}/#{ACCESIBILITY_PLIST}", 'w') do |f|
    f <<ACCESIBILITY_CONTENT
  end

  ENV['SDKROOT']               = "#{SDKDIR}"
  ENV['DYLD_ROOT_PATH']        = "#{SDKDIR}"
  ENV['IPHONE_SIMULATOR_ROOT'] = "#{SDKDIR}"
  ENV['TEMP_FILES_DIR']        = "#{pwd}/#{APPDIR}"
  ENV['CFFIXED_USER_HOME']     = "#{pwd}/#{USERDIR}"
end

The accessibility stuff is required to create a configuration file for the app that enables accessibility. Without it Frank will not find the accessibilityLabels which are used to identify the ui elements.

Now I’m able to run the cucumber test for my app headless in Jenkins/Hudson. I created a second build configuration for the acceptance tests. It uses nearly the same setup as for unit testing. I just have added an additional “Execute Shell” step after building the frankified app that runs cucumber:

cd trunk
cucumber -f junit --o ../cucumber --tags ~@ignore features.frank

Using the -f junit parameter cucmber will create an junit xml file so Jenkins/Hudson can display the test results in the same ways as the unit tests. features.frank is the folder were I keep my cucumber scenarios.

Code coverage is set up the same way as for the unit tests (see iOS: Running Unit Tests with Code Coverage in Jenkins/Hudson). Now here is also the last complication. Frank currently doesn’t shut down the app and the result is that the app doesn’t write the coverage information.

To fix this I simply added a super simple exit command (only calling exit(0)) to Franks embedded http server and a ruby function to call it. I have forked the Frank git repository here http://github.com/hauner/Frank.

My cucumber After block now shuts down the app so it does write the code coverage info:

After do
  frankly_exit
end

with franky_exit beeing:

def frankly_exit
  get_to_uispec_server('exit')
  # calling exit in the app will not return any response
  # so we simply catch the error caused by exiting.
  rescue EOFError
end

Now the app will shut down with a clean exit, write the code coverage information and the code coverage step in Jenkins/Hudson will properly display the code coverage for my acceptance tests.

Done, finally :-)

Summary: I have reached my goal to have a test environment to create unit tests (using Google Toolbox for MacOCMock & OCHamcrest) and acceptance tests (using Cucumber & Frank) of an iOS  app. Both test categories are running in Jenkins/Hudson as two separate builds and  both builds provide code coverage information. It took a little bit of work, but it is running now.

That is all I have to say regarding testing an iOS app for now! :-)

This is the 6th article of iOS: Setting Up a Test Environment is a Mess.