# Testing Guide

version 2.4.16

## 1. Introduction

The Groovy programming language comes with great support for writing tests. In addition to the language features and test integration with state-of-the-art testing libraries and frameworks, the Groovy ecosystem has born a rich set of testing libraries and frameworks.

This chapter will start with language specific testing features and continue with a closer look at JUnit integration, Spock for specifications, and Geb for functional tests. Finally, we’ll do an overview of other testing libraries known to be working with Groovy.

## 2. Language Features

Besides integrated support for JUnit, the Groovy programming language comes with features that have proven to be very valuable for test-driven development. This section gives insight on them.

### 2.1. Power Assertions

Writing tests means formulating assumptions by using assertions. In Java this can be done by using the `assert` keyword that has been added in J2SE 1.4. In Java, `assert` statements can be enabled via the JVM parameters `-ea` (or `-enableassertions`) and `-da` (or `-disableassertions`). Assertion statements in Java are disabled by default.

Groovy comes with a rather powerful variant of `assert` also known as power assertion statement. Groovy’s power `assert` differs from the Java version in its output given the boolean expression validates to `false`:

    def x = 1
    assert x == 2

    // Output:             (1)
    //
    // Assertion failed:
    // assert x == 2
    //        | |
    //        1 false

1 | This section shows the std-err output

The `java.lang.AssertionError` that is thrown whenever the assertion can not be validated successfully, contains an extended version of the original exception message. The power assertion output shows evaluation results from the outer to the inner expression.

The power assertion statements true power unleashes in complex Boolean statements, or statements with collections or other `toString`-enabled classes:

    def x = [1,2,3,4,5]
    assert (x << 6) == [6,7,8,9,10]

    // Output:
    //
    // Assertion failed:
    // assert (x << 6) == [6,7,8,9,10]
    //         | |     |
    //         | |     false
    //         | [1, 2, 3, 4, 5, 6]
    //         [1, 2, 3, 4, 5, 6]

Another important difference from Java is that in Groovy assertions are enabled by default. It has been a language design decision to remove the possibility to deactivate assertions. Or, as Bertrand Meyer stated, `it makes no sense to take off your swim ring if you put your feet into real water`.

One thing to be aware of are methods with side-effects inside Boolean expressions in power assertion statements. As the internal error message construction mechanism does only store references to instances under target, it happens that the error message text is invalid at rendering time in case of side-effecting methods involved:

    assert [[1,2,3,3,3,3,4]].first().unique() == [1,2,3]

    // Output:
    //
    // Assertion failed:
    // assert [[1,2,3,3,3,3,4]].first().unique() == [1,2,3]
    //                          |       |        |
    //                          |       |        false
    //                          |       [1, 2, 3, 4]
    //                          [1, 2, 3, 4]           (1)

1 | The error message shows the actual state of the collection, not the state before the `unique` method was applied

| If you choose to provide a custom assertion error message this can be done by using the Java syntax `assert expression1 : expression2` where `expression1` is the Boolean expression and `expression2` is the custom error message. Be aware though that this will disable the power assert and will fully fallback to custom error messages on assertion errors.

### 2.2. Mocking and Stubbing

Groovy has excellent built-in support for a range of mocking and stubbing alternatives. When using Java, dynamic mocking frameworks are very popular. A key reason for this is that it is hard work creating custom hand-crafted mocks using Java. Such frameworks can be used easily with Groovy if you choose but creating custom mocks is much easier in Groovy. You can often get away with simple maps or closures to build your custom mocks.

The following sections show ways to create mocks and stubs with Groovy language features only.

#### 2.2.1. Map Coercion

By using maps or expandos, we can incorporate desired behaviour of a collaborator very easily as shown here:

    class TranslationService {
        String convert(String key) {
            return "test"
        }
    }

    def service = [convert: { String key -> 'some text' }] as TranslationService
    assert 'some text' == service.convert('key.text')

The `as` operator can be used to coerce a map to a particular class. The given map keys are interpreted as method names and the values, being `groovy.lang.Closure` blocks, are interpreted as method code blocks.

| Be aware that map coercion can get into the way if you deal with custom `java.util.Map` descendant classes in combination with the `as` operator. The map coercion mechanism is targeted directly at certain collection classes, it doesn’t take custom classes into account.

#### 2.2.2. Closure Coercion

The 'as' operator can be used with closures in a neat way which is great for developer testing in simple scenarios. We haven’t found this technique to be so powerful that we want to do away with dynamic mocking, but it can be very useful in simple cases none-the-less.

Classes or interfaces holding a single method, including SAM (single abstract method) classes, can be used to coerce a closure block to be an object of the given type. Be aware that for doing this, Groovy internally create a proxy object descending for the given class. So the object will not be a direct instance of the given class. This important if, for example, the generated proxy object’s meta-class is altered afterwards.

Let’s have an example on coercing a closure to be of a specific type:

    def service = { String key -> 'some text' } as TranslationService
    assert 'some text' == service.convert('key.text')

Groovy supports a feature called implicit SAM coercion. This means that the `as` operator is not necessary in situations where the runtime can infer the target SAM type. This type of coercion might be useful in tests to mock entire SAM classes:

    abstract class BaseService {
        abstract void doSomething()
    }

    BaseService service = { -> println 'doing something' }
    service.doSomething()

#### 2.2.3. MockFor and StubFor

The Groovy mocking and stubbing classes can be found in the `groovy.mock.interceptor` package.

The `MockFor` class supports (typically unit) testing of classes in isolation by allowing a strictly ordered expectation of the behavior of collaborators to be defined. A typical test scenario involves a class under test and one or more collaborators. In such a scenario it is often desirable to just test the business logic of the class under test. One strategy for doing that is to replace the collaborator instances with simplified mock objects to help isolate out the logic in the test target. MockFor allows such mocks to be created using meta-programming. The desired behavior of collaborators is defined as a behavior specification. The behavior is enforced and checked automatically.

Let’s assume our target classes looked like this:

    class Person {
        String first, last
    }

    class Family {
        Person father, mother
        def nameOfMother() { "$mother.first $mother.last" }
    }

With `MockFor`, a mock expectation is always sequence dependent and its use automatically ends with a call to `verify`:

    def mock = new MockFor(Person)      (1)
    mock.demand.getFirst{ 'dummy' }
    mock.demand.getLast{ 'name' }
    mock.use {                          (2)
        def mary = new Person(first:'Mary', last:'Smith')
        def f = new Family(mother:mary)
        assert f.nameOfMother() == 'dummy name'
    }
    mock.expect.verify()                (3)

1 | a new mock is created by a new instance of `MockFor`
2 | a `Closure` is passed to `use` which enables the mocking functionality
3 | a call to `verify` checks whether the sequence and number of method calls is as expected

The `StubFor` class supports (typically unit) testing of classes in isolation by allowing a loosely-ordered expectation of the behavior of collaborators to be defined. A typical test scenario involves a class under test and one or more collaborators. In such a scenario it is often desirable to just test the business logic of the CUT. One strategy for doing that is to replace the collaborator instances with simplified stub objects to help isolate out the logic in the target class. `StubFor` allows such stubs to be created using meta-programming. The desired behavior of collaborators is defined as a behavior specification.

In contrast to `MockFor` the stub expectation checked with `verify` is sequence independent and its use is optional:

    def stub = new StubFor(Person)      (1)
    stub.demand.with {                  (2)
        getLast{ 'name' }
        getFirst{ 'dummy' }
    }
    stub.use {                          (3)
        def john = new Person(first:'John', last:'Smith')
        def f = new Family(father:john)
        assert f.father.first == 'dummy'
        assert f.father.last == 'name'
    }
    stub.expect.verify()                (4)

1 | a new stub is created by a new instance of `StubFor`
2 | the `with` method is used for delegating all calls inside the closure to the `StubFor` instance
3 | a `Closure` is passed to `use` which enables the stubbing functionality
4 | a call to `verify` (optional) checks whether the number of method calls is as expected

`MockFor` and `StubFor` can not be used to test statically compiled classes e.g for Java classes or Groovy classes that make use of `@CompileStatic`. To stub and/or mock these classes you can use Spock or one of the Java mocking libraries.

#### 2.2.4. Expando Meta-Class (EMC)

Groovy includes a special `MetaClass` the so-called `ExpandoMetaClass` (EMC). It allows to dynamically add methods, constructors, properties and static methods using a neat closure syntax.

Every `java.lang.Class` is supplied with a special `metaClass` property that will give a reference to an `ExpandoMetaClass` instance. The expando meta-class is not restricted to custom classes, it can be used for JDK classes like for example `java.lang.String` as well:

    String.metaClass.swapCase = {->
        def sb = new StringBuffer()
        delegate.each {
            sb << (Character.isUpperCase(it as char) ? Character.toLowerCase(it as char) :
                Character.toUpperCase(it as char))
        }
        sb.toString()
    }

    def s = "heLLo, worLD!"
    assert s.swapCase() == 'HEllO, WORld!'

The `ExpandoMetaClass` is a rather good candidate for mocking functionality as it allows for more advanced stuff like mocking static methods

    class Book {
        String title
    }

    Book.metaClass.static.create << { String title -> new Book(title:title) }

    def b = Book.create("The Stand")
    assert b.title == 'The Stand'

or even constructors

    Book.metaClass.constructor << { String title -> new Book(title:title) }

    def b = new Book("The Stand")
    assert b.title == 'The Stand'

| Mocking constructors might seem like a hack that’s better not even to be considered but even there might be valid use cases. An example can be found in Grails where domain class constructors are added at run-time with the help of `ExpandoMetaClass`. This lets the domain object register itself in the Spring application context and allows for injection of services or other beans controlled by the dependency-injection container.

If you want to change the `metaClass` property on a per test method level you need to remove the changes that were done to the meta-class, otherwise those changes would be persistent across test method calls. Changes are removed by replacing the meta-class in the `GroovyMetaClassRegistry`:

    GroovySystem.metaClassRegistry.setMetaClass(java.lang.String, null)

Another alternative is to register a `MetaClassRegistryChangeEventListener`, track the changed classes and remove the changes in the cleanup method of your chosen testing runtime. A good example can be found in the Grails web development framework.

Besides using the `ExpandoMetaClass` on a class-level, there is also support for using the meta-class on a per-object level:

    def b = new Book(title: "The Stand")
    b.metaClass.getTitle {-> 'My Title' }

    assert b.title == 'My Title'

In this case the meta-class change is related to the instance only. Depending on the test scenario this might be a better fit than the global meta-class change.

### 2.3. GDK Methods

The following section gives a brief overview on GDK methods that can be leveraged in test case scenarios, for example for test data generation.

#### 2.3.1. Iterable#combinations

The `combinations` method that is added on `java.lang.Iterable` compliant classes can be used to get a list of combinations from a list containing two or more sub-lists:

    void testCombinations() {
        def combinations = [[2, 3],[4, 5, 6]].combinations()
        assert combinations == [[2, 4], [3, 4], [2, 5], [3, 5], [2, 6], [3, 6]]
    }

The method could be used in test case scenarios to generate all possible argument combinations for a specific method call.

#### 2.3.2. Iterable#eachCombination

The `eachCombination` method that is added on `java.lang.Iterable` can be used to apply a function (or in this case a `groovy.lang.Closure`) to each if the combinations that has been built by the `combinations` method:

`eachCombination` is a GDK method that is added to all classes conforming to the `java.lang.Iterable` interface. It applies a function on each combination of the input lists:

    void testEachCombination() {
        [[2, 3],[4, 5, 6]].eachCombination { println it[0] + it[1] }
    }

The method could be used in the testing context to call methods with each of the generated combinations.

### 2.4. Tool Support

#### 2.4.1. Test Code Coverage

Code coverage is a useful measure of the effectiveness of (unit) tests. A program with high code coverage has a lower chance to hold critical bugs than a program with no or low coverage. To get code coverage metrics, the generated byte-code usually needs to be instrumented before the tests are executed. One tool with Groovy support for this task is Cobertura.

Various frameworks and build tools come with Cobertura integration. For Grails, there is the code coverage plugin based on Cobertura, for Gradle there is the gradle-cobertura plugin, to name only two of them.

The following code listing shows an example on how to enable Cobertura test coverage reports in a Gradle build script from a Groovy project:

    def pluginVersion = '<plugin version>'
    def groovyVersion = '<groovy version>'
    def junitVersion = '<junit version>'

    buildscript {
        repositories {
            mavenCentral()
        }
        dependencies {
            classpath 'com.eriwen:gradle-cobertura-plugin:${pluginVersion}'
        }
    }

    apply plugin: 'groovy'
    apply plugin: 'cobertura'

    repositories {
        mavenCentral()
    }

    dependencies {
        compile "org.codehaus.groovy:groovy-all:${groovyVersion}"
        testCompile "junit:junit:${junitVersion}"
    }

    cobertura {
        format = 'html'
        includes = ['**/*.java', '**/*.groovy']
        excludes = ['com/thirdparty/**/*.*']
    }

Several output formats can be chosen for Cobertura coverage reports and test code coverage reports can be added to continuous integration build tasks.

## 3. Unit Tests with JUnit 3 and 4

Groovy simplifies JUnit testing, making it more Groovy. In the following sections we will have a closer look at JUnit 3/4 Groovy integration.

### 3.1. JUnit 3

Maybe one of the most prominent Groovy classes supporting JUnit 3 tests is the `GroovyTestCase` class. Being derived from `junit.framework.TestCase` it offers a bunch of additional methods that make testing in Groovy a breeze.

| Although `GroovyTestCase` inherits from `TestCase` doesn’t mean you can’t use JUnit 4 features in your project. In fact, the most recent Groovy versions come with a bundled JUnit 4 and that comes with a backwards compatible `TestCase` implementation. There have been some discussion on the Groovy mailing-list on whether to use `GroovyTestCase` or JUnit 4 with the result that it is mostly a matter of taste, but with `GroovyTestCase` you get a bunch of methods for free that make certain types of tests easier to write.

In this section, we will have a look at some of the methods provided by `GroovyTestCase`. A full list of these can be found in the JavaDoc documentation for groovy.util.GroovyTestCase , don’t forget it is inherited from `junit.framework.TestCase` which inherits all the `assert*` methods.

#### 3.1.1. Assertion Methods

`GroovyTestCase` is inherited from `junit.framework.TestCase` therefore it inherits a large number of assertion methods being available to be called in every test method:

    class MyTestCase extends GroovyTestCase {

        void testAssertions() {
            assertTrue(1 == 1)
            assertEquals("test", "test")

            def x = "42"
            assertNotNull "x must not be null", x
            assertNull null

            assertSame x, x
        }

    }

As can be seen above, in contrast to Java it is possible to leave out the parenthesis in most situations which leads to even more readability of JUnit assertion method call expressions.

An interesting assertion method that is added by `GroovyTestCase` is `assertScript`. It ensures that the given Groovy code string succeeds without any exception:

    void testScriptAssertions() {
        assertScript '''
            def x = 1
            def y = 2

            assert x + y == 3
        '''
    }

#### 3.1.2. shouldFail Methods

`shouldFail` can be used to check whether the given code block fails or not. In case it fails, the assertion does hold, otherwise the assertion fails:

    void testInvalidIndexAccess1() {
        def numbers = [1,2,3,4]
        shouldFail {
            numbers.get(4)
        }
    }

The example above uses the basic `shouldFail` method interface that takes a `groovy.lang.Closure` as a single argument. The `Closure` instance holds the code that is supposed to be breaking during run-time.

If we wanted to assert `shouldFail` on a specific `java.lang.Exception` type we could have done so by using the `shouldFail` implementation that takes the `Exception` class as first argument and the `Closure` as second argument:

    void testInvalidIndexAccess2() {
        def numbers = [1,2,3,4]
        shouldFail IndexOutOfBoundsException, {
            numbers.get(4)
        }
    }

If anything other than `IndexOutOfBoundsException` (or a descendant class of it) is thrown, the test case will fail.

A pretty nice feature of `shouldFail` hasn’t been visible so far: it returns the exception message. This is really useful if you want to assert on the exception error message:

    void testInvalidIndexAccess3() {
        def numbers = [1,2,3,4]
        def msg = shouldFail IndexOutOfBoundsException, {
            numbers.get(4)
        }
        assert msg.contains('Index: 4, Size: 4')
    }

#### 3.1.3. notYetImplemented Method

The `notYetImplemented` method has been greatly influenced by HtmlUnit. It allows to write a test method but mark it as not yet implemented. As long as the test method fails and is marked with `notYetImplemented` the test goes green:

    void testNotYetImplemented1() {
        if (notYetImplemented()) return   (1)

        assert 1 == 2                     (2)
    }

1 | a call to `notYetImplemented` is necessary for `GroovyTestCase` to get the current method stack.
2 | as long as the test evaluates to `false` the test execution will be successful.

An alternative to the `notYetImplemented` method is the `@NotYetImplemented` annotation. It allows for annotating a method as not yet implemented, with the exact same behavior as `GroovyTestCase#notYetImplemented` but without the need for the `notYetImplemented` method call:

    @NotYetImplemented
    void testNotYetImplemented2() {
        assert 1 == 2
    }

### 3.2. JUnit 4

Groovy can be used to write JUnit 4 test cases without any restrictions. The `groovy.test.GroovyAssert` holds various static methods that can be used as replacement for the `GroovyTestCase` methods in JUnit 4 tests:

    import org.junit.Test

    import static groovy.test.GroovyAssert.shouldFail

    class JUnit4ExampleTests {

        @Test
        void indexOutOfBoundsAccess() {
            def numbers = [1,2,3,4]
            shouldFail {
                numbers.get(4)
            }
        }

    }

As can be seen in the example above, the static methods found in `GroovyAssert` are imported at the beginning of the class definition thus `shouldFail` can be used the same way it can be used in a `GroovyTestCase`.

| `groovy.test.GroovyAssert` descends from `org.junit.Assert` that means it inherits all JUnit assertion methods. However, with the introduction of the power assertion statement, it turned out to be good practice to rely on assertion statements instead of using the JUnit assertion methods with the improved message being the main reason.

It is worth mentioning that `GroovyAssert.shouldFail` is not absolutely identical to `GroovyTestCase.shouldFail`. While `GroovyTestCase.shouldFail` returns the exception message, `GroovyAssert.shouldFail` returns the exception itself. It takes a few more keystrokes to get the message, but in return you can access other properties and methods of the exception:

    @Test
    void shouldFailReturn() {
        def e = shouldFail {
            throw new RuntimeException('foo',
                                       new RuntimeException('bar'))
        }
        assert e instanceof RuntimeException
        assert e.message == 'foo'
        assert e.cause.message == 'bar'
    }

## 4. Testing with Spock

Spock is a testing and specification framework for Java and Groovy applications. What makes it stand out from the crowd is its beautiful and highly expressive specification DSL. In practice, Spock specifications are written as Groovy classes. Although written in Groovy they can be used to test Java classes. Spock can be used for unit, integration or BDD (behavior-driven-development) testing, it doesn’t put itself into a specific category of testing frameworks or libraries.

| Beside these awesome features Spock is a good example on how to leverage advanced Groovy programming language features in third party libraries, for example, by using Groovy AST transformations.

| This section should not serve as detailed guide on how to use Spock, it should rather give an impression what Spock is about and how it can be leveraged for unit, integration, functional or any other type of testing.

The next section we will have an first look at the anatomy of a Spock specification. It should give a pretty good feeling on what Spock is up to.

### 4.1. Specifications

Spock lets you write specifications that describe features (properties, aspects) exhibited by a system of interest. The "system" can be anything between a single class and an entire application, a more advanced term for it is system under specification. The feature description starts from a specific snapshot of the system and its collaborators, this snapshot is called the feature’s fixture.

Spock specification classes are derived from `spock.lang.Specification`. A concrete specification class might consist of fields, fixture methods, features methods and helper methods.

Let’s have a look at a simple specification with a single feature method for an imaginary `Stack` class:

    class StackSpec extends Specification {

        def "adding an element leads to size increase"() {  (1)
            setup: "a new stack instance is created"        (2)
                def stack = new Stack()

            when:                                           (3)
                stack.push 42

            then:                                           (4)
                stack.size() == 1
        }
    }

1 | Feature method, is by convention named with a String literal.
2 | Setup block, here is where any setup work for this feature needs to be done.
3 | When block describes a stimulus, a certain action under target by this feature specification.
4 | Then block any expressions that can be used to validate the result of the code that was triggered by the when block.

Spock feature specifications are defined as methods inside a `spock.lang.Specification` class. They describe the feature by using a String literal instead of a method name.

A feature method holds multiple blocks, in our example we used `setup`, `when` and `then`. The `setup` block is special in that it is optional and allows to configure local variables visible inside the feature method. The `when` block defines the stimulus and is a companion of the `then` block which describes the response to the stimulus.

Note that the `setup` method in the `StackSpec` above additionally has a description String. Description Strings are optional and can be added after block labels (like `setup`, `when`, `then`).

### 4.2. More Spock

Spock provides much more features like data tables or advanced mocking capabilities. Feel free to consult the Spock GitHub page for more documentation and download information.

## 5. Functional Tests with Geb

Geb is a functional web testing and scraper library that integrates with JUnit and Spock. It is based upon the Selenium web drivers and, like Spock, provides a Groovy DSL to write functional tests for web applications.

Geb has great features that make it a good fit for a functional testing library:

- DOM access via a JQuery-like `$` function

- implements the page pattern

- support for modularization of certain web components (e.g. menu-bars, etc.) with modules

- integration with JavaScript via the JS variable

| This section should not serve as detailed guide on how to use Geb, it should rather give an impression what Geb is about and how it can be leveraged functional testing.

The next section will give an example on how Geb can be used to write a functional test for a simple web page with a single search field.

### 5.1. A Geb Script

Although Geb can be used standalone in a Groovy script, in many scenarios it’s used in combination with other testing frameworks. Geb comes with various base classes that can be used in JUnit 3, 4, TestNG or Spock tests. The base classes are part of additional Geb modules that need to be added as a dependency.

For example, the following `@Grab` dependencies have to be used to run Geb with the Selenium Firefox driver in JUnit4 tests. The module that is needed for JUnit 3/4 support is `geb-junit`:

    @Grapes([
        @Grab("org.gebish:geb-core:0.9.2"),
        @Grab("org.gebish:geb-junit:0.9.2"),
        @Grab("org.seleniumhq.selenium:selenium-firefox-driver:2.26.0"),
        @Grab("org.seleniumhq.selenium:selenium-support:2.26.0")
    ])

The central class in Geb is the `geb.Browser` class. As its name implies it is used to browse pages and access DOM elements:

    def browser = new Browser(driver: new FirefoxDriver(), baseUrl: 'http://myhost:8080/myapp')  (1)
    browser.drive {
        go "/login"                        (2)

        $("#username").text = 'John'       (3)
        $("#password").text = 'Doe'

        $("#loginButton").click()

        assert title == "My Application - Dashboard"
    }

1 | A new `Browser` instance is created. In this case it uses the Selenium `FirefoxDriver` and sets the `baseUrl`.
2 | `go` is used to navigate to an URL or relative URI
3 | `$` together with CSS selectors is used to access the `username` and `password` DOM fields.

The `Browser` class comes with a `drive` method that delegates all method/property calls to the current `browser` instance. The `Browser` configuration must not be done inline, it can also be externalized in a `GebConfig.groovy` configuration file for example. In practice, the usage of the `Browser` class is mostly hidden by Geb test base classes. They delegate all missing properties and method calls to the current `browser` instance that exists in the background:

    class SearchTests extends geb.junit4.GebTest {

        @Test
        void executeSeach() {
            go 'http://somehost/mayapp/search'              (1)
            $('#searchField').text = 'John Doe'             (2)
            $('#searchButton').click()                      (3)

            assert $('.searchResult a').first().text() == 'Mr. John Doe' (4)
        }
    }

1 | `Browser#go` takes a relative or absolute link and calls the page.
2 | `Browser#$` is used to access DOM content. Any CSS selectors supported by the underlying Selenium drivers are allowed
3 | `click` is used to click a button.
4 | `$` is used to get the first link out of the `searchResult` block

The example above shows a simple Geb web test with the JUnit 4 base class `geb.junit4.GebTest`. Note that in this case the `Browser` configuration is externalized. `GebTest` delegates methods like `go` and `$` to the underlying `browser` instance.

### 5.2. More Geb

In the previous section we only scratched the surface of the available Geb features. More information on Geb can be found at the project homepage.