Tagging Cucumber Tests

Cucumber allows tests to be tagged at multiple levels which can help with finding scenarios grouped across scenarios or feature files. It also allows tests to be run in smaller groups. If used in conjunction with a ticketing system, it can direct a reader to the origin of the the scenario by utilising the unique ticket identifier.

Applying tags is as easy as placing the tag at feature or scenario level using the symbol @ to identify it as a tag. Multiple tags can be applied and simple logic can be used when selecting a tag.

Tagging a Scenario

Adding a tag is easy. You can tag at feature, rule or scenario level, giving fine grained control over the code. Just add the tag with the @ symbol. For example, here I've tagged a scenario with the ticket number FTF-100.

  @FTF-100
  Scenario: Should print "Hello Dave!" to the screen when the program is run passing name=Dave
    Given the helloworld application
    When the application is run with argument Dave
    Then "Hello Dave!" is output to the console 

To restrict Cucumber to only run this scenario we can edit the CucumberTest.kt class and add another option as follows for JUnit 4:

@CucumberOptions(features = ["src/test/resources"], tags = "@FTF-100")

When using JUnit Platform with the @Suite annotation, I've found that there are two approaches that work. Either specify the Cucumber option like this:

@ConfigurationParameter(key = io.cucumber.core.options.Constants.FILTER_TAGS_PROPERTY_NAME, value = "@FTF-100")

Or use a JUnit’s tagging annotations as follows – note the @ symbol is not required or this case:

@IncludeTags("FTF-100")

These approaches will restrict JUnit to just running the tagged test scenario, so it acts a little more like an individually run unit test when clicking on the icon beside the test within IntelliJ. This might be useful when testing a bug as it will allow the tests to run faster.

Specifying Tags at the Command Line

There are a couple of issue with having to place the tags into the code. It lacks a lot of flexibility as you have to edit and save the file before the changes are applied. This can be especially annoying if a tag is mistyped altogether, as if no scenario is found to match the tag expression, then no scenarios will be run.

Another issue with this approach is that a developer, happy that their new feature is working, commits their code having forgotten to remove the tag from CucumberTest class. That means that all Cucumber tests might be filtered if no one spots the issue. I'm afraid I've worked on a project where it was missed by both the developer and the team at code review time. Only a subset of tests were run for several weeks before someone spotted the issue.

Gradle does not have a means to specify the tags you want to include to JUnit 5 when running from the command line with simple runtime option – at least not yet or that I can find with a simple Google search. It is possible to specify either includTags or excludeTags directly in the build.gradle.kts file like this:

tasks.test {
    useJUnitPlatform{
        includeTags("FTF-100")
        excludeTags("slow")
    }
    systemProperty("cucumber.junit-platform.naming-strategy", "long")
}

However, this still requires code changes that I am unwilling to encourage. The best workaround that I have found is to take the parameters from the command line as system properties and then use them if they are set. Here's what the code in gradle.build.kts might look like:

tasks.test {
    val includeTags = System.getProperty("include.tags")
    val excludeTags = System.getProperty("exclude.tags")
    useJUnitPlatform{
        includeTags?.let {
            includeTags(includeTags)
        }
        excludeTags?.let {
            excludeTags(excludeTags)
        }
    }
    systemProperty("cucumber.junit-platform.naming-strategy", "long")
}

Now when we build and run tests from the command line with ./gradlew clean test -Dinclude.tags="FTF-100" or via IntelliJ by adding a VM option to the run configuation, only the tagged tests will run (or not run when using excludes). Note that to add VM options use the “Modify options” dropdown).

Using Tags

While I argue that behavioural tests should offer sparser coverage than unit tests, it is possible to build up quite a collection of them fast, especially with engaged business users who can supply new scenarios every time functionality is added. Running all the tests each time you build the project may start creating some delays to the development process through time waiting for tests to run.

Some teams use tags to flag specific scenarios using labels, such as @fast or @smoke, to identify tests that are important and quick to run. These can be run automatically every time a build is done by using configuration in build.gradle.kts. For slower tests and for a large volume of tests, it may be more advisable to only run all the tests when building on a CI server. This can be done by overriding the default behaviour via parameters as shown above.

Another use for tags is to label new features or scenarios that help to resolve bugs with the identifier from the ticketing system. This will tie the work back to specific user stories or bug tickets and help to create additional context. During development, we could limit the Cucumber tests to just the tagged tests for that feature or bug, speeding the process up a little.