Generating Cucumber Reports

Having behavioural tests that describe what our application does is a great way to have a living documentation, one that must be changed whenever the code changes or else the tests begin to fail. Unlike regular documentation, which can become stale or out of date all too easily, living documentation will always match what the current version of the code does.

Once you have a collection of Behavioural tests written and are adding to them as new features are developed, and your Continuous Integration (CI) pipeline is running the tests after each commit to make sure there are no failures, it is time to start publishing the results. Because the features are written in a plain, domain specific language that is shared between the developers and the business users, there should be a much wider audience to the test results than just developers and quality assurance.

Cucumber can generate reports formatted in HTML or also produce JSON and XML versions of the data that can be consumed by another process. It is beyond the scope of this article to discuss how the raw data in the latter formats might be used, or to discuss how a CI pipeline might be configured to publish the results. I'll just concentrate on showing how to generate the different output and placing the results into files within the build folder.

Cucumber manages output using plugins. You can add plugins from a number of places such as configuring with properties or passing a parameter on the command line when running the tests. This configurability is useful because different results might be required when running at different times. The developer that wants to check the tests pass might prefer textual output, but a CI server needs to have JSON output so that it can interpret the results.

Just like with getting the Cucumber tests to run with JUnit, there are some differences to how this needs to be configured between JUnit 4 and 5. I’ll cover both here showing that they are both related closely.

Cucumber Options for JUnit 4

Let's start with the simplest format which is textual output to the console. This can be achieved by adding the pretty plugin using the @CucumberOptions annotation. Here's what the code might look like when the option is added to CucumberTest.kt:

@RunWith(Cucumber::class)
@CucumberOptions(
    features = ["src/test/resources"], 
    plugin = ["pretty"]
)
class CucumberTest

The plugin field in @CucumberOptions takes a list of strings that identify the plugin. When the test is run within IntelliJ the console output will display something like the following. Each matched scenario is coloured green and has a comment to the right which points to the step definition that it was bound to. This can be useful for debugging a test that won't work as expected

Cucumber can generate reports in HTML format which can is probably the easiest way to disseminate the results using the html plugin. This time it requires a filename to be specified, and can be added with the following in addition to the pretty output. Make sure that you generate the output into the build folder so that it is ignored from being checked into source control.

@CucumberOptions(
    features = ["src/test/resources"],
    plugin = ["pretty", "html:build/cucumber/cucumber.html"]
)

It should be possible to configure a CI server to copy or publish this file to an internal wiki or file share so that it is available to all users in the organisation. This could be a great way to share what behaviours the current state of the application has based on the tests that are passing.

Configuration Parameters for JUnit 5

If you are using JUnit 5 with the @Suite annotation to identify the test class, the plugins need to be specified in a slightly different way. Rather than using the @CucumberOptions annotation, they need to be passed to the Cucumber engine as @ConfigurationParameter annotations. Here is how to turn on the pretty plugin:

@Suite
@IncludeEngines("cucumber")
@SelectClasspathResource("test-cucumber.feature")
@ConfigurationParameter(key = GLUE_PROPERTY_NAME, value = "com.failedtofunction.cucumbersample")
@ConfigurationParameter(key = PLUGIN_PROPERTY_NAME, value = "pretty")
class CucumberTest

Multiple reporting options can be set by separating each one with a comma. For example, to turn on the html and json plugins and also to print the pretty output, this can be configured with the following:

@ConfigurationParameter(key = PLUGIN_PROPERTY_NAME, value = "pretty,html:build/reports/cucumber/report.html,json:build/reports/cucumber/report.json")

Other Plugins

I'd like to mention a few other plugins that Cucumber ships with. The first is the junit plugin which can be used to improve the integration with CI servers. Because we defined a single test class to run the Cucumber scenarios, it means that many CI servers will think that there is just a single test. That makes it awkward to check which test has failed or get decent output. The junit plugin provides output in XML format compatible with JUnit tests that splits each scenario into a separate test result. The CI pipeline usually has to be configured to locate this output file.

Another useful output format is given with the json plugin. Not much explanation required here, but JSON is a great format for interchange with other systems, perhaps triggered by the CI build. It could also be used to supply the results of a test to a front end that can format the results nicely. You can also create your own custom output formatter, and many third party formatters are available for all sorts of uses.

Online Reporting

Cucumber also offers a free service which allows uploading your test report files to their server from where they are accessible to anyone with the link. Avoid using this if you have information in the scenarios that you want to keep secret as anyone would be able to read the results of the tests. At time of writing there is no way to secure access to the reports, but I understand that the Cucumber team are working on this.

One Final Thought

The living documentation that can be produced with behavioural testing can only be truly useful if everyone that is involved has access to the latest reports that are generated. Often people will instead look in old user stories to find out what it was agreed that a product should do, but it can sometimes be hard to see what the true picture of a system is if there has been gradual changes made that have caused some old conditions of satisfaction to become obsolete.

If you do publish the report output of the Cucumber tests for you product so that anyone in the organisation can find them, I'd encourage you to edit the user story before closing it off. Add a link to where the latest reports can be found with a note explaining what was delivered. Cucumber supports tagging scenarios (something I'll cover next post) so that you may be able to search for anything related to the user story specifically. That's a great way to ensure people find the living documentation and get the fullest understanding of how the product works.