Quality pattern 2: Automate your acceptance tests

Welcome to my second blog in the series of five quality patterns in Agile development that can help you to deliver the right software with great quality. In my previous blog, I’ve introduced Example Mapping as a method to get to specific examples for scenarios or rules that your user story is made up of. The output of the refinement sessions are your requirements and thus your tests. In this blog, we will take a further look at these test cases and why it is important to automate these acceptance tests. Not just from a development team perspective, but also what they can bring to your business.

The perfect development process

If you haven’t done so already, then I highly recommend reading the book “How Google Tests Software” written by James A. Whittaker, Jason Arbon, and Jeff Carollo.

“Imagine for a moment the perfect development process. It would begin with test. Before even a single line of code is written, a developer would ponder what it will take to test it. He will write tests for boundary cases, for data values too large and too small, for values that would push loops beyond their limits, and for a myriad of other concerns. Some of these tests will be part of the functions she writes, self-testing code or unit tests. For these types of tests, the person who writes the code and understands it best is the one who is most qualified to test it.”

— James A. Whittaker, Jason Arbon & Jeff Carollo. “How Google Tests Software”.

It’s pretty clear. In a perfect development process, testing already starts at the beginning of the process. This perfect development process is hard to achieve, but starting with thinking about tests at the start of everything you do is certainly possible.

So we need a tester?

In the book, it is argued that you need a tester at this point.

“This is the first place where a perfect development process requires a tester. There is a different kind of thinking involved in writing feature code and writing test code. It becomes necessary to distinguish between a feature developer and a test developer. For feature code, the mindset is creating and considering users, use cases, and workflow. For test code, the mindset is about breaking and writing code that will draw out cases that disrupt the user and his workflow.”

— James A. Whittaker, Jason Arbon & Jeff Carollo. “How Google Tests Software”.

While it’s true that we need people with different skills, mindset and focus onboard, this does not mean that this should be done separately. As explained in my last blog, Example Mapping requires all disciplines to be involved through the “three amigos” concept. Perform your refinements well and you end up with detailed test scenarios including the boundary cases.

Structuring your output

The output of your Example Mapping sessions is a wall full of multi-colored stickies. This medium works very well for creating rules and examples. It’s a very visual way of writing user stories. But to use this output as input for our development, testing, and later documentation it’s needed to digitize this output.

Step one can simply be to take a picture so that someone can digitize it later. Rotate the person who is doing this. While putting the output in a system you have to keep the readability in mind. The structure is extremely important. One format to document your scenarios is Gherkin.

The Gherkin syntax is a set of keywords that can structure executable specifications. We call the specifications executable since there are tools like Cucumber who can automate these specs. Each specification with examples leads therefore directly to functional test cases.

Given When Then

The main keywords used in the Gherkin syntax are Given, When, Then (GWT). The advantage of using this syntax is that it creates a human-readable format that is used the same every time. Like our Example Mapping output, this helps in creating a common understanding.

Let’s revisit our “Receiving change” from a candy machine story we used in the first blog.

Writing down a rule as an executable specification could look like this:

User Story: Receiving Change
Specification: Change should be returned in the lowest amount of coins

Examples:

Note:

• Rules can one-on-one translate to specifications.
  Specification by Example works on the basis of specifications, where Example Mapping splits stories into small and unambiguous rules. These rules are the same concept as the specifications.
• Do not start with writing in the GWT format. Format your stories after the refinement. Starting with the GWT format has the risk of falling in the trap of making the GWT scenarios the goal. The conversation when creating the scenarios is way more important than the format. Formatting directly in GWT scenarios limits this conversation.

Do, or do not automate

In the previous blog, I’ve said: “Make sure that you automate these scenarios and examples. All of them, without exception.” While this is a good goal to have, do not make it your first action! Good testing is about making sure that you write good specs. Embedding Specification by Example with Example Mapping in your refinement process is more important than automating the specifications.

However, by not automating them you place yourself in a difficult position. Releasing more often en running the tests continuously becomes very difficult and time-consuming. Especially when your application grows. Testing is all about getting fast feedback. The earlier you discuss your specs and the sooner you run your tests the faster you know if you’re on the right track.

The nice thing about executable specs is that there are several tools that help you to couple your test code to your application code. This is done in the form of Step Definitions. Step Definitions form the “glue code” between your tests and your application.

Please beware that you don’t have to use this type of tooling. I’ve automated lots of specs without using the executable specs to really drive the tests. Let’s say you have a modern web application and you want to use Cypress.io for test automation, then you have two options. Either you use the BDD plugin, or you create a .spec file per specification where you copy in the GWT scenario and examples as a comment at the top of the file. This way it’s immediately clear what the test tries to accomplish and you keep the test small, limited to one specification.

Next steps

Now that you’ve automated your tests it’s time for the next step.
Automating your build and release process. In the next blog, I’ll dive into the why’s, how you could accomplish that and what to look out for. See you next time!

I am a specialist at Qxperts. We empower companies to deliver reliable & high-quality software. Any questions? We are here to help! www.qxperts.io