Excerpt: How to Gauge What Your Tests Are Doing

Excerpted from Continuous Testing for DevOps Professionals (2018), featuring stories from over twenty software testers. Order your copy now!

How to Gauge What Your Tests Are Doing

Introduction

Every software development team struggles with answering two questions:

• How can the entire team, from business analysts to developers to testers, determine how your product should behave? 
• How can you gauge how your software product is truly behaving? 

Dan North, as an answer to those questions, came up with the concept of Behavior Driven Development (BDD). Phrasing the Testing part of Test Driven Development (TDD) as a Behavior, it transformed BDD from a testing tool into a communication tool.

With BDD, the entire team as a whole can brainstorm together on how the product should (and should not) work. By breaking down behaviors into small, simple, and understandable sentences, the documentation being created can be used as a product specification, as a blueprint developers and testers can use and as acceptance criteria the business analysts can evaluate.

Ever since Dan North and Elizabeth Keogh took this concept to Java with JBehave, it has spread like wildfire to many other languages. Steven Baker, and Dave Astels developed RSpec for Ruby. Aslak Hellesoy took his work on RSpec and designed Cucumber and its Gherkin language. John Ferguson Smart created a reporting framework for JBehave and Cucumber with SerenityBDD (initially called Thucydides).

In this chapter, we will be exploring Gauge, a new BDD framework created by Thoughtworks India. Although Gauge 1.0 just came out of beta in June 2018, it made its debut January 2015 as an open-source product. I first encountered Gauge while working on Threat Stack’s Test Engineering team as a Software Development Engineer in Test. Testing involved spinning up various Linux distros on AWS instances, attaching the Threat Stack agent to monitor and detect Cloud Trail logs, verifying the correct microservice displayed the correct API output. After a proof-of-concept, Gauge was shown to handle these complexities well.

What Makes Gauge Different?

“What if we wanted to break away from every form of syntax and write specifications in natural language, like you’d write an email?”

As the Gauge team asked the audience at a software testing Meetup back in 2015: What if you could create a new BDD toolset that avoided the restraining Given / When / Then format? One that was more simple? Customizable? These were the ideas the Gauge team presented in their “Gauge Your BDD Tests” talk at vodQA, Value Oriented Discussion about Quality Analysts, a software testing conference in Hyderabad, India.

Gauge is based on Twist (2009 - 2015), a Java-based commercial automation framework which worked with the Eclipse IDE, built by the ThoughtWorks Studio team in Bangalore, India. With more of ThoughtWorks customers seeking open-source solutions, Gauge was creating as a lightweight cross-platform alternative.

What makes Gauge different than all the other BDD toolsets? Deepthi Chandramouli, back in February 2015 on the Google Groups for Gauge, outlined five major points:

"1. Gauge is a single product that supports multiple language across different platforms. This ensures a uniformity and consistency irrespective of which language and IDE you use to write your tests.

"2. Gauge uses Markdown as the markup to author specifications. This makes it feature rich from an execution and documentation point of view.

"3. Gauge’s architecture is plugin based which makes it highly extensible. This means that the core product can be enhanced by adding in support for a new language, a new IDE, a customised report, external tool integration etc. all as plugins. What this means is that any features added to Gauge core automatically scale to all the supported languages.

"4. Gauge has first class refactoring support both on command line and IDE. We feel that this is the extremely important to maintain a large test suite.

"5. Gauge has first class parallelisation support as a core feature (in development), which applies to all supported languages".

What Platforms Does Gauge Support

It is quite easy to get going with Gauge, on the Mac, Windows, or Linux Machines, running in C#, Java, or Ruby, providing you have the requisite .NET framework, JDK 6+ or Ruby 2.0+ for the platform.

Gauge can be installed on:

• MacOS through Homebrew
• Windows using Chocolatey or downloading the zip installer
• Linux using apt-get, dnf, or the zip installed. 

Once Gauge is installed, you can install the language runner, and a fully scaffolded demo project, with the project and folder structure already set up, by choosing to type into the command line either:

• gauge install csharp
• gauge install java
• gauge install ruby

Although for the purposes of this chapter we will be using Java, the demo projects set up for each language are all the same: Given a word, count the number of vowels, and assert that the expected and the actual count match. Tests are data driven, with input captured in a data table with two columns, ‘Word’, and the expected ‘Vowel Count’.

The projects can be run from the command line with `gauge run spec`.

Sample Output, Command Line:

Want to see other options on the command line? View them at manpage.gauge.org.

How Can Gauge Be Used For Web Application Testing

To see how Gauge can be used to test web applications, let’s take a look at one of Gauge’s many sample projects on Gauge’s GitHub site, java-maven-selenium.

Specifications: Imagine that you have a web-based bookstore application, where users can log into the site, search for books, place an order, and log out. It could look something like this:

 

Running activeadmin-demo.war included in java-maven-selenium

A test case specification to place an order could look something like this:

java-maven-selenium / specs / PlaceOrder.specs

Place an Order

==============

* Go to active admin store

Buy a book

----------

tags: customer

* Log in with customer name "ScroogeMcduck" and "password"

* Place order for "Beginning Ruby: From Novice to Professional".
The cart should now contain "1" items

* Log out

With Gauge, this isn’t simply a specification for a feature of a product: It’s a way to both organize the test code, and set up the HTML or XML reports. Each element in the *.spec file corresponds to an element in Markdown, the text formatting language created by Josh Gruber and Aaron Swartz back in 2004. Specifications are a “business layer test cases which can also act as your feature documentation. They are written in the business language. Typically a spec or specification describe a particular feature of the application under test”, according to the official Gauge documentation.

• The Specification Heading, “Place an Order”, at the top of the page is in H1, Header 1 format, the main header. 
• The Scenario Heading, “Buy a book”, is in H2, Header 2 format, a subheader.
• Each step, written in plain English, clearly and concisely sums up, in a bullet point format, step-by-step what each test will entail. 

Concept Files: When writing a spec, readability is key. The steps referencing Login and Logout are actually rollups of a multi-step process called concepts.

"Concepts provide the ability to combine re-usable logical groups of steps into a single unit. It provides a higher level abstraction of a business intent by combining steps. [...] They are defined in .cpt format files in the specs directory in the project. They can be inside nested directories inside the specs directory" (Docs.Gauge.Org).

Below, you can see that the line “Log in with customer name <name> and <password>” corresponds with the specification, above. The values passed in for the name (“ScroogeMcduck”) and value for the password (“password") are inputted into the corresponding concept header. 

java-maven-selenium / specs / concepts / Authentication.cpt

# Check if the user <name> is logged in

* Show the log in status for user <name>

* Give an option to Log Out

# Log out

* Log out the customer

* Show a message "You have been logged out."

* Give an option to Log In

# Log in with customer name <name> and <password>

* Login as name <name> and <password>

* Show the log in status for user <name>

* See items available for purchase.

Here, you can see that Log In actually consists of three steps: the action to be taken, and two ways to confirm that everything is working.

Step Definitions: Each step listed in the specification corresponds to either a concept or a reusable step definition pairing the plain language of the spec with the Java, C#, Javascript, Python, or Ruby code that executes.

Example: In the Place Orders.spec, the log in and password is initialized, fed into the Authentication.cpt concept, and passed into the LogIn.java step definition.

Each test is set up to be data driven. Enter a parameter in quotation marks in the spec, a corresponding variable name in angle brackets, and that value can now be used in that particular block of code.

java-maven-selenium / src / test / java / LogIn.java

@Step("Login as name <name> and <password>")

public void LoginAsCustomerDetails(String name, String password) {

WebDriver webDriver = Driver.webDriver;

webDriver.findElement(By.linkText("Log in")).click();

    webDriver.findElement(By.name("login")).sendKeys(name);

    webDriver.findElement(By.name("password")).sendKeys(password);

    webDriver.findElement(By.name("commit")).click();

}

Here, we can see the glue that connects the spec with the step definitions, the test plan to the code that executes the test.



To read more, purchase your copy of Continuous Testing for DevOps Professionals 

Bio

Thomas F. Maher, Jr -- best known as “T.J. Maher” -- is a tinkerer, exploring ways to craft automation solutions. Back in 2015, T.J. shook the dust off his BSCS and Masters of Software Engineering, combined them with his twenty years of software testing experience and his passion for writing, and launched both a software testing blog and a career in automation development. Whenever T.J. experiments with a new automation tool or technology, he blogs about the experience. On “Adventures in Automation”, at tjmaher.com, he provides downloadable sample code and code walkthroughs of frameworks he has created. By 2016, T.J. became a contributing writer at TechBeacon.com, giving talks to Boston-area software testing Meetup groups. By 2017, T.J. became the Meetup Organizer of the Ministry of Testing - Boston, the Massachusetts branch of the MinistryOfTesting.com, a UK based community of software testers. And in 2018, T.J. was a guest speaker at the AutomationGuild.com and the TestingGuild.com. T.J. can be reached on Twitter at @tjmaher1.



Happy Testing!

-T.J. Maher
Sr. QA Engineer, Software Engineer in Test
Meetup Organizer, Ministry of Testing - Boston

Twitter | YouTube | LinkedIn | Articles