10 Automated Testing Tools That Threat Stack Uses — and Why

Previously published on the Threat Stack Blog, September 19, 2019, co-author Laura Haiduck. 

All software development projects, whether they’re large or small, can benefit from well-planned and well-executed testing. It’s your way to ensure that the software you’re developing performs as expected and delivers value to the customer. More important — given the nature of our current cyber landscape — well-executed testing is your way to ensure that your software doesn’t ship with errors or vulnerabilities that could compromise its integrity. In a word, good testing lets you pass on performance value to customers — while also providing them with underlying security.

At Threat Stack, we have a complex, full stack development environment that requires a great deal of planning and decision making within the Test Engineering Team to make sure that all the functional areas of our platform are being tested effectively and efficiently. This includes decisions about what testing tools to use, since the right ones allow us to introduce automation as well as speed, repeatability, consistency, and accuracy of results.



In this post, we’re going to discuss ten of the automated testing tools we use —- with the goal of sharing some insights that, we hope, will help you in your testing activities.

Before looking at the tools, however, let’s create some context by examining key areas that our Test Engineers take into account when they’re testing our platform and its infrastructure.

• Threat Stack agents are embedded within customers’ fleets of Amazon Web Services virtual machines in order to track and monitor them, whether they have Linux distributions — such as Ubuntu, Amazon Linux, Red Hat Enterprise Linux, CentOS, CoreOS — or whether they are of the Windows variety, such as Windows Server 2012 R2, 2016, or 2019.
• Our customers might be using containers or Kubernetes that need to be monitored for signs of security incidents.
• Each agent securely sends data to the Threat Stack Cloud Security Platform®, which is then analyzed and processed through one of our many microservices.
• Event data is processed by Host, File, CloudTrail, Threat Intelligence, and custom rules to throw alerts, broken down by severity, to help our customers’ DevOps and Security teams analyze what is happening within their systems.
• All the Event, Rule, and Alert data must be accessible not only through our browser-based application, but also through our Threat Stack APIs.
• Our customers have the ability to take this data and export it into their own external storage to perform their own analysis.
• Threat Stack Application Security Monitoring, a component of the Threat Stack Cloud Security Platform, runs as a microagent, watching application payloads, sending information to the Threat Stack platform, warning developers if it finds secure-coding mistakes. It also watches application payloads for malicious behavior such as cross-site scripting attacks.

As you can see, our software covers a lot of territory, and it has to do it fast, securely, accurately, and at scale. With that in mind, let’s take a look at some of the testing tools we use.
Testing Tools We Use at Threat Stack

Software Engineers in Test, embedded on their own feature teams, are not only responsible for testing new features and functions that are constantly being developed, but also for adding automation to make sure that the rest of the product still works as expected.

As a testament to the value our organization places on these people, each Test Engineer is also given the responsibility for recommending automation methods, tools, and languages that they deem appropriate. Below is a list of some of the tools we use to help us in our automated testing efforts. Please note that we use a more extensive toolset overall; our goal here is to discuss a selection of tools and to remind you to choose tools that work best for the specific projects you are working on.
Disclaimer: The resources mentioned in this post are not ranked in terms of perceived value. Nor do our comments imply an endorsement of any kind. Our intent is simply to provide you with information that we feel could be helpful as you add to your knowledge of testing tools.

1. Capybara

Website: https://teamcapybara.github.io/capybara/
Type: Free, Open source
Support: Ruby-Capybara Google Group

Latest version: 3.29.0 (September 2, 2019)

Capybara is the Ruby gem we use with Selenium in order to interact with our browser-based UI Automation tests when performing end-to-end user interface tests. Capybara contains built-in methods that emulate how a user can visit a page, fill_in a textbox, click_button, or select_dropdown.

The bane of a software tester’s existence is when it takes an absurd amount of time for web elements to load on a page, causing unexplained test failures. Each method provided by Capybara automatically has waits built in.

Capybara tests for user experience problems, operating our product much as our customers would, logging into our application, navigating through the site, viewing events triggered by our test systems, creating new rules that govern which events should trigger alerts, drilling down into alert information found on our dashboard, and updating various settings.

2. Gauge by ThoughtWorks

Website: http://gauge.org
Type: Free, Open source
Support: GetGauge Google Group, Gitter chat
Latest Version: 1.0.7 (August 29, 2019)

Gauge is a test automation framework created by ThoughtWorks Studio India, which we use to set up our UI and API Acceptance Tests. Although we pair Gauge with Ruby, it is also compatible with Java, C#, JavaScript, and Python. Gauge is much like Cucumber, the BDD framework, with minor differences:

• Test plans are in specification files written in Markdown instead of Cucumber’s feature files, and this makes writing and maintaining tests easier.
• Instead of Cucumber’s Given / Then / When format, test steps are listed as bullet points.

As a Behavior Driven Development (BDD) tool, Cucumber can become a bit wordy, the more complex the test is. With Gauge, each step is a clear, concise, easy-to-read bullet point. In addition, Gauge has built-in reporting available across multiple formats (XML, JSON, HTML). It can be used both on the command line and within an IDE such as Visual Studio Code. Finally, ample documentation is provided to cover a range of topics including installation, specification writing, example tutorials, and more.

3. Gatling Load Testing

Website: https://gatling.io/
Type: Both Enterprise and Open source versions
Support: Gatling Google Group
Latest Version: 3.2.1 (August 27, 2019)

Gatling provides continuous load testing here at Threat Stack. Written in Scala, it works either as a standalone tool, or can be used as a dependency to your product with Maven.

As a demonstration of its speed and power, within ten minutes we were able to generate three million requests in our test environment, and were able to determine that all of the requests returned results within under 800 milliseconds.

As BlazeMeter states in their blog, Eight Reasons You Should Use Gatling For Your Load Testing, “Gatling created detailed metrics dashboard that you can see after tests execution without having to add any additional plugins. The report is stored as an HTML file, which can be easily saved for some future analyses and metrics comparison. In addition to that, the report is interactive. which allows you to perform more detailed analyses and concentrate specific requests in addition to the overall picture.”

4. Test Kitchen by Chef.io

Website: https://kitchen.ci/
Type: Open source
Support: Chef.io Community
Latest Version: 2.2.0 (April 26, 2019)

Test Kitchen is the Chef.io product we use to automate spinning up and tearing down the different Amazon Web Services environments we use during our testing efforts. Test Kitchen is a test harness tool you can use to execute your configured code on one or more platforms in isolation. Many testing frameworks are supported by Kitchen out of the box, including Chef InSpec. Kitchen is used by all Chef-managed community cookbooks and is the integration testing tool of choice for cookbooks.

Learn.Chef.io has a well-thought-out series of training modules and demos, including an excellent module called Getting Started with Test Kitchen. 

5. ScalaTest

Website: http://www.scalatest.org
Type: Free, Open source
Support: ScalaTest Google Group
Latest Version: 3.0.8 (June 10, 2019)

Test Engineers at Threat Stack have started to partner with our Software Developers in order to write service tests for their Scala code using ScalaTest.

Our integration tests use the ScalaTest style trait, FeatureSpec, describing the test-driven development (TDD) style unit tests in a Given / When / Then format. As the ScalaTest Doc mentions, FeatureSpec is a “suite of tests in which each test represents one scenario of a feature. FeatureSpec is intended for writing tests that are ‘higher level’ than unit tests — for example, integration tests, functional tests, and acceptance tests.”

ScalaTest provides plenty of helpful documentation on their site, including a detailed quick start, installation procedures, and a user guide.

6. Chef InSpec

Website: https://kitchen.ci/docs/verifiers/inspec/
Type: Free, Open source
Support: Chef.io Community
Latest Version: (August 11, 2019)

At Threat Stack, we use Chef InSpec to create Ruby-based tests to verify that everything is working as expected once a test environment has been created on-the-fly. InSpec tests are meant to be human-readable. If you’re familiar with testing frameworks such as RSpec, InSpec code should be familiar. InSpec detects discrepancies and documents the differences in the form of a report. We can then go in and make the Chef changes needed to get a system to a desired state.

To get you started, the Learn Chef Rally has a useful segment that walks you through setting up a Docker and Docker Compose environment, downloading the sample environment, and exploring the environment using InSpec. 

7. Mocha

Website: https://mochajs.org/
Type: Free, Open source
Support: Mocha Gitter Group, Mocha Google Group
Latest Version: 3.2.1 (August 27, 2019)

Mocha is a JavaScript test framework that runs on Node.js and in a browser, enabling us to undertake asynchronous testing. Because Mocha tests run serially, it allows us more flexible and accurate reporting as well as the ability to map uncaught exceptions to the correct test cases.

Using Mocha in a Node.js environment, you can use the built-in assert module as your assertion library. At Threat Stack, we pair Mocha with Chai to create the test framework for the Threat Stack Application Security Monitoring functionality.

The Mocha site makes available an array of documentation on subjects covering everything from a detailed feature overview, installation, and getting started, to information on specific features and tasks. 

8. Cypress.io

Website: https://www.cypress.io/
Type:

• Free, Open source test runner
• Commercial Dashboard service, free for up to three users

Support: Gitter chat; Email support for paid users
Latest Version: 3.4.1 (July 29, 2019)

Cypress.io is built for JavaScript front-end developers who only need to test their work in Chrome. Cypress is executed in the same run loop as our application, meaning that all the functions, windows, web elements, timers, and services the Threat Stack application can access, our test code can access as well. We can also read and alter network traffic on the fly, which allows us to add some interesting negative test cases, such as testing what happens when a backend server is inaccessible.

Cypress.io offers well-organized documentation that provides a solid overview, getting started information, and core concepts that include information on writing and organizing tests. Looking to see what Cypress can do? TestAutomationU just released a new course, Introduction to Cypress.

9. Jest

Website: https://jestjs.io
Type: Free, Open source
Support: React Discord Chat, JestJs Google Group
Latest Version: 24.9 (August 2019)

Our full stack software developers have switched testing the Threat Stack Cloud Security Platform ReactJS-based frontend from Mocha / Chai to Jest, which has better built-in support for testing the ReactJS components we use. With this JavaScript testing framework, we are able to create better unit and integration tests. Both toolsets, ReactJs and Jest, are brought to you by Facebook.

As with most of the other tools in this post, the Jest website provides example-rich documentation covering how to get started, configuration, and a comprehensive range of use case scenarios.

10. Apache JMeter

Website: https://jmeter.apache.org/
Type: Free, Open source
Support: JMeter Google Group
Latest Version: 5.1.1 (March 13, 2019)

Apache JMeter is open source software designed to load test functional behavior and measure performance. According to their site, JMeter “may be used to test performance both on static and dynamic resources, Web dynamic applications. It can be used to simulate a heavy load on a server, group of servers, network or object to test its strength or to analyze overall performance under different load types.”

We use JMeter to analyze and measure the performance of our application and services. This allows us to test our application against heavy loads, with multiple and concurrent user traffic to see how we can improve our ability to scale.

JMeter makes available a useful array of documentation (including Getting Started, User Manual, and Best Practices) as well as a number of step-by-step tutorials.

As good as our Test Engineers are at Threat Stack, they wouldn’t get far without systematic processes backed up by powerful tools. As our platform evolves, our team continues to hone its expertise, continually upgrading its skills and knowledge, adopting and incorporating new tools, and developing new methods to ensure that they continue to add value to the Threat Stack SDLC. Hopefully this post has given you some insight into the ways our Test Team uses a range of automated testing tools to address the multiple functions within our Cloud Security Platform.


Happy Testing!

-T.J. Maher
Sr. QA Engineer, Software Engineer in Test
Meetup Organizer, Ministry of Testing - Boston

Twitter | YouTube | LinkedIn | Articles