How to add Static Application Security Testing (SAST) with SonarQube Cloud Free Tier to a GitLab CI/ CD pipeline

ESLint catches code quality issues. Typecheckers ensures variables and functions use the correct data types. Prettier automatically formats code for consistent style. But how can we start incorporating security testing into this GitLab CI/CD pipeline we have built? 

• GitLab repo: https://gitlab.com/tjmaher/bun-create-playwright

For this blog post we will be looking at one way to do Static Application Security Testing (SAST), adding SonarQube to our project.

What is Static Application Security Testing?

Static Application Security Testing (SAST) is a method of security testing that scans an application's source code, bytecode, and binaries for any security flaws. Because it analyzes the code without actually running the application -- meaning the code is "static" -- developers can use it early in the coding phase to catch and fix bugs immediately.

Although source code scanners have been around a while in the computer industry,  the term "Static Application Security Testing (SAST)" originated as a buzzword by Gartner, Inc. Gartner is known for their "Magic Quadrant" a visualization technique they use to map out industry leaders on a four part grid "Completeness of Vision" and "Ability to Execute" showing who are the market leaders, who are the challengers, who are the visionaries, and who are the niche players, as they put it.

Joseph Feiman and Neil Macdonald published in 2009 their paper Magic Quadrant for Static Application Security Testing (See PDF). 

Refactoring The-Internet: Deploying the web app to Render

We are well on our way to place a copy of my favorite test site, The-Internet, onto its new home on Render.com, a place where developers can deploy web apps using their free tier, much like Heroku in its heyday. Sure, web apps that live on the free tier of Render takes a minute or so to spin up, but, hey, you get what you pay for!

With our last blog post, we: 

• Examined using Google AI the Gemfile of the original The-Internet, a Ruby project at https://github.com/saucelabs/the-internet, hosting on the failing site, https://the-internet.herokuapp.com/
• Created a new copy of the web app onto my GitHub site, forking the original repo
• Updated the Ruby architecture of the web app per Google AI's findings. 
• Tested the web app could be deployed locally and pushed the changes to my GitHub site: https://github.com/tjmaher/the-internet
• Almost have the site at its new home at https://the-internet-8uaj.onrender.com/

With this blog post, we will create a new account on Render, a new project, and set up a new home for The-Internet.  

Refactoring The-Internet: Fork, Fix, Farewell to Heroku and hello to Render!

With this blog post, we will be taking Dave Haeffner's The-Internet, https://the-internet.herokuapp.com/, a ten-year old web app test site, fork it, investigate using Google AI how to modernize the server architecture, and port it over to a new home at Render.com. 

• Spoilers: The original version, but with updated Ruby architecture: https://the-internet-8uaj.onrender.com/ (takes a minute to spin up the environment)

It finally happened. Dave Haefner's The-Internet, a test site I have been using ever since I first started blogging, is now failing to load half the time. It's not the test site's fault. It's Heroku. 

The thoroughness of Tom Johnson's free Documenting API's course on OpenAPI 3.0 and Swagger UI

While researching information for my job about the Swagger UI and how it can be used for testing, I came across this free course, called Documenting APIs: A guide for technical writers and engineers at https://idratherbewriting.com/learnapidoc/docapis_overview.html written by Tom Johnson, a technical writer based in Seattle who works for Google. 

The course looks intense, containing 17 chapters talking about Introduction to REST APIs, using APIs like a developer, Documenting API endpoints, the Open API spec, Testing API docs, Publishing API docs, and more.

From the Course Introduction: "You’ll first learn about API documentation by using a simple weather API to put a weather forecast on your site. [...] As you use the API, you’ll learn about endpoints, parameters, data types, authentication, curl, JSON, the command line, Chrome’s Developer Console, JavaScript, and more. The idea is that rather than learning about these concepts independent of any context, you learn them by immersing yourself in a real scenario while using an API. Immersion in real scenarios makes these tools and technologies more meaningful.

"We’ll then transition into standards, tools, and specifications for REST APIs. You’ll learn about the required sections in API documentation, analyze examples of REST API documentation from various companies, learn how to join an open-source project to get experience, and more.

Testing the Swagger Petstore: Manually Testing An API Using Swagger UI

With the last post, Testing the Swagger Petstore: Reviewing API Documentation Formatted in OpenAPI 3.0 with Swagger UI we started exploring an API with Swagger UI, such as the Swagger Petstore at https://petstore3.swagger.io/

Here, we will start exploring how to test an API using the Swagger UI.

The tests we can run are:

• Happy Path Testing, checking the Positive Scenarios
• Negative Testing, reviewing the Invalid Inputs & Error Handling
• Boundary & Edge Case Testing, checking how the API handles the extreme limits of allowed input ranges
• Authentication & Authorization Testing
• Contract and schema validation

We can see in the Swagger PetStore there are three tag groups: Pet, Store, and User. 

Because this is a Swagger UI doc, we can perform manual testing using only the "Try it out" feature in Swagger UI. No external tools needed! Live HTTP requests can be executed directly from your browser.

Testing the Swagger Petstore: Reviewing API Documentation Formatted in OpenAPI 3.0 with Swagger UI

For this next project, we will be examining the test site Swagger Petstore - OpenAPI 3.0 at https://petstore3.swagger.io/

https://petstore3.swagger.io/

If we wanted to really dive deep into creating API documentation, we could use:

• I'dRatherBeWriting's Free Documenting APIs course, showing an Introduction to REST APIs, using APIs like a developer, Documenting API endpoints, the Open API spec, Testing API docs, Publishing API docs, and more... 

... But that goes too deep for this blog post. 

Before we begin, let's get into some background information. 

What is Swagger?

According to OpenAPISpec.com's article, Who Created Swagger?, Swagger was created in 2010 by Tony Tam, the CTO of Wordnik, an online dictionary company that leveraged so many APIs, it was difficult for the six person engineering team to manage and scale them, writing API-clients and documentation by hand. After a 3 AM conversation, they thought... what if the API documentation could describe itself? 

Publishing Playwright HTML reports on GitLab Pages

We've added a lot to our demo project, bun--create-playwright. 

We've explored setting up a Playwright framework using bun, a new package manager bundled into Claude Code. We've added typechecking, and formatting and linting. We've set up a CI/ CD pipeline for our tests using GitLab, one with three stages: 

• Quality: We check that the formatting, linting and typechecking is correct for any changes we attempt to push to the code base.
• Test: We run the smoke and then the regression tests to make sure that everything still works. 
• Report: We bundle an archive of the Playwright generated HTML report, with screenshots and videos if things fail.

For this post, we will add a new stage to this GitLab pipeline:

•  Deploy: Where we will deploy the HTML report to GitLab Pages, so we can view it online. 

Examining the artifacts job of a Playwright GitLab CI / CD pipeline

You may have noticed that in the GitLab CI / CD Pipeline for our Playwright project, in the .gitlab-ci.yml file, after our scripts has run, in the Test stage, there is a subsection called "artifacts" with certain paths to something called "playwright-report", "test-results", and "reports". 

The Playwright -> Artifacts -> Reports stage

playwright:

  extends: .bun_playwright

  stage: test

  timeout: 30 minutes

...

...

  artifacts:

    when: always

    paths:

      - playwright-report/

      - test-results/

      - reports/

    reports:

      junit: reports/junit/results.xml

    expire_in: 30 days

https://gitlab.com/tjmaher/bun-create-playwright/-/blob/main/.gitlab-ci.yml

Every time Playwright tests run, it generates proof of the test execution: screenshots capturing the state of the UI when a test failed, video recordings of an entire browser session, HTML and JUnit XML test reports, trace reports showing action logs, network requests. These artifacts, when the tests run locally, are placed by the Playwright Test runner in generated folders, playwright-report and test-results, along with a folder called reports/junit. 

With this post, we will be examining how this artifacts stage of the GitLab CI/ CD pipeline produces downloadable artifacts we can examine.

How Developers Can Test Their Feature Branch Against Various Playwright Configurations Before Creating a Merge Request in GitLab CI/CD

Developers need the option to test out their feature branch before merging it into main. In the last section, we set up a GitLab CI/CD Pipeline in our bun-create-playwright project to check every GitLab Merge Request. 

Now, we will be giving developers option to create a new pipeline where they can choose:

• Which feature branch to test against.
• Which test suite to execute (LoginPage tests? Secure Area? Or all of them?)
• Which Playwright browser to use: Chromium, Firefox, WebKit, or all.

All these features can be set up using our .gitlab-ci.yaml file!

Go to to bun-create-playwright, view the Pipelines section, and select the New Pipeline button. 

Setting up a CI/ CD pipeline with GitLab: Quality, Test and Report

So far, we have reviewed how to review code with ES Lint + Prettier and Typecheck, how to set up and run smoke tests, how to run all Playwright + TypeScript tests, and reviewed the HTML report of results.

In this post, we are going to set up a three stage GitLab CI/ CD pipeline that will run against every merge request: 

•  Quality (lint, typecheck, prettier)  --> Test ( smoke + regression ) --> Report ( Downloadable )

I haven't used GitLab since when I worked at ThreatStack back in 2020. ( See my blog entry Getting to Know GitLab and How They Test the UI )

GitLab reads .gitlab-ci.yml from the repository root and turns it into a pipeline, a set of jobs, grouped into stages, running whenever you push a change to a repository. 

Stages run one after another, and jobs within a stage run in parallel.  If one stage fails, the stages after it are skipped. 

This begs the question: What is GitLab? What is CI/ CD? Or a pipeline? Or a merge request?