Testing the Swagger Petstore: Manually Testing An API Using Swagger UI

With the last post, Testing the Swagger Petstore: Reviewing API Documentation Formatted in OpenAPI 3.0 with Swagger UI we started exploring an API with Swagger UI, such as the Swagger Petstore at https://petstore3.swagger.io/

Here, we will start exploring how to test an API using the Swagger UI.

The tests we can run are:

• Happy Path Testing, checking the Positive Scenarios
• Negative Testing, reviewing the Invalid Inputs & Error Handling
• Boundary & Edge Case Testing, checking how the API handles the extreme limits of allowed input ranges
• Authentication & Authorization Testing
• Contract and schema validation

We can see in the Swagger PetStore there are three tag groups: Pet, Store, and User. 

Because this is a Swagger UI doc, we can perform manual testing using only the "Try it out" feature in Swagger UI. No external tools needed! Live HTTP requests can be executed directly from your browser.

Testing the Swagger Petstore: Reviewing API Documentation Formatted in OpenAPI 3.0 with Swagger UI

For this next project, we will be examining the test site Swagger Petstore - OpenAPI 3.0 at https://petstore3.swagger.io/

https://petstore3.swagger.io/

If we wanted to really dive deep into creating API documentation, we could use:

• I'dRatherBeWriting's Free Documenting APIs course, showing an Introduction to REST APIs, using APIs like a developer, Documenting API endpoints, the Open API spec, Testing API docs, Publishing API docs, and more... 

... But that goes too deep for this blog post. 

Before we begin, let's get into some background information. 

What is Swagger?

According to OpenAPISpec.com's article, Who Created Swagger?, Swagger was created in 2010 by Tony Tam, the CTO of Wordnik, an online dictionary company that leveraged so many APIs, it was difficult for the six person engineering team to manage and scale them, writing API-clients and documentation by hand. After a 3 AM conversation, they thought... what if the API documentation could describe itself? 

Publishing Playwright HTML reports on GitLab Pages

We've added a lot to our demo project, bun--create-playwright. 

We've explored setting up a Playwright framework using bun, a new package manager bundled into Claude Code. We've added typechecking, and formatting and linting. We've set up a CI/ CD pipeline for our tests using GitLab, one with three stages: 

• Quality: We check that the formatting, linting and typechecking is correct for any changes we attempt to push to the code base.
• Test: We run the smoke and then the regression tests to make sure that everything still works. 
• Report: We bundle an archive of the Playwright generated HTML report, with screenshots and videos if things fail.

For this post, we will add a new stage to this GitLab pipeline:

•  Deploy: Where we will deploy the HTML report to GitLab Pages, so we can view it online. 

Examining the artifacts job of a Playwright GitLab CI / CD pipeline

You may have noticed that in the GitLab CI / CD Pipeline for our Playwright project, in the .gitlab-ci.yml file, after our scripts has run, in the Test stage, there is a subsection called "artifacts" with certain paths to something called "playwright-report", "test-results", and "reports". 

The Playwright -> Artifacts -> Reports stage

playwright:

  extends: .bun_playwright

  stage: test

  timeout: 30 minutes

...

...

  artifacts:

    when: always

    paths:

      - playwright-report/

      - test-results/

      - reports/

    reports:

      junit: reports/junit/results.xml

    expire_in: 30 days

https://gitlab.com/tjmaher/bun-create-playwright/-/blob/main/.gitlab-ci.yml

Every time Playwright tests run, it generates proof of the test execution: screenshots capturing the state of the UI when a test failed, video recordings of an entire browser session, HTML and JUnit XML test reports, trace reports showing action logs, network requests. These artifacts, when the tests run locally, are placed by the Playwright Test runner in generated folders, playwright-report and test-results, along with a folder called reports/junit. 

With this post, we will be examining how this artifacts stage of the GitLab CI/ CD pipeline produces downloadable artifacts we can examine.

How Developers Can Test Their Feature Branch Against Various Playwright Configurations Before Creating a Merge Request in GitLab CI/CD

Developers need the option to test out their feature branch before merging it into main. In the last section, we set up a GitLab CI/CD Pipeline in our bun-create-playwright project to check every GitLab Merge Request. 

Now, we will be giving developers option to create a new pipeline where they can choose:

• Which feature branch to test against.
• Which test suite to execute (LoginPage tests? Secure Area? Or all of them?)
• Which Playwright browser to use: Chromium, Firefox, WebKit, or all.

All these features can be set up using our .gitlab-ci.yaml file!

Go to to bun-create-playwright, view the Pipelines section, and select the New Pipeline button. 

Setting up a CI/ CD pipeline with GitLab: Quality, Test and Report

So far, we have reviewed how to review code with ES Lint + Prettier and Typecheck, how to set up and run smoke tests, how to run all Playwright + TypeScript tests, and reviewed the HTML report of results.

In this post, we are going to set up a three stage GitLab CI/ CD pipeline that will run against every merge request: 

•  Quality (lint, typecheck, prettier)  --> Test ( smoke + regression ) --> Report ( Downloadable )

I haven't used GitLab since when I worked at ThreatStack back in 2020. ( See my blog entry Getting to Know GitLab and How They Test the UI )

GitLab reads .gitlab-ci.yml from the repository root and turns it into a pipeline, a set of jobs, grouped into stages, running whenever you push a change to a repository. 

Stages run one after another, and jobs within a stage run in parallel.  If one stage fails, the stages after it are skipped. 

This begs the question: What is GitLab? What is CI/ CD? Or a pipeline? Or a merge request? 

How to Configure Playwright Test to run smoke tests, headed tests, and debug versions through scripts in package.json

Earlier, we went over how we could create scripts in the package.json file of our Playwright framework to add typechecking, linting, and formatting your code with prettier. 

• Want to see the entire bun-create-playwright project? It is on tinyurl.com/bun-create-playwright!

With this post, we will explore how the built-in test runner for Playwright Test can run headed tests, debug versions of tests, and smoke tests.

... and shortcuts for all of these can be set up in the scripts in your package.json! If you are using "bun" as a package manager, as we are in bun-create-playwright, just type out "bun run", a space and then the shortcut such as: bun run test

package.json

"scripts": {

    "test": "playwright test",

    "test:headed": "playwright test --headed",

    "test:trace": "playwright test --trace on",

    "test:chromium": "playwright test --project=chromium",

    "test:firefox": "playwright test --project=firefox",

    "test:webkit": "playwright test --project=webkit",

    "test:smoke": "playwright test --project=chromium --grep '@smoke'",

    "test:flaky": "playwright test --project=chromium --repeat-each=20",

    "test:ui": "playwright test --ui",

    "test:debug": "playwright test --debug",

    "test:failed": "playwright test --last-failed",

    "test:login": "playwright test tests/login.spec.ts",

    "test:secure-area": "playwright test tests/secure-area.spec.ts",

    "report:list": "playwright test --reporter=list",

    "report:line": "playwright test --reporter=line",

    "report:dot": "playwright test --reporter=dot",

    "report:blob": "playwright test --reporter=blob",

    "report": "playwright show-report",

    "codegen": "playwright codegen",

    "lint": "eslint .",

    "lint:ci": "eslint . --max-warnings 0",

    "lint:fix": "eslint . --fix",

    "format": "prettier --write .",

    "format:check": "prettier --check .",

    "format:debug": "prettier --check . --log-level debug",

    "format:diff": "prettier --list-different",

    "typecheck": "tsc --noEmit"

  },

bun-create-playwright / package.json

Do you need to really set up shortcuts like these? Certainly not! But it is easier than typing out: bunx playwright test --project=chromium --grep '@smoke'.

Feel free to name these commands anything you want! 

Implementing Page Objects in Playwright

Picture a login screen, such as The-Internet / Login. 

On this LoginPage, there is:

• a heading: Login Page
• a user name textbox with the label, "Username"
• a password textbox with the label, "Password"
• a login button, with the role of a button, and the name of "Login"
• a flash message that appears if you enter invalid credentials such as "NotAUser" and "NotAPassword".

If you successfully log in with "tomsmith" and "SuperSecretPassword!, there is a SecureArea:

• a heading: "Secure Area"
• a flash message "You logged into a secure area!"
• a Logout button. 

Sure, you could interact with each web element in your test... but what if the username text box locator changes? You would have to update multiple tests every time the element changed. 

... Instead, you could place it in a Page Object, something that Playwright handles well!

"A page object represents a part of your web application. An e-commerce web application might have a home page, a listings page and a checkout page. Each of them can be represented by page object models.

"Page objects simplify authoring by creating a higher-level API which suits your application and simplify maintenance by capturing element selectors in one place and create reusable code to avoid repetition".

Using Playwright's Built-In Test Runner? Or Something Else?

You may have noticed in https://playwright.dev/docs/pom that there are two different styles of page objects. One for "Test". One for "Library". 

• Test: If you are writing actual Playwright test suites, and Playwright's built in test runner, use the Test section as a guide when creating page objects. 
• Library: If you are integrating Playwright into an existing test framework such as Jest or Cucumber and just want browser automation, instead of having pre-built page fixtures, etc, you can use this format. 

How Playwright Frameworks get configured with playwright.config.ts

When we installed bun, a new package manager owned by Anthropic, then ran "bun create playwright", a new automation framework was stood up, along with sample tests, and a Playwright configuration file. In this post, we will be examining the file generated: playwright.config.ts.  

Personally, I find the pre-generated file very hard to scan... there are so many options and documentation in the comments, it is difficult for me to focus on the code. Let's examine just the code generated below. If you need to see the whole file, you can see it here: https://playwright.dev/docs/test-configuration

Playwright.dev / Configuration mentions, "Playwright has many options to configure how your tests are run. You can specify these options in the configuration file". 

Running Tests with Playwright Test Explorer and Generating Tests with Codegen

Finding the best locator for a web element can be a hassle:

• Right clicking on an element in Google Chrome. Inspecting the element. Going to Chrome Developer Tools. Try to decide what to do if there isn't a clear test id. 

Playwright comes with a built in code generator where it can built out a rough draft of a test while you interact with a website. "Playwright will look at your page and figure out the best locator, prioritizing role, text and test id locators. If the generator finds multiple elements matching the locator, it will improve the locator to make it resilient that uniquely identify the target element", according to Playwright.dev / Test Generator.

Do you have the Integrated Development Environment (IDE) by Microsoft, VS Code? You can get it at the Visual Studio Marketplace. 

Playwright Test Explorer

• Playwright.dev / Getting Started VS Code: https://playwright.dev/docs/getting-started-vscode

After installation, you will see a beaker icon in your VS Code left navigation menu. Selecting that, you can see your tests, such as the default ones Playwright automatically adds when it is installed: