Integrated Tests are a Scam: The Lecture That Sparked Pact.io

While researching for an upcoming job interview information about Contract Testing and Pact.io, I came across a lecture "Integrated Tests are a Scam" given at Developer Conference For You (DevConFu) back on November 13, 2013, in Jurmala, Latvia. It's amazing what historical records one can find on the internet!

I found a blurb on Pact.io / History that when Pact.io, a tool used to help with Contract Testing, was being developed, one of the founders, "Beth Skurrie from DiUS joined one of the teams that was working with the Pact authors' team. She had recently seen a talk by J. B. Rainsberger entitled 'Integration tests are a scam', which promoted the concept of 'collaboration' and 'contract' tests, so she was immediately interested when she was introduced to Pact". This blurb intrigued me, so, of course, I had to find a copy of this talk.

J. B. (Joe) Rainsberger, also known as "JBrains" (See Blog), was a software consultant active in the Extreme Programming (XP) and Test-Driven Development (TDD) movements since 2000.

https://vimeo.com/80533536

Below are my research notes on Joe Rainsberger's lecture:

"Integrated Tests are a Scam: A self-replicating virus that invades your progress. It threatens to destroy your codebase, to destroy your sanity, to destroy your life".

Here's One Simple Trick I do as an SDET to Ramp Up Quickly at a New Company!

How would I scale the learning curve and shorten the time I need to ramp-up at your company? How would I verify that I understand the material? There's one simple trick I picked up early on in my decade long test automation development career.

Can You Prompt Claude Into Being A Good Tester? Experiments with AI-Assisted Testing

Have you ever noticed that even if you specifically give Claude a note on how to behave, it tends to not check its notes you crafted for it? Things can quickly go off the rails!

• Claude Sonnet 4 silently drops requirements you spell out.
• Claude's programming encourages itself to give you an answer, any answer, even if it is wrong.
• Claude always pats itself on the back. It's code is the best ever! You question it. It sulks.
• Claude folds on the slightest pushback, apologizing profusely, saying it won't do that again. But it always, always does it again.

Let me give you an example:

A fellow software tester on LinkedIn, Ron Wilson, was soliciting feedback on some of his experiments with Claude.

Python Project: Blogger Spam Bulk Deleter Code Walkthrough: Pair-Coded with Claude but Human Explained!

Problem: My blog, Adventures in Automation, has collected over 11,000 spam comments over the past ten years, and unfortunately bare-bones Blogger.com does not have a bulk delete function. Through the Blogger UI, you can only delete a hundred at a time.

Pair-programming with Claude.ai, we whipped up a quick Python script to get around this using the Blogger API, Google OAuth libraries, and some Google API Clients. The errors that appeared after running the code, I fed back to Claude, who then fixed the issues, and added some setup documentation I was able to muddle through.

• Blogger Spam Bulk Deleter: https://github.com/tjmaher/blogger-spam-bulk-deleter

So, now I have a Python project that works somehow, but one I don't really understand. Since becoming an automation developer, I have worked on-the-job with Java, Ruby, JavaScript, and TypeScript, but not yet with Python.

Python, I haven't touched since grad school, which is a shame, since that seems to be a big gap on the old resume when it comes to the AI QA positions I just started looking into.

Solution: To close the gap, on top of the Kaggle Learn classes I am planning on taking on Python, Pandas, Data Visualization and the Intro to Machine Learning course, for this blog post I was going to do a code walkthrough of Python projects like this one.

Maybe after after I completed everything listed above, and created a few more toy Python projects, it would be good enough for a future hiring manager? Who knows?

When Claude Acts Like a Clod: Catching AI Fabrications: A QA Engineer's Field Notes

Image created by Bing AI, powered by DALL-E 3

Using AI as a research assistant? Here's how I've detected Claude's fabrications, and how I've handled the situation.

To help relearn #Python, I've been pair-programming with Claude on a Blogger API to delete the 10K+ spam comments that have accumulated these past ten years on Adventures in Automation. 

• Blogger Spam Bulk Deleter: https://github.com/tjmaher/blogger-spam-bulk-deleter

Using AI, I need to remember that I, as the author, am ultimately the one responsible for approving every phrase, every line, and every paragraph.

Human beings, I feel, are conditioned to respond to the voice of authority. 

Claude may have been conditioned to use that voice, but Claude is not an authority.

• Looking for technical information? Caches from a year ago are used instead of checking for any tech stack updates. 
• Need AI to recheck a web page after editing it with AI's suggestions? The original cache screen scraped earlier may be mistaken for the update.
• Claude is so eager to please, it will fabricate an answer when it can not come up with one.

Review its answers. Be skeptical. Use critical thinking. Ask it to cite its sources.

Becoming AI QA: Jupyter Notebook + Python

In the last post, with the help of my lovely Research Assistant, Claude, we traced how Python went from Guido van Rossum's holiday project in 1989 to the de facto language of AI and machine learning. 

Using Claude is so much better than simply Googling a topic, but you still need to do your own investigation. Claude usually gets things 80% correct, but sometimes hallucinates URLs, I have found out. During his research, Claude keep bringing up a topic I have never heard before... Jupyter notebooks... What, is that a typo? 

What Is a Jupyter Notebook?

According to the Project Jupyter official documentation, a Jupyter Notebook is a web-based interactive computing platform. The notebook combines live code, equations, narrative text, visualizations, and interactive dashboards into a single shareable document.

The file format is .ipynb -- short for "IPython Notebook," a holdover from the tool's origins.

A notebook is organized into cells. Each cell is either:

• Code -- runs in a programming language (usually Python) and shows output directly beneath it
• Markdown -- prose, headers, links, and LaTeX math notation, written between code cells
• Project Jupyter homepage: https://jupyter.org/
• Official Jupyter Documentation: https://docs.jupyter.org/
• Official Jupyter Blog: https://blog.jupyter.org/
• Project Jupyter on GitHub: https://github.com/jupyter

QA Blogosphere

Have a software testing blog? Care to trade links? 

It's tough out there being a software tester. Testing framework change every few years. Tech moves at too fast a pace to keep up. What about exploratory testing? What about examining the business requirements? Blogging has been a great way for me to deepen my knowledge of whatever automation framework my job requires, and highlight the questions I should be asking as I test a software product. 

Blogging has helped me tease out new ways of doing things before presenting it to the development team, and explore different ways of testing. I highly recommend it! 

Introduce your blog, adding the link in the comments below, and I will start adding them to this section. 

It's tough out there. Let's try to navigate the tech industry together. 

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

Becoming AI QA: Why Python? How AI and Python became linked

Image created by Bing AI, powered by DALL-E 3

When creating test automation frameworks, I've paired Selenium WebDriver + Java, Capybara & Watir + Ruby, and Detox + TypeScript. What I haven't used since grad school? Python. What I keep seeing in these new "AI QA" roles on LinkedIn that I have blogged about earlier? Python... I wonder why?  

Before I begin, let's get back to basics... What is Python? 

Hey, Claude.ai! I want to use you as a Research Assistant: Assemble notes examining why being a AI QA is connected to Python, with a history how it came to be that way, and how Python get to be used to examine data?

Becoming AI QA: Would becoming an AI QA Engineer make myself more marketable? What should I study?

Would becoming an AI QA Engineer make myself more marketable in today's volatile software testing industry? Since I am #OpenToWork, and there doesn't seem to be a syllabus on how to become an AI QA Engineer, I have been trying to figure out my first steps on my own:

• Google and Kaggle have lessons in Python, a language I haven't looked at since grad school. There also is Automate the Boring Stuff with Python by Al Sweigart. Kaggle has lessons in Pandas, Pytest, and Data Visualization. (See Kaggle.com/learn).
• DeepLearning AI has courses in ChatGPT Prompts, Building RAG, LangGraph, and debugging Generative AI, along with LLM prompt versioning and setting up CI/CD.
• Hugging Face has an LLM course.
• Google has a Machine Learning crash course.
• OWASP has a Top 10 for LLMs.
• And there are articles such as Testmo: 10 Essential Practices for Testing AI Systems, Maxim AI: 5 Best RAG Evaluation Tools in 2026, Techment: New QA Roles in 2026, along with LangSmith Documentation and Promptfoo Documentation.
• And there is something called.. Jupyter notebooks... I should look into?

... All of those courses, I think are free? And there are on Udemy:

• Test AI and LLM App with DeepEval, RAGAs and Ollama
• Complete RAG Testing with RAGAS, DeepEval, and Python
• Mastering LLM Evaluation: RAG and Agentic AI

Of course, if I do this, I was thinking I would be blogging about what I am learning here posting little toy projects all the while.

... Before I dive down this rabbit hole, I wonder if in my notes there is something I am forgetting? 

Not sure.  Ah, well. I'll find out, and make sure to let you all know. 

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

Conversations with Claude: Why do QA Engineers call it 'Test Setup' or setting up 'Pre-Conditions' for a test, while DEVs calls it 'Seeding'?

Image created by Bing AI, powered by DALL-E 3

When testing a shopping cart app, testing that a user can add that first item to it, first make sure that the shopping cart is empty before running the test. If  the cart isn't empty, delete every item in the shopping cart. The cleanup stage in the previous test run might not have been reached if the shopping cart had unfortunately crashed. 

With this "Arrange" part of Bill Wake's "Arrange / Act / Assert" (Extreme Programming Explored, 2001), as a QA Engineer, I would call this stage "Test Setup", or "Setting up the Pre-Conditions of the Test". 

Playwright and Cypress calls this... seeding. 

... Er, what? Why do they use that term?

Hey, Claude! How come I only have heard this term in the past year or two?