AI in QA: From ChatGPT Experiments to Agentic Testing

Vojta, how did you get started with AI?

It was around 2022, when I was still at Škoda. I started using ChatGPT mainly in my Java work, for example, to explore architecture ideas and best practices for Selenium or Appium. To be honest, it was pretty painful back then.

You would write a prompt and sometimes have to rewrite it twenty times before you got anything useful. There was also a limit: after a few more questions, the conversation would disappear, you would get an HTTP 500 error, and you had to start all over again.

These days, people rely on AI a little too much. Back then, we were learning that it was not always right: models hallucinated and came across a bit like horoscopes. AI can help you, but it can also throw you completely off track.

Where is QA as a field heading?

Shift-left testing is now crucial: testing starts in the backlog and continues all the way through to the production merge. Instead of the traditional approach, where testing only happens at the end of development, it runs continuously from the design and coding stages onward.

At the same time, shift-right testing is emerging. You monitor the application in production, collect data, and feed what you learn back into development. It is not just about “watching” the application; it is about bringing observability and production monitoring data into the testing strategy. By 2024, this had become standard practice.

Shift-left testing + shift-right testing = continuous testing. Testing in a loop.

QA is not just the final stage of a project. It is an ongoing process that lives alongside the product.

How are you actually using AI in QA today?

The biggest shift I see is in test automation and in AI starting to work as an agent. For example, AI can be used with Playwright to:

• guide testing and generate tests
• help identify edge cases
• suggest structures for E2E, component, accessibility, API, and UI testing
• cover integration tests, everything except unit tests
• run visual tests that compare differences between components (Applitools Eyes)
• support self-healing tests by detecting when a locator has changed and replacing it

In less than two years, by 2025, Playwright was releasing new features and improvements almost every month, including AI capabilities. Then MCP arrived, followed a few months later by Playwright CLI. That took the work of QA and SDET teams to a new level.

Which general-purpose QA tools are people using?

I have put the tools QA specialists have used over the years into a table, covering everything from web testing and mobile platforms to reporting.

Jam and similar tools: what works and what does not?

Jam is a triage tool, not a replacement for testing. It is a Chrome extension (also available for iOS) that can:

• record the last minute as a video
• capture console errors, user annotations, and network activity
• use AI to describe the steps a user took

There is a free version (30 reports per month) and a paid one (USD 14 to 16). It also has drawbacks: everything runs in its cloud, so you need to think about data security, and you will usually only catch an edge case if the recording happens to be running at the time.

Tip: Connecting Jam with Sentry takes it to an even higher level.

AI tools for everyday work: ChatGPT, Gemini, or NotebookLM?

Each tool has its place:

ChatGPT is good for thinking things through, but you need to check its output. It is fine for small tasks; for work, Gemini or Claude is a better choice.

Gemini has a huge context window (up to two million words), but it is not as good at working with conversation threads.

NotebookLM is a game changer for me. You upload your sources and it works with them. It is useful for research and as a kind of memory for Gemini or Claude Code. I think it is one of the biggest advances of the past four years.

How do we move from “chatting with a bot” to agentic testing?

MCP is the technological bridge to real agentic testing, where AI can control a developer environment on its own. MCP is a protocol that gives LLMs secure access to local data and tools.

If you do not want to test everything manually, Playwright and its ecosystem (MCP, CLI, Axe, and Chromatic) are the strongest option today.

How are developers different from vibe coders?

You can already see the difference:

• Vibe coders generate code quickly but ignore security and legal considerations. This creates technical debt (“vibe debt”): they produce code they do not understand (the “ownership gap”). When something breaks or the code needs refactoring, they cannot debug it themselves.
• Developers with a framework also show promise, at least until a major update comes along.
• Traditional developers / agentic engineers understand what is happening under the hood. AI can help with simple projects, but complex systems still require deep expertise.

AI can help you with WordPress or a simple project. But complex systems still call for deep knowledge, critical thinking, and a process of your own.

Where do we need human intelligence, and where is AI enough?

Here is an overview of which QA tasks AI can execute, which need a human review (HI Review), and which must remain owned by a person (HI Owns).

Where does AI fall short in QA?

AI currently struggles badly with stateful integrations, complex data orchestration across microservices, and long-running asynchronous dependencies. Without human intelligence (HI) and architectural oversight, this approach can lead to a huge increase in false positives.

Will developer jobs disappear?

Developers will not be replaced; their roles will change. New professions are emerging:

• AI checker, reviews AI output for quality, security, and correctness
• Localization QA / LLM fine-tuning specialist, teaches AI a language and its culture to prevent it from getting things wrong
• AI coordinator / orchestrator, coordinates AI tools and agents and orchestrates an AI sandbox and AIOS
• In QA and SDET: AI orchestrator, AI-driven SDET, AI-augmented quality engineer, and QA machine learning engineer

What is the one rule that matters most?

Learn how to do the sums first; then use a calculator. If you do not understand the basics, AI can easily mislead you.

AI is a tool. It is not a substitute for thinking.

Some more advice from Vojta:

• It is fine to feel behind, the tools, models, methods, and skills are changing in real time.
• If you are not in IT, feel free to code, but think about the impact and the ethics.
• AI is bringing a major shift to QA: agent-based automation, continuous testing throughout the lifecycle, better use of production data, and new tools such as NotebookLM.
• Look for new tools and replace old ones, but keep the core tools that still make sense.

One last question: do you have any hobbies outside QA?

I used to have lots of hobbies, but these days our puppy Ulli takes up most of my free time. We go to obedience and agility training at weekends, and we both enjoy it. I am also working on a platform for dog owners.

Thanks for the interview, Vojta, and all the best!