AI in QA 2026: What's Real vs. What's Hype

Every test tool now claims AI. We checked the actual product docs behind a dozen named vendors’ claims — not marketing pages, the docs — and found a real spread: some AI is genuinely shipped and well-documented, some is real but narrower than marketed, and some is a rebrand with a public roadmap still listing it as “in progress.”

This isn’t a hit piece on any one vendor — most of what follows is a mix within the same company. It’s a look at how to tell the difference, since “AI-powered” has stopped being a useful signal on its own.

Genuinely real and disclosed

The clearest sign a claim is real: the vendor’s own docs name a specific mechanism, not just the word “AI.” A few examples from our own research building comparison pages this year:

• Tricentis’ Vision AI uses documented deep convolutional neural networks combined with heuristics to recognize UI elements visually rather than by DOM structure — specific enough to work on virtualized and remote apps (Citrix, VMware) where DOM-based tools can’t see anything. That’s a named architecture solving a named problem, including its own self-healing behavior when a UI changes underneath it.
• ReportPortal’s Auto-Analysis and Unique Error Analysis do genuine failure clustering: Auto-Analysis is documented as scoring similarity across roughly 30 features (error message, stack trace, defect-type statistics) backed by an OpenSearch index of previously-triaged results, while a related ML Suggestions feature uses an XGBoost classifier over a larger ~40-feature set to surface similar past failures. Genuinely detailed, open-source, and disclosed in enough depth to evaluate on its merits.
• QA Sphere’s screenshot-to-test-case generation — upload a screenshot, and it scans the visual layout and identifies interactive elements to draft a structured test case. Concrete, demonstrable, and not a stretch of what the phrase implies.

Real, but narrower than the marketing suggests

• “AI-detected flaky tests” doesn’t always mean historical statistical analysis. Currents, a Playwright-focused dashboard tool, defines a flaky test as one that “did not succeed on the first attempt” when retries are enabled — a real and useful signal, but mechanically a retry counter, not independent pattern analysis across a test’s run history. Nothing wrong with retry-based detection; the issue is when it’s presented as equivalent to deeper historical scoring — see test retry strategies for where that line actually sits.
• BrowserStack’s “Test Failure Analysis agent” is a real, shipped product — but its own documentation can’t agree on what powers it. One page describes “sophisticated algorithms” with no AI/ML language at all. Another calls it “machine learning categorization.” A third, newer page says only “AI models analyze the data,” with no architecture ever named. That’s not us being skeptical — that’s three different official pages from the same vendor, at the time of our research, describing the same feature three different ways. The feature works; what exactly it is stays genuinely unclear even after reading the docs closely.

Marketing ahead of what’s actually shipped

• Testiny’s AI is currently limited to an MCP server — letting an external AI assistant read and write test data via natural language. That’s a real, useful integration, but it’s an access layer, not built-in intelligence. Tellingly, Testiny’s own public roadmap lists “improved AI integrations” as a future item, not something already delivered — a rare case where a vendor’s own forward-looking roadmap undercuts their current marketing.
• Tricentis, the same company that ships real Vision AI, also names “Self-healing AI” and “Risk AI” on its editions page. Self-healing AI checks out as part of Vision AI’s own documented behavior — see self-healing tests: how they work, where they fail for the general mechanics and failure modes across the category, beyond this one vendor. Risk AI is a subtler mismatch: it’s real, well-documented technology, but it belongs to Tricentis LiveCompare — a separate, SAP-focused impact-analysis product the company also owns — not Tosca. Naming it alongside Vision AI on a page about Tosca implies a capability Tosca itself doesn’t have.

Confirmed absent, despite what the category implies

Not every test-management tool in this space has any AI at all, and that’s a legitimate, honest choice for a self-hosted open-source project competing on cost and control rather than intelligence — as long as it’s not disguised. Kiwi TCMS has zero AI anywhere in its official docs or years of blog history; the only AI-adjacent mention on the entire site is a skeptical 2019 blog post musing about the concept, not a shipped feature. Confirmed absence isn’t a criticism here — it’s an honest gap the tool doesn’t pretend to fill.

This isn’t unique to testing tools

The pattern above — real capability, narrower-than-marketed capability, roadmap-not-reality capability, honest absence — maps almost exactly onto what analysts are finding across the AI industry generally, not just QA tooling. Gartner’s Anushree Verma put it plainly in a June 2025 press release: “Most agentic AI projects right now are early stage experiments or proof of concepts that are mostly driven by hype and are often misapplied.” Gartner’s own term for the specific failure mode — vendors “rebranding existing products, such as AI assistants, robotic process automation (RPA) and chatbots, without substantial agentic capabilities” — is “agent washing.” Testing tools aren’t a special case of AI hype; they’re one visible instance of an industry-wide pattern.

For the specific case of agentic testing tools — AI agents claiming to own test creation and maintenance end to end — see our deeper breakdown in agentic testing explained, which covers exactly where that category’s claims hold up and where they don’t.

How to actually check, instead of trusting the label

• Look for a named mechanism. “AI-powered” with no further detail is a weaker signal than a docs page willing to say XGBoost, CNN, or “historical pass/fail scoring across N runs.”
• Check if the claim requires history or just a single event. Genuine flaky-test scoring needs pass/fail data across many runs. A feature that only reacts to a single test’s single failure is doing something simpler, even if it’s still useful.
• Read more than one page of their docs. If three different pages describe the same feature three different ways, that’s a real signal worth noticing, not paranoia.
• Check the vendor’s own roadmap, not just their homepage. A roadmap that still lists an already-marketed feature as upcoming is the most honest source a vendor publishes about themselves.

Given that Google reports roughly 1.5% of all its test runs come back flaky, the underlying problem AI-in-testing claims to solve is real and common — which is exactly why it’s worth being precise about which claimed solutions actually solve it.

Start free with Qualflare — check our own failure clustering and predictive flaky scoring against the same standard this post applies to everyone else.

Frequently asked questions

How can I tell if a vendor’s AI claim is real?

Check whether their own documentation names a specific mechanism — a model type, a scoring method, a named algorithm — versus just the word “AI” with no further detail. Real capabilities tend to be over-explained by vendors proud of them; vague ones tend to stay vague no matter how many pages you read.

What is “agent washing”?

Gartner’s term for vendors rebranding existing products — chatbots, robotic process automation, assistants — as agentic AI without delivering substantial new autonomous capability. Gartner’s Anushree Verma described most agentic AI projects as “early stage experiments or proof of concepts that are mostly driven by hype and are often misapplied.” It’s a general software-industry pattern, and testing tools aren’t exempt from it.

Is retry-based flaky detection the same as AI-based flaky detection?

No, and the distinction matters. Retry-based detection means a test is flagged “flaky” because it failed once and passed on a built-in retry — a simple, useful mechanism, but not statistical or historical analysis. AI/ML-based flaky scoring analyzes a test’s pass/fail pattern across many runs over time to produce a confidence score. Both are legitimate, but they’re not the same claim, and vendors don’t always distinguish them clearly.

Does a vendor’s own documentation ever contradict itself on AI claims?

Yes — this happens more often than you’d expect. We found one major vendor whose own docs describe the same AI feature as “sophisticated algorithms” on one page, “machine learning categorization” on another, and just “AI models” with no named architecture on a third. That’s not a hostile reading — it’s what their own current documentation says in three different places.

Does Qualflare’s own AI hold up to this same scrutiny?

We think so, and we’d rather you check than take our word for it: failure clustering groups results by error message, stack trace, and timing correlation; flaky scoring is derived from historical pass/fail variance across runs, not a single retry; release-risk assessment is generated from that same cluster and flakiness data, not a separate black box. All of it is documented in our own docs, the same standard this post holds every vendor to.