Dashboards Aren't Analysis: Why Reporting Isn't Observability

A dashboard can only display what someone thought to graph in advance — which means it structurally can’t answer a question nobody anticipated, like whether a specific test is flaky or whether 40 failures share one root cause. That’s not a tooling gap you close by adding more charts; it’s a category difference. Reporting shows you data. Test observability lets you ask it questions.

We’ve already covered the definitional line between the two — reporting is a single run, observability is history across runs — in what is test observability. This post is about why that line holds up: the actual mechanism by which dashboards fail as an analysis tool, not just a restatement that they’re different things.

A dashboard can only show what someone thought to graph

Charity Majors, CTO of observability company Honeycomb, put this precisely in a January 2025 interview: “Unless your dashboard is dynamic and allows you to ask questions, I feel like it’s a really poor view into your software.” Her point isn’t that dashboards are useless — it’s that a static chart is a pre-committed answer to a pre-committed question. If nobody built a chart for “which of these failures share a root cause,” the dashboard has no way to surface it, no matter how many other charts sit next to it. As she put it: “there are things that you won’t see; because you did not graph it on your dashboard!”

Applied to test results, this is exactly the shape of the problems teams actually hit: nobody predicts in advance that a specific test is about to start flaking, or that three unrelated-looking failures are about to trace back to one broken fixture. Those are questions you discover you need to ask after the failures happen — which a pre-built chart, by definition, can’t be waiting for.

A metric that doesn’t change a decision is theater, not analysis

There’s a sharper version of this same problem: metrics that exist to be displayed rather than acted on. Bryan Finster, quoted by Abi Noda, describes DORA dashboards as often becoming “vanity radiators” instead of “information we can use to help us improve” — teams put the numbers on a screen, watch them move, and never connect a specific number to a specific action. The same failure mode shows up in test dashboards constantly: a pass-rate percentage that ticks up and down with no one asking why, a chart nobody has opened in a decision-making meeting in months. If a number wouldn’t change what your team does next, displaying it isn’t analysis — it’s decoration.

A green pass-rate widget hides exactly the things that matter

This is the sharpest practical failure. A dashboard showing “94% passing” looks reassuring, but it’s an aggregate that erases the two signals that actually determine whether a release is safe:

• Correlation — whether the 6% of failures are one root cause wearing six different stack traces, or six unrelated problems (failure clustering is what surfaces that). A single number can’t distinguish these, and the difference changes how urgently you should treat them.
• Flakiness — whether any of the passing 94% only passed because it happened to succeed this run, and failed last time on unchanged code. A dashboard built from a single run’s results has no memory of the run before it, so a flaky test looks identical to a genuinely stable one until it fails again.

Both of these require historical, cross-run analysis — which is exactly the capability list worth checking for when evaluating a platform that claims to do more than report.

A real vendor conceded this exact line

In May 2025, BrowserStack renamed their “Test Observability” product to “Test Reporting & Analytics.” We can’t speak to their internal reasoning, but the product itself — dashboards, metrics, an AI-assisted failure-analysis add-on — reads more accurately under the new name than the old one. It’s a concrete, public example of a vendor’s own naming decision landing on the same side of this distinction this post is making: what most tools ship is reporting, and calling it observability doesn’t change what it does. Tricentis’ Tosca tells a similar story from a different angle — its own analytics layer, Tricentis Analytics, is described by Tricentis itself as BI templates and Power BI export, not failure correlation or root-cause clustering. Both are real products, not hypotheticals, and both land on the reporting side of this line despite reporting-adjacent names.

What this means for evaluating a tool

Don’t ask a vendor “do you have a dashboard” — every tool in this category does. Ask what happens when you have a question the dashboard wasn’t built for: can it tell you, without you manually cross-referencing three runs yourself, whether a specific test’s failure history means it’s flaky? Can it group today’s failures by shared cause without you reading every stack trace? Those answers require a system built to be queried, not a chart library — the distinction this whole post has been making concrete. The same vanity-vs-actionable question applies beyond dashboards to the metrics teams track generally — see QA metrics that actually matter for which ones actually change a decision.

Start free with Qualflare — connect your CI pipeline and see failure clustering and flaky scoring work on your own test history, not just a pass-rate chart.

Frequently asked questions

What’s the difference between a dashboard and analysis?

A dashboard displays predefined metrics you decided to track in advance — pass rates, counts, durations. Analysis answers a question you didn’t anticipate: why did these 40 tests fail together, is this specific test flaky, is this release actually safe to ship. A dashboard can only show what someone thought to graph; analysis can respond to a question nobody asked yet.

Why can’t you just add more charts to a dashboard to get analysis?

Because the problem isn’t coverage, it’s structure. Every chart still requires someone to have predicted the question in advance. You can’t pre-build a dashboard for “which of these failures are actually the same underlying bug” — that requires correlating results across runs and tests in real time, which is a different kind of system than a chart library.

What is a vanity metric?

A metric that looks good or busy on a dashboard but doesn’t change what anyone does. DORA metrics dashboards, for example, get criticized as “vanity radiators” when teams display them without using them to identify or drive actual improvements — the number moves, but no decision follows from it.

Can a passing test dashboard still be hiding real problems?

Yes. A pass-rate dashboard shows an aggregate number, not the pattern underneath it. It won’t tell you that one test has been intermittently failing and passing for three weeks (a flaky test), or that five failures in different files actually trace back to one broken fixture. Both require analyzing history and correlating results — neither shows up as a single chart.

Why did BrowserStack rename Test Observability to Test Reporting & Analytics?

BrowserStack made this change in May 2025. We can’t speak to their internal reasoning, but the renamed product still centers on dashboards and metrics rather than cross-run failure correlation — so at minimum, the new name is a more accurate description of what the product actually does, and a real-world data point that the reporting/observability distinction is significant enough for a major vendor to revisit their own naming over it.