Test Retry Strategies: When Retries Help vs. Hide Bugs (2026)

A test retry is safe when the failure sits outside your code — a slow third-party API, a DNS blip, noise on a shared CI runner. It’s dangerous when it wraps an assertion on business logic, because a genuine intermittent bug can pass on retry exactly as often as a harmless flake does. Both look identical in a CI log: green after one failed attempt. The only way to tell them apart is to reason about what failed, not just that it eventually passed.

Our complete guide to flaky tests and quarantine strategy guide already cover why retries hide flakiness instead of fixing it. This post is the operational layer on top: concrete criteria for when a retry is the right call, the math behind tuning retry counts, and current configuration for four major frameworks.

When is a retry safe, and when does it hide a bug?

A test retry automatically re-runs a failed test a set number of times and passes it if any attempt succeeds. Whether that’s safe isn’t about the test — it’s about the failure mode. Ask what would have to be true in production for this specific assertion to fail intermittently.

• Retry-safe: the assertion is correct and the flake comes from something you don’t control — a third-party API rate limit, a slow DNS lookup, a shared CI runner under load. Retrying re-runs the same correct check against an environment that’s occasionally unreliable.
• Retry-dangerous: the assertion checks business logic — an order’s status, a computed total, a database write, a state transition after a webhook — where an intermittent failure is exactly the shape a real race condition or concurrency bug takes. Google’s own engineers noted this in discussing their rerun mechanism: a software bug doesn’t have to fail every run to be real — it can pass most of the time and still be a defect.

A useful shorthand: retry the call, not the assertion. If a step genuinely depends on an unreliable external system, isolate the retry to that call or test file, not to every assertion in the suite. Broad, suite-wide retry settings are the ones most likely to bury a real bug, since they apply the same leniency to a flaky network call and a checkout total in the same run.

A decision framework for retry-safe vs. retry-dangerous failures

Failure signature	Retry appropriate?	Why
Third-party API timeout or rate limit	Yes	External flakiness, well outside your code path
DNS blip or transient network error on a shared CI runner	Yes	Infrastructure noise, not application logic
Element-not-visible race in an E2E flow	Retry as a bridge, but fix the wait	Often a real timing gap; a proper wait condition is the actual fix
Assertion on a computed value (price, total, balance)	No	An intermittent failure here can be a genuine race condition
Database state check after a write or webhook	No	The exact shape a real concurrency bug takes
Failure tied to test order or shared state	No — fix isolation	Retrying doesn’t touch the ordering bug, it just outruns it

This mirrors Google’s SRE guidance on retries for production systems: retries are a legitimate tool for absorbing transient failures, but they need limits — a bounded budget, backoff, a clear scope — so they don’t paper over a failure that keeps happening for a real reason. A test suite needs the same discipline, applied to failure reasons instead of failure rates.

How many retries should you configure? The masking-probability math

Retry count is a probability trade-off, and the math is worth doing once so it stops being a guess. If a real bug fails independently on some fraction of runs — say q, its per-run failure probability — then the chance that at least one of r retries (r + 1 total attempts) happens to pass, and the bug goes unnoticed, is:

P(masked) = 1 − q^(r+1)

For a bug that fails 60% of the time (3 of 5 runs, q = 0.6):

Configured retries	Total attempts	Chance the bug still goes green
0 (no retry)	1	40%
1	2	64%
2	3	78%
3	4	87%
5	6	95%

Every extra retry compounds the risk rather than just adding to it: two retries almost double your odds of missing a bug that fails well over half the time, and five make it nearly certain the bug ships silently.

Too few retries isn’t free either. In an industrial case study of a ~30-developer, ~1M-line codebase spanning five years, Leinen et al. found that automatically rerunning a failed test costs about 0.02 cents, versus $5.67 for the manual investigation a false failure triggers — roughly a 28,000× difference. Zero retries on a test with well-understood, genuine external flakiness doesn’t protect anything; it just moves the cost onto an engineer manually re-triggering CI.

The tuning target isn’t one “right” number — it’s matching retry count to failure mode: enough to absorb documented infrastructure noise, none at all where a repeat pass doesn’t tell you the bug is gone.

How do you configure retries in Playwright, Jest, Cypress, and pytest?

The concept is universal; the mechanics and defaults differ per framework.

Framework	How you set it	Granularity	Filters by failure type?
Playwright	retries in config, --retries=N CLI, test.describe.configure({ retries })	Global, per-project, per-describe block	No — retries on any failure
Jest	jest.retryTimes(n, options)	Per spec file or describe block	No
Cypress	retries: { runMode, openMode } in config, per-suite or per-test override	Global, per-suite, per-test	No
pytest-rerunfailures	--reruns N CLI, @pytest.mark.flaky(reruns=N)	Global (CLI) or per-test (decorator)	Yes — --only-rerun / --rerun-except by exception type

Playwright

// playwright.config.ts
export default defineConfig({
  retries: process.env.CI ? 2 : 0, // no retries locally — catch real flakiness on your own machine
});

Scope it by suite instead of applying one number everywhere:

test.describe('checkout — 3rd-party shipping-rate API', () => {
  test.describe.configure({ retries: 2 }); // known external dependency
});

test.describe('checkout — payment webhook updates order status', () => {
  test.describe.configure({ retries: 0 }); // business-logic assertion, must be exact
});

testInfo.retry is available in any test, hook, or fixture to branch on which attempt is running, per Playwright’s docs.

Jest

// at the top of a spec file, or in a global setup file
if (process.env.CI) {
  jest.retryTimes(2, { logErrorsBeforeRetry: true });
}

Jest can’t filter retries by error type or scope them below the file level — if a spec file needs retries, every test in it gets them. Keep retry-eligible specs separate from ones asserting business logic.

Cypress

// cypress.config.js
module.exports = defineConfig({
  retries: {
    runMode: 2,
    openMode: 0,
  },
});

Override per test for an assertion you don’t want retried:

it('marks the order as paid immediately after the webhook fires', {
  retries: { runMode: 0, openMode: 0 },
}, () => {
  // ...
});

pytest-rerunfailures

pytest --reruns 2 --reruns-delay 1 --only-rerun "ConnectionError|TimeoutError"

@pytest.mark.flaky(reruns=3, only_rerun=["requests.exceptions.ConnectionError"])
def test_fetches_shipping_rate_from_carrier_api():
    ...

This is the sharpest tool of the four: --only-rerun (and its inverse, --rerun-except) reruns a test only when the failure matches a specific exception type. That’s “retry the call, not the assertion” enforced by the framework itself — a test gets a second chance on a connection error, not on a failed business-logic assertion.

What does a retry-masked bug actually look like?

The following is illustrative, not a documented public incident, but it’s a common shape. A team sets Playwright’s retries: 2 in CI to quiet a checkout suite that started failing intermittently after a migration to a new payment-webhook handler. The failing test asserted that an order’s status flips to paid within three seconds of the webhook firing. Under two retries, it failed on the first attempt roughly 30% of the time, then passed on the second or third nearly every time — CI stayed green, and nobody looked twice.

Three weeks later, a support ticket surfaced: a handful of real customer orders were stuck in pending after payment had actually succeeded. The root cause was a race between the webhook handler and an async inventory-lock step that occasionally lost — the exact race the test had been catching, intermittently, the whole time. The retry hadn’t fixed anything; it had quietly re-run the race until it won, and the one signal that would have caught it in review was absorbed by a CI setting nobody revisited after the migration shipped.

How do retries relate to quarantine and AI-based flaky detection?

Retry and quarantine solve different problems, and conflating them is how a suite quietly accumulates unreliable tests: retry is a live, per-build decision that hides one bad run; quarantine is a tracked, time-boxed removal from the release gate while someone fixes the root cause. Our quarantine strategy guide covers sizing that SLA and the exact skip syntax per framework.

It’s also worth not mistaking a framework’s retry counter for genuine flaky test detection: a tool that calls a test “flaky” because it failed once and passed on retry is reporting a single event, not analyzing history — a distinction covered in AI in QA 2026: what’s real vs. what’s hype. Genuine detection scores a test’s flakiness from pass/fail variance across many runs — the same history that turns “this test needed a retry” from a shrug into a tracked signal, and one of the clearest early signs that non-determinism is creeping into a corner of the suite.

Start free with Qualflare — send your CI results and see which tests are quietly leaning on retries to pass, scored from the same run history Qualflare uses for flakiness and failure clustering.

Frequently asked questions

When is it safe to retry a failing test?

When the failure mode is a known, external, non-deterministic dependency unrelated to the code path you’re testing — a third-party API timeout, a rate limit, a DNS blip, or infrastructure noise on a shared CI runner. In those cases a retry re-runs the same correct assertion against an environment that’s simply less reliable than your application code.

When does a retry mask a real bug?

When it wraps an assertion on business logic — order status, computed totals, state transitions, data written to a database — where an intermittent failure could be a genuine race condition or concurrency bug in your product. A real bug doesn’t have to fail every run to be real; retrying past it just re-rolls the dice until it passes.

How many retries should I configure?

Enough to absorb known infrastructure noise, not enough to reliably paper over a probabilistic bug. As a rule of thumb, 1–2 retries on tests with a documented external dependency is common; anything higher sharply increases the odds that a real intermittent failure goes unnoticed, since the chance of masking rises with every extra attempt.

How do I retry only network failures, not assertion failures?

Of the frameworks covered here, only pytest-rerunfailures supports this natively, via --only-rerun (or the only_rerun marker argument), which reruns a test only when it fails with a matching exception type or message pattern — for example ConnectionError or TimeoutError. Playwright, Jest, and Cypress retry on any failure reason, so the discipline has to be applied by scoping which test files or suites get retries at all.

How do I configure retries in Playwright, Jest, Cypress, and pytest?

Playwright sets retries in playwright.config.ts (or --retries=N on the CLI, or per test.describe.configure({ retries })). Jest uses jest.retryTimes(n, options) called at the top of a spec file. Cypress sets retries: { runMode, openMode } in cypress.config.js, overridable per suite or test. pytest-rerunfailures uses --reruns N on the CLI or @pytest.mark.flaky(reruns=N) per test. Full snippets are in the configuration section above.

What’s the difference between retrying a test and quarantining it?

Retry re-runs a single failed attempt within the same build and reports a pass if any attempt succeeds — it’s a live, per-run decision. Quarantine removes a chronically flaky test from the release gate entirely, for a tracked period, while someone investigates the root cause. Retry hides one bad run; quarantine manages a known-bad test on a deadline.