Skip to content
compiler.dev
Start free

What Is a Flaky Test? Definition and Causes

Last updated

A flaky test is a test that sometimes passes and sometimes fails on exactly the same code, with no change to the test or the application. Because the result is not deterministic, a failure tells you nothing reliable, and teams learn to ignore red builds.

How it works

Flakiness comes from something the test does not control. The test is correct under some conditions and wrong under others, and CI hits the wrong ones occasionally. Common sources:

  • Timing and races. Fixed sleeps, async code not awaited, UI elements clicked before they render.
  • Shared state. Tests that depend on order, leave data behind, or share a database with parallel tests. Test sharding often exposes this.
  • External services. Real network calls, rate limits, third-party sandboxes.
  • Time and randomness. Tests that depend on the current date, time zones, or unseeded random numbers.
  • Resource limits. An under-powered runner causes timeouts that never happen on a laptop.
  • Environment drift. Different versions of browsers, locales or dependencies between runs.

Detecting flakiness

  1. Rerun and compare. If a failure passes on retry with no code change, it is flaky.
  2. Run the test many times. A loop of 50 to 100 runs often reproduces it.
  3. Track history. Record pass/fail per test per commit. A test that flips on the same commit is the culprit.
  4. Randomise order. Many runners can shuffle tests to surface order dependence.

Example

Playwright can retry in CI and keep traces for the failures so you can investigate:

- run: npx playwright test --retries=2 --trace=on-first-retry
- uses: actions/upload-artifact@v4
  if: failure()
  with:
    name: playwright-traces
    path: test-results/

Retries make the build green but hide the problem. Treat a test that needed a retry as a bug to fix. Deeper guides: flaky tests and flaky end-to-end tests.

Common mistakes

  • Retrying forever. Automatic retries mask flakes and inflate CI minutes.
  • Adding longer sleeps. It reduces the rate, but never removes the race. Wait on a condition instead.
  • Deleting the test. You lose coverage. Quarantine it into a separate non-blocking job, file an issue, and fix it.
  • Ignoring the cost in a merge queue. One flake ejects a pull request and restarts those behind it.
  • Blaming the CI. Sometimes the machine is the trigger, but the test is still the bug.

Related terms

FAQ

How common are flaky tests?

Large codebases nearly always have some. Rates vary, so measure your own pass/fail history rather than trusting an industry number.

Should I retry flaky tests in CI?

Use retries as a stopgap with reporting, so each retried test is visible and gets fixed.

Can slow runners cause flakiness?

Yes. Timeouts that are tight on a 2-core machine may pass on a faster one, but the real fix is waiting on conditions instead of time.

Related