Quarantine flaky tests with an owner and an exit date

Why it works

Retries stop one flaky failure from failing a build, but the flaky test stays, and when it fails every attempt it blocks pull requests that never touched its code. Quarantine moves a known-flaky test out of the blocking job while it keeps running elsewhere, so a red build means something again and the test keeps producing evidence for whoever fixes it. Slack automated the loop for its mobile suites: detect the flaky test, disable it through a pull request, and file a ticket for the team that owns it.

Quarantine fails in two ways. It can hide real failures: Slack's first version filtered flaky results out of reports and treated tests without history as flaky, so a broken new test reached main. Or it becomes a graveyard of skipped tests nobody owns. An explicit list that names the owner, the issue and an exit date, reviewed like code, prevents both. Atlassian gives each owning team a ticket with a deadline and takes a test out of quarantine after it stays healthy for a set period.

Apply it

1. Find the flaky tests

A flaky test fails and then passes on the same code. If you record retries as in retry narrowly, count how many branches each test needed a retry on:

gh run list --workflow ci.yml --limit 200 --json databaseId,headBranch \
  --jq '.[] | "\(.databaseId) \(.headBranch)"' |
while read -r id branch; do
  gh run download "$id" --pattern 'reruns-*' --dir "reruns/$id" >/dev/null 2>&1 || continue
  find "reruns/$id" -name reruns.txt -exec awk -v b="$branch" '{print $3, b}' {} +
done | sort -u | awk '{print $1}' | sort | uniq -c | sort -rn | head -20

A test that needed retries on two or more unrelated branches is a candidate. Without those files, find first-attempt failures with the query on the flaky tests page, list a run's job IDs with gh run view RUN_ID --attempt 1, and read the failing tests with gh run view --job JOB_ID --log-failed. Before you quarantine, confirm the test is flaky rather than broken: Dropbox reruns a failing test on an earlier commit where it passed.

2. Keep the quarantine in one reviewed file

[
  {
    "test": "tests/test_billing.py::test_webhook_retry",
    "owner": "@acme/payments",
    "issue": "https://github.com/acme/app/issues/1234",
    "expires": "2026-12-07"
  }
]

Save it as quarantine.json, and add /quarantine.json @acme/ci-owners to CODEOWNERS so every change to it needs a review.

3. Skip it in the blocking job, run it in another

jobs:
  test:
    runs-on: ubuntu-24.04
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.13"
      - run: pip install -r requirements-dev.txt
      - name: Test without quarantined tests
        run: pytest $(jq -r '.[].test | "--deselect=" + .' quarantine.json)

  quarantined:
    runs-on: ubuntu-24.04
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.13"
      - run: pip install -r requirements-dev.txt
      - name: Warn about expired entries
        run: |
          jq -r --arg today "$(date -u +%F)" '.[] | select(.expires < $today)
            | "::warning title=Quarantine expired::\(.test) (owner \(.owner), \(.issue))"' quarantine.json
      - name: Run quarantined tests
        continue-on-error: true
        run: |
          tests=$(jq -r '.[].test' quarantine.json)
          if [ -n "$tests" ]; then pytest $tests --junitxml=quarantine.xml; fi
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: quarantine-results
          path: quarantine.xml
          if-no-files-found: ignore

• The test job keeps its name, so branch protection still matches it. With an empty list, it runs everything.
• continue-on-error keeps the quarantined job green while its results go to the artifact. Never make it a required check.
• Also run the workflow on pushes to the default branch and on a nightly schedule, so quarantined tests keep running when pull requests are quiet.
• Take a test out in the pull request that fixes it, once it has passed every quarantine run for about two weeks.

The same list works with other runners:

• Go 1.20 or later: go test ./... -skip '^(TestWebhookRetry|TestCacheEviction)$' in the blocking job, and -run with the same pattern in the quarantine job.
• Playwright: tag the test with { tag: '@quarantine' }, then run --grep-invert @quarantine in the blocking job and --grep @quarantine in the other.
• JUnit 5 with Gradle: add @Tag("quarantine"), and in tasks.test { useJUnitPlatform { ... } } call includeTags("quarantine") when the build gets -Pquarantine and excludeTags("quarantine") otherwise.

Keep it safe

• Required checks. Keep the blocking job's name, and don't add the quarantine job to required checks: it is expected to fail.
• Who can quarantine. A pull request that adds its own failing test to the list would merge green, which is why the file needs a CODEOWNERS review (with "Require review from Code Owners" turned on). If a bot opens quarantine pull requests, run it on pushes to the default branch or on a schedule, never on pull_request_target, and give its token only the permissions it needs. Slack suppresses tests only from main-branch results.
• Flaky, not broken. A test that fails on every attempt on the default branch is broken: fix it or revert the change that broke it. Tests that fail at a date boundary, such as a new month, need a fix rather than a quarantine.
• Lost coverage. A quarantined test protects nothing, and its flakiness may be a real race in the code under test. Keep expiry dates two to four weeks out, cap the list (for example at 1% of tests), and have owners fix or delete rather than extend.
• Silent skips. --deselect matches node ID prefixes, so an entry such as tests/test_billing.py drops the whole file. Use full node IDs, and check that pytest's deselected count equals the number of entries.

Verify

• After two weeks, rerun the baseline. The Reruns column should fall, and so should failed runs on the default branch.
• The test and quarantined jobs together should run as many tests as the test job did before. Compare pytest's passed and deselected counts with the quarantine job's total.
• Every entry should have an owner, an open issue and a future date, and the list should shrink over time.
• Required checks still report under the same names.