Retry only the test or step that failed, and record the retry

Why it works

A flaky failure and a real one look the same the first time. Running the failed test again, in the same job against the same build, tells them apart in seconds: a real bug fails again, and a flake usually passes. Rerunning the whole workflow answers the same question at the cost of a full run. Broad retries also waste work on failures that can't recover: Bun found that most automatic retries of timed-out test shards timed out again.

Keep the attempts few and fixed, and treat every pass on retry as a finding. Shopify retries a failed test up to three times and tells developers which tests passed only after a retry. GitHub retries a failed test three ways (same process, clock moved forward, another host) to tell chance, time and test order apart, and assigns high-impact flakes to the people who last changed them. Without that record, retries hide a flaky test until it fails every attempt.

Apply it

Retry failed tests in the test runner

Use the runner's own retry, so only the failed tests run again:

Runner	Retry only failed tests	Retried tests show up in
pytest	pytest-rerunfailures: --reruns 2	RERUN lines with -rR
Playwright	retries: process.env.CI ? 2 : 0 in the config	the flaky status
Jest	jest.retryTimes(2, { logErrorsBeforeRetry: true }) in the test file	errors logged before each retry
Go	gotestsum --rerun-fails=2 --rerun-fails-report=reruns.txt --packages=./...	the report file
Gradle	org.gradle.test-retry: retry { maxRetries = 2; maxFailures = 20 }	one report entry per execution
Maven	-Dsurefire.rerunFailingTestsCount=2	flakyFailure in the XML

A complete pytest job that also records which tests needed a retry:

jobs:
  test:
    runs-on: ubuntu-24.04
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.13"
      - run: pip install -r requirements-dev.txt
      - name: Test, retrying failed tests up to twice
        shell: bash
        run: pytest --reruns 2 --reruns-delay 1 --max-suite-reruns 20 -rR | tee pytest.log
      - name: Record tests that needed a retry
        if: always()
        shell: bash
        run: |
          grep '^RERUN ' pytest.log | sort | uniq -c > reruns.txt || true
          if [ -s reruns.txt ]; then
            echo "::warning title=Flaky tests::$(wc -l < reruns.txt) test(s) were retried"
            { echo '### Tests that needed a retry'; echo; sed 's/^/    /' reruns.txt; } >> "$GITHUB_STEP_SUMMARY"
          fi
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: reruns-${{ github.job }}-${{ strategy.job-index }}
          path: reruns.txt
          if-no-files-found: ignore

• Pin pytest-rerunfailures in requirements-dev.txt like any other test dependency.
• shell: bash adds -o pipefail. Without it, pytest | tee reports success even when tests fail.
• --max-suite-reruns 20 stops retrying when many tests fail at once, which signals a real break. gotestsum (--rerun-fails-max-failures, 10 by default) and Gradle (maxFailures) have the same guard.
• strategy.job-index keeps artifact names unique across matrix jobs. These files feed quarantine.

Retry a failed infrastructure step

Shopify retries failed setup (loading a container, installing dependencies, git fetch) inside the job instead of starting over. This run script for a service-start step tries three times and warns when it needed a retry:

for attempt in 1 2 3; do
  if docker compose up --wait --wait-timeout 120; then
    [ "$attempt" -gt 1 ] && echo "::warning title=Retried setup::services started on attempt $attempt"
    exit 0
  fi
  docker compose down --volumes
  sleep $((attempt * 15))
done
exit 1

For downloads and checkout, use the stall limits in resilient network calls.

Never retry these

• Whole workflows or jobs. Automatic reruns double runner minutes and hide flakes. Rerun a job automatically only when its runner was lost and it produced no result, and only once, as Bun's rules do. Otherwise a person runs gh run rerun RUN_ID --failed.
• Timeouts. A test or job that hit its limit is hung or too slow, and a retry usually ends the same way. See job timeouts.
• Deterministic checks such as compiling, type checks, lint and formatting.
• Steps with side effects: deploys, migrations, publishing and notifications. A retry after a partial success can apply the change twice.

Keep it safe

• Required checks. Check names don't change, and a test that fails every attempt still fails the job. Don't fake a retry with continue-on-error, which lets the job pass when the step fails.
• Flakes still need fixing. A retry can hide a real race in the code under test. Google found that failing a flaky-marked test only after three failures in a row encouraged people to ignore flakiness. Keep attempts at two or three, read the reruns list weekly, and quarantine repeat offenders.
• Shared state. pytest-rerunfailures keeps module- and session-scoped fixtures between attempts. A test that passes only on its second attempt may depend on test order; see isolate test state.
• Security. A retry reruns the same code with the same token and secrets. Don't add a workflow_run rerun bot with actions: write that acts on fork pull requests, don't set -x in loops that handle tokens, and never skip TLS or checksum checks to get a download through.
• Time budget. Size timeout-minutes for the first run plus a few retried tests, not a second full run.

Verify

• After one or two weeks, rerun the baseline. The Reruns column for the changed workflows should fall, and Success should rise.
• If the baseline showed reruns, the reruns files shouldn't stay empty.
• The test job's p50 wall time should barely move; its p90 can rise slightly on runs that retried.
• The first attempt should run as many tests as before (pytest's summary line, gotestsum's DONE line), and every required check should report under its old name. Gradle and Surefire list each execution, so count unique test names.