Run new and changed tests many times before they can block merges

Why it works

A flaky test is cheapest to fix before it merges, while its author still has the context and before it has blocked anyone else. One green run in review proves little, because a test that fails one run in 50 will usually pass it. Running only the tests a pull request adds or edits, many times in a row, costs minutes rather than a full suite, and it catches the timing, ordering and shared-state bugs that new tests often carry.

Teams that do this put the check where the author will see it. Mozilla runs each new or modified test 10 times, then 5 more times in a new browser instance each time, then repeats the sequence in chaos mode, within about an hour per task. Spotify's Flakybot lets an engineer run a pull request's tests and get a flakiness report before merging. Shopify's mobile team admits a new end-to-end test to blocking CI only after a dedicated pipeline has run it multiple times under a failure threshold. Uber ran a repaired integration test 50 times with Bazel's --runs_per_test before merging the fix. Repetition finds the frequent flakes. Rare ones still slip through, so keep narrow retries and quarantine as the backstop.

Apply it

Add a workflow that lists the test files a pull request adds or edits and runs them repeatedly with retries off:

name: Changed test stability

on:
  pull_request:

permissions:
  contents: read

jobs:
  repeat-changed-tests:
    runs-on: ubuntu-24.04
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 2
      - name: List added or edited test files
        id: changed
        run: |
          files=$(git diff --name-only --diff-filter=AM HEAD^1 HEAD -- tests/ \
            | grep -E '(^|/)test_[^/]*\.py$' | head -n 20 | tr '\n' ' ')
          echo "files=$files" >> "$GITHUB_OUTPUT"
      - uses: actions/setup-python@v7
        if: steps.changed.outputs.files != ''
        with:
          python-version: "3.13"
      - name: Install dependencies
        if: steps.changed.outputs.files != ''
        run: pip install -r requirements-dev.txt
      - name: Run each changed test 30 times, retries off
        if: steps.changed.outputs.files != ''
        env:
          FILES: ${{ steps.changed.outputs.files }}
        run: pytest -p no:rerunfailures --count=30 -x $FILES

• In a pull_request run, HEAD is GitHub's merge commit and HEAD^1 is the base branch, so fetch-depth: 2 is enough for the diff. --diff-filter=AM keeps added and modified files and skips deleted ones.
• head -n 20 caps the work when a pull request rewrites many tests at once.
• --count=30 comes from pytest-repeat, which belongs in requirements-dev.txt. -x stops at the first failure, and -p no:rerunfailures turns off retries that would hide the flake you are looking for.
• File names come from the pull request, so they reach the shell through env, not through ${{ }} inside the script.
• The step-level if conditions keep the job green when no tests changed.

Go

Run every test in the packages whose _test.go files changed, in random order and with the race detector:

pkgs=$(git diff --name-only --diff-filter=AM HEAD^1 HEAD -- '*_test.go' | xargs -r -n1 dirname | sort -u | sed 's|^|./|')
if [ -n "$pkgs" ]; then go test -count=20 -shuffle=on -race $pkgs; fi

Playwright

With the same fetch-depth: 2 checkout, npx playwright test --only-changed=HEAD^1 --repeat-each=20 --retries=0 runs the test files changed against the base branch 20 times each. --retries=0 overrides the retries in your config.

Start the job as an optional check. After two weeks without false alarms, add it to the required checks, so the pull request that adds a flaky test is the one it blocks.

Keep it safe

• Required checks. Once required, the job must report on every pull request. Keep the step-level if conditions rather than a trigger-level paths filter, which would leave the check pending forever on pull requests without test changes. Renaming the job later breaks branch protection.
• Security. Use pull_request, never pull_request_target: this job runs the pull request's own code, so it gets a read-only token and no secrets. Never interpolate file names into run with ${{ }}, where a crafted name could inject shell commands.
• Cost. Repeats multiply runtime. Keep the file cap and timeout-minutes. Tests that call real services 30 times can hit rate limits; fake those services first (fake slow services).
• Blind spots. Repeating a test in one process can miss failures that depend on other test files or on order. Shuffle where the runner supports it, and see isolate test state. pytest-repeat runs unittest.TestCase classes only once, with a warning.
• Coverage. This job adds a check. The normal test job still runs every test.

Verify

• Open a throwaway pull request that adds a test failing about one run in five, such as assert random.random() > 0.2. The new check should fail. Close the pull request.
• Over the next month, look at where new quarantine entries come from. Fewer of them should be tests added in the last few weeks.
• In the baseline, the new job should only take minutes, and only on pull requests that touch tests.
• The main test job should run the same number of tests as before, and every required check should report under its old name.