Balance shards by recorded test timings

Why it works

A sharded job ends when its slowest shard ends. Jest and Vitest hand each shard the same number of files, picked by a hash of the path. Playwright hands out equal numbers of files, or of tests with fullyParallel. pytest-split without timings hands out equal numbers of tests. None of them looks at duration, so one shard can draw the three slowest files and run twice as long as the others while they sit idle.

Recorded timings fix the split. Sort the tests longest first and give each to the shard with the smallest total so far, and the shards finish close together. pytest-split does this with --splitting-algorithm least_duration; Jest and Vitest let you replace their shard() method. The same imbalance shows up between workers inside one job, which is what Ramp fixed for pytest-xdist.

Two limits stay. Jest and Vitest move whole files, so a file that takes longer than the total ÷ N sets the floor until you split it. And every shard must compute its split from the same timings file. If two shards read different versions, one test can run twice while another runs nowhere.

Apply it

Measure the spread first. Run this for three to five recent runs:

gh run view RUN_ID --json jobs --jq '.jobs[] | select(.name | startswith("test (")) | [.name, ((.completedAt | fromdateiso8601) - (.startedAt | fromdateiso8601))] | @tsv'

If the slowest shard leads by less than about 15%, stop here. Otherwise split the files that run longer than the total ÷ N, then add timings.

pytest

jobs:
  timings:
    runs-on: ubuntu-latest
    timeout-minutes: 5
    outputs:
      key: ${{ steps.lookup.outputs.cache-matched-key }}
    steps:
      - id: lookup
        uses: actions/cache/restore@v6
        with:
          path: test-durations.json
          key: test-durations-${{ github.run_id }}
          restore-keys: test-durations-
          lookup-only: true

  test-shard:
    needs: timings
    name: test (${{ matrix.shard }}/${{ strategy.job-total }})
    runs-on: ubuntu-latest
    timeout-minutes: 20
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.13"
          cache: pip
      - run: pip install -r requirements-dev.txt pytest-split
      - if: ${{ needs.timings.outputs.key != '' }}
        uses: actions/cache/restore@v6
        with:
          path: test-durations.json
          key: ${{ needs.timings.outputs.key }}
          fail-on-cache-miss: true
      - name: Run shard
        run: |
          pytest --splits ${{ strategy.job-total }} --group ${{ matrix.shard }} \
            --splitting-algorithm least_duration \
            --durations-path test-durations.json --store-durations --clean-durations \
            --junitxml=reports/junit-${{ matrix.shard }}.xml
          mv test-durations.json reports/durations-${{ matrix.shard }}.json
      - if: ${{ !cancelled() }}
        uses: actions/upload-artifact@v7
        with:
          name: junit-shard-${{ matrix.shard }}
          path: reports/
          retention-days: 7
          overwrite: true

  save-timings:
    needs: test-shard
    if: ${{ github.event_name == 'push' && github.ref_name == github.event.repository.default_branch }}
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/download-artifact@v8
        with:
          pattern: junit-shard-*
          path: reports
          merge-multiple: true
      - run: jq -s 'add' reports/durations-*.json > test-durations.json
      - uses: actions/cache/save@v6
        with:
          path: test-durations.json
          key: test-durations-${{ github.run_id }}-${{ github.run_attempt }}

• The timings job looks up the newest saved timings without downloading them and passes the exact key on. Every shard restores that one entry, so a save that lands mid-run can't give shards different splits. If the shards already need a build job, do the lookup there.
• With no saved timings yet, the restore is skipped and pytest-split splits by count.
• --clean-durations makes each shard write only the tests it ran, so jq -s 'add' joins the shard files into one complete set.
• save-timings runs only on default-branch pushes where every shard passed (a job without a status function in if requires its needs to succeed). Keep the aggregate test job from Shard tests as the required check.

The simpler alternative is pytest-split's default workflow: commit the durations file and refresh it by hand when the split drifts. Every shard reads the same commit.

Vitest or Jest

Replace the sequencer's shard() and register it with test: { sequence: { sequencer: TimingSequencer } } in vitest.config.ts:

import { existsSync, readFileSync } from 'node:fs'
import { relative } from 'node:path'
import { BaseSequencer, type TestSpecification } from 'vitest/node'

const timings: Record<string, number> = existsSync('test-durations.json')
  ? JSON.parse(readFileSync('test-durations.json', 'utf8'))
  : {}

export class TimingSequencer extends BaseSequencer {
  async shard(files: TestSpecification[]): Promise<TestSpecification[]> {
    const { index, count } = this.ctx.config.shard!
    const known = Object.values(timings)
    const average = known.length ? known.reduce((a, b) => a + b, 0) / known.length : 1
    const weighted = files
      .map((spec) => {
        const path = relative(this.ctx.config.root, spec.moduleId)
        return { spec, id: `${spec.project.name} ${path}`, seconds: timings[path] ?? average }
      })
      .sort((a, b) => b.seconds - a.seconds || (a.id < b.id ? -1 : 1))
    const totals = new Array<number>(count).fill(0)
    const mine: TestSpecification[] = []
    for (const { spec, seconds } of weighted) {
      const target = totals.indexOf(Math.min(...totals))
      totals[target] += seconds
      if (target === index - 1) mine.push(spec)
    }
    return mine
  }
}

Use the same workflow as above, minus the pytest flags. In save-timings, build the file from the shards' JUnit reports, where each <testsuite> is one file named by its path from the project root:

python3 - <<'EOF'
import glob, json, xml.etree.ElementTree as ET
timings = {}
for path in glob.glob("reports/junit-*.xml"):
    for suite in ET.parse(path).getroot().iter("testsuite"):
        timings[suite.get("name")] = float(suite.get("time"))
json.dump(timings, open("test-durations.json", "w"), indent=1, sort_keys=True)
EOF

Jest works the same way: extend @jest/test-sequencer, override shard(tests, { shardIndex, shardCount }), key on path.relative(rootDir, test.path) and set JEST_JUNIT_SUITE_NAME: "{filepath}".

Playwright

Playwright has no public option for timing-based sharding. Set fullyParallel: true so it balances individual tests instead of files, and split spec files whose tests must run in order.

Keep it safe

• Never restore timings in the shards with restore-keys. Each shard would take whatever entry is newest when it starts.
• Tests missing from the timings file (new or renamed) must still be assigned. pytest-split and the sequencer above give them the average duration. Keep the exactly-once check from Shard tests, because a sequencer bug skips tests silently.
• pytest-split's default duration_based_chunks algorithm breaks with pytest-randomly or pytest-random-order: shards can overlap or miss tests. Use least_duration.
• Save timings only from default-branch pushes. Pull request caches are scoped to their pull request anyway, and timings affect only balance, not which tests run.
• Cache entries unused for 7 days are evicted. Shards then fall back to count-based splits until the next default-branch run saves new timings.

Verify

• Over three to five runs, the slowest shard should finish within about 15% of the fastest. Its p50 should be below the old slowest shard's p50 from the baseline.
• The aggregate job's test count should match the count before the change.
• gh cache list --key test-durations- should show a new entry after each default-branch push.