Benchmark the long pole on a bigger runner

Why it works

A CPU-bound job finishes sooner on more cores only when its tools split the work: compilers with parallel jobs, Cargo, Gradle and Bazel workers, test runners with worker pools, type checkers that build several projects. Memory matters as well. On 8 GB, bundlers, tsc and Gradle daemons can swap, crash or limit their own parallelism. Newer CPUs and faster disks help even at the same core count; Linear and Spotify both credit faster hardware for their gains.

The price per minute usually rises about in step with cores. If doubling the cores halves the job's time, cost per run stays flat and the wait halves. If it saves less than half, each run costs more. Every job also has serial parts that more cores can't touch, so gains flatten: OpenMeter's build barely moved from 16 to 32 cores. GitHub-hosted runners also round each job up to a whole minute, which matters for short jobs. You can't predict where your job's curve flattens, so measure it.

Apply it

Pick the job and the candidates

Take the long-pole job from your baseline and pick two or three sizes: the current one, double and quadruple. The options include:

• GitHub larger runners (Team and Enterprise Cloud plans): Linux from 2 vCPU and Windows from 4, up to 96, plus larger macOS machines. An organization admin creates them, and you target them by runner name or runner group.
• Third-party hosted runners, which you add by installing the provider's GitHub App and changing runs-on. BerryBox (berrybox.io), for example, uses labels such as berrybox-8vcpu-ubuntu-2404.
• Self-hosted runners on hardware you choose, for example with Actions Runner Controller on Kubernetes.

Benchmark on the same commit

Push a temporary workflow on a branch of its own. It runs the long-pole job's steps unchanged on each candidate, three times each:

name: Runner benchmark
on:
  push:
    branches: [runner-benchmark]

jobs:
  bench:
    strategy:
      fail-fast: false
      matrix:
        # Use labels that exist for your account: a larger runner name,
        # a provider label or a self-hosted label.
        runner: [ubuntu-latest, ubuntu-24.04-8core, ubuntu-24.04-16core]
        attempt: [1, 2, 3]
    runs-on: ${{ matrix.runner }}
    timeout-minutes: 60
    steps:
      - uses: actions/checkout@v7
      # Paste the long-pole job's steps here, unchanged.
      - run: make test

• A push to the runner-benchmark branch runs every matrix job on the same commit. A workflow_dispatch trigger would only work once the file is on the default branch.
• fail-fast: false keeps one failing size from stopping the others.
• The attempt axis gives you three samples per size in one run, so you can compare medians.
• Run it with the job's caches warm, the way pull requests see it.

Then read each job's duration:

gh run view RUN_ID --json jobs \
  --jq '.jobs[] | "\(.name)\t\((.completedAt | fromdate) - (.startedAt | fromdate))s"'

For each size, take the median duration and compute cost per run: minutes, rounded up per job if the provider bills that way, times that runner's per-minute price from the provider's pricing page. Record queue time separately, because a pool that starts cold can hide a fast machine.

Decide

Pick the size where the next step up stops cutting time by close to half, unless the wait matters more to your team than the extra cost. Switch only the long-pole job, and match its worker counts to the new core count (match workers to cores). Delete the benchmark branch.

Keep it safe

• Images differ. GitHub's larger runners can use the same images as standard ones. Other providers and self-hosted runners may ship different tools, versions and users. Set up toolchains with setup actions instead of relying on what's preinstalled, and compare test counts.
• Check names. Changing runs-on keeps the check name unless the runner label appears in it, as in a matrix job named test (ubuntu-latest). Update branch protection in the same change if it does.
• Labels with no runner. A job whose label no runner serves waits in the queue instead of failing. Create the runner and grant the repository access before you merge.
• Security. Third-party and self-hosted runners see your code and any secrets the job uses. Prefer ephemeral runners that take one job and are destroyed, and never let fork pull requests run on persistent self-hosted runners.
• Cost. GitHub's larger runners are not free in public repositories, where standard runners cost nothing. Set a budget before you switch.

Verify

• Over a few days of pull requests, compare the long pole's p50 and p90 duration and the run's p50 wall time with the baseline.
• Compare cost per run using each job's billed minutes and rate.
• Check queue time on the new label, and confirm the same tests ran and the same checks reported.