The full suite ran at 1, 2, 3, 4, 6 and 8 workers, many times over, on the local dev box and on GitHub's CI runner. Wall clock stops improving long before the flake rate stops rising. This page records where that crossover sits on each machine, and what it took to tell real test bugs apart from CPU contention.
The answer
On the 4-core dev box. Runs finish 10.2s later than at 4 workers, and went red 1 time in 16 instead of 4 in 8.
On the 4-vCPU runner. The fastest setting measured, at 66.5s a run, and it went red 1 time in 35 against 3 in 35 for 3 workers.
1 red run in 35 is 2.9%, but the 95% confidence interval reaches 12.9%. That is not enough evidence to drop a retry.
The two machines want different settings because a red run costs different things. Locally there are no retries, so one flaky test means running the suite again and stopping to look at why. That costs far more than the 10 seconds a higher worker count saves, so the local count stays where the machine is not oversubscribed. On CI, retries make one flaky test cheap, so the fastest count wins as long as its red-run rate is no worse than the alternatives.
One finding held on both machines: a Playwright worker needs about 2 cores.
A worker is a whole Chromium (browser, renderer, GPU, network and utility processes) plus a
Node runner. Total CPU time per run grew in step with workers ÷ 2 on both machines, which
put the cost at 2.0–2.5 cores per worker locally and 2.2–2.8 on CI. So
workers: '100%', one worker per core, oversubscribes any machine by about 2×.
Only 4-core machines were ever measured, and both ran one thread per core. On a hyper-threaded machine the physical core count is the better input.
How it was measured
Both machines have 4 cores and no GPU, which makes the comparison about scheduling rather than hardware class. The whole study ran with retries turned off, so nothing could hide a failure from the count.
Local (cloud dev container)
- CPU
- Intel Xeon @ 2.80 GHz
- Cores
- 4 physical, 1 thread each
- Memory
- 15 GB
- GPU
- none
GitHub Actions ubuntu-latest
- Runner
- ubuntu-latest
- Cores
- 4 vCPU
- Memory
- 16 GB
- GPU
- none, so Chromium rasterizes canvas in software
- Local runs went round-robin across the worker counts, so a slow hour on the box could not favour any one setting.
- CI runs got one runner per worker count, so settings never competed with each other for CPU.
- Every repetition is a full suite run of 203 tests (204 after one was added), and a "red run" is any repetition with at least one failing test.
How to read the sweep cards
- Wall clock
- Median seconds for one full suite run. Lower is better; the fastest row in each card is tagged.
- Red runs
- Runs with at least one failing test, out of the runs measured. This is what a retry count is chosen against.
- Failing tests
- Failed test executions across all runs, shown as one failure in every N executions.
- Slowdown
- Each test's mean duration divided by its own mean at 1 worker. Above 1× means tests are waiting on the CPU.
- CPU
- Total test seconds in one run. The same tests run every time, so any growth is time lost to contention.
Local sweep
Local (cloud dev container). First as the suite stood, then after three magic-brush tests were fixed.
- Almost all the speed-up comes from the first extra worker. 1→2 saves 74.1s, 2→3 saves 9.3s, 3→4 saves 5.1s, 4→6 costs 1.2s, 6→8 costs 4.4s.
- Past 4 workers every run goes red. The number of distinct failing tests goes 2, 2, 10, 18 at 3, 4, 6 and 8 workers. At 8, tests with no canvas timing in them start failing too.
- Even 1 worker flaked once in 3 runs, so a lower count on its own could never reach a clean suite. Some tests were genuinely broken.
After fixing three magic-brush tests
Same box, a freshly started preview server, and the three repaired tests. Only the candidate counts were re-run.
- Magic-brush dropped out of the failure lists at 3 and 4 workers entirely, and 6 workers went from 0 green runs in 8 to 5 in 8.
- The best count did not move. At 4 workers, 5 failures in 1,624 test executions after the fix is the same rate as 3 in 1,624 before it. Other tests with thin timing margins took over. The worker count sets the flake rate; test fixes only raise how high the count can go.
- So local stays at 2 workers. 92.3s and 1 red run in 16, against 82.1s and 4 red runs in 8 at 4.
CI sweep
GitHub Actions ubuntu-latest, with the magic-brush fixes in place. The production config
sets retries: 2 on CI, which would hide the very rate being measured, so retries
were off here too.
Failure columns are not a real rate. Every run reused one preview server and inherited the previous run's spent rate-limit windows, so some failures here were manufactured by the harness. The harness bug section explains; the wall-clock column is unaffected.
- Wall clock bottoms out at 4 workers (60.2s). 8 workers is no faster and fails far more often.
- 1 worker still failed, with nothing to contend with. Two of its three failures were 30-second timeouts. The runner has no GPU, so Chromium rasterizes the magic-brush reveal in software and that test sits near its 15-second budget however many workers run.
- CI is faster than the local box overall (60–95s against 80–169s) while being slower at canvas work. "Faster hardware" is not one number.
A bigger reveal timeout, tried and reverted
The CI failures landed right at the magic-reveal budget, so it was raised from 15s to 30s and the two most informative counts were re-measured (run 30476652762). The change helped in one place and hurt in the other.
4 workersshipped
1 worker
- At 4 workers it worked. 5 green runs in 5 where there had been 3, for about 4.6s of wall clock per run.
- At 1 worker it did not. The same three tests failed, and now cost far more: with
test.slow()in play, a reveal that never converges ran to its full 90s budget instead of failing at 30s. The median run went from 95s to 138s. - Those failures were never short on time. The reveal helper loops draw, check, undo, redraw. A bigger budget just lets a loop that never converges run longer.
- Reverted. At 4 workers the two failures it fixed were already covered by retries, while a stuck reveal becoming the longest test in the run was a real cost. Bounding the loop's attempts fixed the genuinely slow cases instead.
The harness was measuring itself
Every rate above came from runs that shared one preview server per worker count. That sharing turned out to be most of what was being measured, so the retry question could not be answered until the sweep driver was fixed and the CI numbers re-taken on 2026-07-30.
What went wrong
The suite deliberately fills the API's 60-second per-IP rate-limit windows: one spec
exhausts the bring-your-own-key bucket and bursts the managed token's. A run takes about as
long as those windows last, so the next run started with a spent budget, and its guard tests
got a 429 Too Many Requests where they assert a 415.
- On the CI runner at 4 workers,
throttles a managed token hammered in a burstfailed in 12 of 12 runs. A deterministic failure had been counted as a flake rate. - Locally at 4 workers, the bring-your-own-key guard tests were 4 of the 5 failures across 7 runs.
- Those are the specs most of the failure columns above are made of.
The fix: every run gets its own freshly started preview server, which clears the
in-memory limiter and matches what CI does anyway: one server, one suite run. The sweep
driver started that server itself at the time; Playwright's own webServer starts it
per run now (issue 1044).
Wall clock here includes the 4 or so seconds the driver spends starting and probing each fresh server, so compare the shape with the first CI sweep, not the values.
- Almost all of this card is one test. 6 workers failing in 15 of 15 runs is not a flake rate, it is a deterministic failure.
a burst of screenshot taps shares one save before allowing the nextwaited a fixed 500ms for a save, so the more starved the worker, the more reliably it missed. Read on its own, the card says the rate climbs steeply with workers, and the first pass of this study believed it and set CI to 3 workers. - 1 worker is still one of the worst settings (6 red in 15) with no contention to blame. The GPU-less runner keeps canvas-heavy tests near their budgets however few workers run. That is why the curve is a U rather than a slope, and why the worker count alone could never reach zero.
Fixing the tests, and the retry decision
Same runner image, 35 runs each at the two candidate counts, retries still off. This is the evidence the shipped worker count and retry count rest on.
The four red runs came from three tests, all about zoom and pinch gesture state:
- 2 of 35 at 3 workers
closing the overlay resets the zoom for the next openfixed since - 1 of 35 at 4 workers
navigating to another section resets the zoomfixed since - 1 of 35 at 3 workers
a pinch swallows the trailing click, so it never toggles the control beneath itfixed since
- The climb was the test, not the worker count. 3 and 4 workers are statistically the same (Fisher's exact test, p = 0.61) and 4 is 3.2s faster per run, so CI went back to one worker per core. The detour through 3 workers stays in the record because the mistake generalises: a worker count tuned against a rate that one bad test dominates is really tuned around that test.
- Retries stay at 2, and the confidence interval is the whole argument. 1 red run in 35 is 2.9%, but one failure in 35 cannot establish a rate: the 95% interval reaches 12.9%.
retries: 0goes red every time that happens.retries: 1needs the same test to fail twice, which looks like 0.1% if the attempts were independent, and they are not: the retry runs straight afterwards on the same starved machine. (The 3-worker row is kept separate on purpose. Pooling the two into "4 in 70" would quote a figure for a setting that was measured 35 times.) - Retried passes are no longer silent. Each one becomes a GitHub Actions annotation plus a row in the job summary, so "green, but only on attempt 2" shows on the run page instead of in a log nobody opens.
- Lowering the retry count depends on fixing those tests, not on another sweep. Fixing one test took 4 workers from 6 red runs in 15 to 1 in 35.
The three zoom tests were one bug, and the rate stayed put
Same runner image, 35 runs at the shipped count, retries still off.
The one red run came from a test that had never failed before, in a different part of the app:
- 1 of 35
pointer exploration still snaps a hexagon gap and commits the highlighted color
- All three were one missing wait in a helper they shared. A dialog opens by flying in from the button that launched it, starting at 5% scale on top of that button. The modal also ignores pointer events within 72px of that launch point for 600ms, so the tap that opened it cannot land on the content. For the first frames the whole dialog sits inside that dead zone: the Settings pane starts 6px from the launch point and only clears the radius about 13ms in. The tests read the pane's live rectangle and dispatched synthetic pointer events at it, skipping the checks a real click performs, so the pinch landed in the dead zone and did nothing.
- A CSS animation only advances when a frame renders, which is how contention got in: a starved worker holds the dialog on that first keyframe far longer than 13ms. The tell was
a two-finger pinch enlarges the pane, a structurally identical test with 0 failures in 70 runs, whose one extra round trip lets the fly-in move on first. - Fixing it did not move the retry count. The three tests went 0 for 35, and in the same sweep a colour-picker test that had never failed went red once. The red-run rate at the shipped count is where it was, so
retries: 2stays and the next fix starts from that test. A rate that one test dominates is still not a rate.
What was tried
Three of these four explanations for the magic-brush flakes were wrong. They are recorded because they are cheap to think of again and expensive to re-test.
WrongCPU contention makes Chromium drop or merge pointer events, so strokes change shape
- Tested by
- Instrumented the page to count pointermove events and getCoalescedEvents() samples, then repeated the stroke with 12 busy-loop processes saturating every core.
- Result
- Identical to idle: 2 pointermove events and 2 coalesced samples, every time. Input delivery does not lose events under load.
WrongThe eraser test under-counts its baseline because it reads the canvas once instead of polling until the reading settles
- Tested by
- Replaced the single read with a poll that waits for two equal readings, then re-ran the test 200 times at 8 workers.
- Result
- Failure rate went from 16 of 200 to 15 of 200, which is no change. (A single un-polled read is still an anti-pattern under .claude/rules/testing.md. It just was not this bug.)
WrongpickBrush() returns before the brush is active, so waiting for aria-pressed="true" will fix it
- Tested by
- Added await expect(entry).toHaveAttribute("aria-pressed", "true") inside pickBrush() and re-ran the test 200 times at 8 workers.
- Result
- 16 of 200 before, 16 of 200 after. The spec file header already explains why: the mode reaches the engine through a Svelte $effect, so the button is correct while the engine is not. Reverted rather than kept as a wait that buys nothing.
Half rightThe strokes commit in the wrong brush mode, and counting filled pixels cannot detect it
- Tested by
- Read the actual assertion values out of the failing reports. Then, once the engine could be queried directly (ADR-0080), read its committed mode and the painted colours at each failure.
- Result
- The `revealed` pixel count came back as a few distinct values (132, 874, 895, 2314) where about 2314 is a real magic reveal. Redrawing all three sites took the file from 16 of 200 failures to 4 of 200, so that fix worked. The explanation did not: the 132 pixels were the coloring page’s own colours, not ink, and the engine reported the correct mode at every failure across about 700 recorded reveals. The stroke was a magic stroke the engine had cut short. A sample more than 100 ms and more than 10% of the paper away from the last one reads as a lifted finger (strokeMath.pointerWasResumed), which is exactly what a starved worker dispatching 180 px hops produces. Redrawing helped only by giving it a second, luckier stroke. Pacing the samples fixed it at the source, and the redraws are gone.
Re-tuning later
The numbers above belong to their hardware. The shape of the curve, with saturation near cores ÷ 2, should carry over; the exact optimum will not. To measure again:
# 1. Build once. The sweep driver builds nothing, so this is the only build, and
# PUBLIC_ENABLE_DEV_HARNESS has to be set HERE — it gates the /dev/* routes
# the specs drive.
PUBLIC_ENABLE_DEV_HARNESS=true ADMIN_ACCESS_TOKEN=test-admin-secret \
node tools/run-web-tool.mjs vite build
# 2. Sweep. The driver owns the whole protocol: a fresh preview server per rep,
# CI unset for the run, and one SWEEPRESULT line per rep.
for w in 1 2 3 4 6 8; do
node tools/e2e-tuning/run-worker-sweep.mjs --workers=$w --reps=30 --out=runs
done
# 3. On CI hardware the same driver runs from .github/workflows/worker-sweep.yml
# (manual dispatch, one runner per worker count so configs never contend):
# Actions -> "Worker sweep (manual)" -> Run workflow -> reps
# Read the numbers straight out of each job log:
# grep SWEEPRESULT
# Why the driver rather than a loop over `playwright test`: reps that share one
# server are not independent. generate-image.spec.ts deliberately fills the
# per-IP BYOK rate-limit bucket, which takes 60s to clear, and a rep takes about
# that long — so the next rep's guard tests take a 429 where they expect a 415
# and the sweep measures a flake it manufactured. A fresh server per rep clears
# the in-memory limiter and matches what CI does: one server, one suite run.
Then replace the datasets at the top of tools/e2e-tuning/gen-tuning-report.mjs and run
npm run gen:e2e-tuning-report. The decision and its reasoning live in
ADR-0078.