perf(bench): add a repeatable benchmark suite - #252
Merged
Conversation
Adds `bench/`: a performance suite covering the paths behind the CPU and memory complaints, plus a regression gate CI can run. Suites: store (per backend, and how write cost scales with workspace size), render (shiki/markdown-it/diff SSR and output size), api (route latency and response bytes), events (SSE fan-out, long-poll wakeups), process (startup, idle RSS, and which imports the memory belongs to), and viewer (browser CPU, layout work, heap, DOM size via CDP). The design problem is deciding when a number moving is news, so each metric carries a kind: bytes/count are deterministic and gated near-exactly, memory and time get generous ratios plus an absolute floor. Timings are normalized by a per-run machine index so a baseline recorded on one machine still means something on another. CI runs with `--gate deterministic`, failing only on machine-independent metrics — timings are still measured and printed, but a flaky perf gate is one people learn to ignore. Fixtures are pure functions of a seed, so a difference in the numbers is a difference in the code. Write benchmarks use fixed iteration counts because each write changes the thing being measured. Tooling only; no runtime behavior changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MGMuudU8anVrxB7sjY9hpt
The benchmark suite's process attribution showed a server holding ~132 MB before serving a request, with ~72 MB of it attributable to importing server/app.ts — mostly shiki, @pierre/diffs, markdown-it and @mermaid-js/parser, loaded at boot whether or not the workspace ever used them. Two dynamic imports, both at the point of use: - app.ts loads richRender.ts when it renders a markdown/code/diff/terminal surface. It sits after the html and mermaid branches, so an html-only workspace never loads any of it. - postSurfaces.ts loads @pierre/diffs and @mermaid-js/parser when validating a published diff or mermaid surface. Both halves were needed: leaving either static keeps the whole graph resident and neither saves anything. Idle RSS 132 MB -> 102 MB, boot 486 ms -> 275 ms. The mermaid parser and the diff parsers now load outside the try/catch that reports parse failures, so a module-load error can't be reported to the user as invalid mermaid (a 400 where it should be a 500). A dynamic import is the kind of change that works on Node and fails only once deployed, so test/workerIntegration now renders all four rich surface kinds on real workerd and asserts on markup only the real renderers emit — the existing html render deliberately never reaches that path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MGMuudU8anVrxB7sjY9hpt
CodeQL flagged one high-severity alert on the benchmark suite: a post id parsed out of a publish response flowed straight into the URL of the next request (request forgery). The viewer suite did the same with a session id. The bench only ever talks to a server it spawned itself, so this was not a live vulnerability — but the shape genuinely was unverified, and the unchecked version also fails confusingly: when a publish returns an error body instead of a post, the next line 404s rather than reporting what actually went wrong. Both ids are now checked against the url-safe base64 shape newId produces, and throw with the offending value when they don't match. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MGMuudU8anVrxB7sjY9hpt
CodeQL's high-severity alert on this PR was regex injection at bench/run.ts:104
— a command-line argument going straight into the RegExp constructor.
--filter now takes comma-separated literal substrings, OR'd, case-insensitive.
That removes the taint flow, and it is the better CLI anyway: metric names are
full of regex metacharacters ("GET /s/:id code (cache hit)"), so the obvious
move — pasting a name off the results table — used to match nothing. Now it
works, and no hand-written pattern can backtrack a benchmark run into a hang.
Lowercasing happens inside the matcher rather than in the arg parser, so a
caller constructing a context directly still gets case-insensitive matching;
a test covers that seam along with the pasted-name and OR cases.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MGMuudU8anVrxB7sjY9hpt
The /s/:id benches asked for `?part=0` with no `&theme=&mode=`. The viewer always sends both (Card.tsx builds every surface iframe src that way), and the server renders differently when the mode is pinned — so the suite was measuring a URL shape no client ever requests. That is the failure mode a benchmark can't afford: profiling a shiki change against these numbers showed no improvement at all, while the same change through the real viewer URL was 47% faster and 30% smaller. A bench that misses the win is worse than no bench, because it argues against the fix. Baseline re-recorded for the corrected URLs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MGMuudU8anVrxB7sjY9hpt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
bench/: a performance suite covering the paths behind the CPU andmemory complaints, plus a regression gate CI can run.
Suites: store (per backend, and how write cost scales with workspace size),
render (shiki/markdown-it/diff SSR and output size), api (route latency and
response bytes), events (SSE fan-out, long-poll wakeups), process (startup,
idle RSS, and which imports the memory belongs to), and viewer (browser CPU,
layout work, heap, DOM size via CDP).
The design problem is deciding when a number moving is news, so each metric
carries a kind: bytes/count are deterministic and gated near-exactly, memory
and time get generous ratios plus an absolute floor. Timings are normalized by
a per-run machine index so a baseline recorded on one machine still means
something on another. CI runs with
--gate deterministic, failing only onmachine-independent metrics — timings are still measured and printed, but a
flaky perf gate is one people learn to ignore.
Fixtures are pure functions of a seed, so a difference in the numbers is a
difference in the code. Write benchmarks use fixed iteration counts because
each write changes the thing being measured.
Tooling only; no runtime behavior changes.
Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01MGMuudU8anVrxB7sjY9hpt