Skip to content

perf(bench): add a repeatable benchmark suite - #252

Merged
benvinegar merged 5 commits into
mainfrom
claude/sideshow-perf-benchmarks-9zhrho
Aug 15, 2026
Merged

perf(bench): add a repeatable benchmark suite#252
benvinegar merged 5 commits into
mainfrom
claude/sideshow-perf-benchmarks-9zhrho

Conversation

@benvinegar

Copy link
Copy Markdown
Member

Adds bench/: a performance suite covering the paths behind the CPU and
memory complaints, plus a regression gate CI can run.

Suites: store (per backend, and how write cost scales with workspace size),
render (shiki/markdown-it/diff SSR and output size), api (route latency and
response bytes), events (SSE fan-out, long-poll wakeups), process (startup,
idle RSS, and which imports the memory belongs to), and viewer (browser CPU,
layout work, heap, DOM size via CDP).

The design problem is deciding when a number moving is news, so each metric
carries a kind: bytes/count are deterministic and gated near-exactly, memory
and time get generous ratios plus an absolute floor. Timings are normalized by
a per-run machine index so a baseline recorded on one machine still means
something on another. CI runs with --gate deterministic, failing only on
machine-independent metrics — timings are still measured and printed, but a
flaky perf gate is one people learn to ignore.

Fixtures are pure functions of a seed, so a difference in the numbers is a
difference in the code. Write benchmarks use fixed iteration counts because
each write changes the thing being measured.

Tooling only; no runtime behavior changes.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01MGMuudU8anVrxB7sjY9hpt

claude added 2 commits August 15, 2026 00:09
Adds `bench/`: a performance suite covering the paths behind the CPU and
memory complaints, plus a regression gate CI can run.

Suites: store (per backend, and how write cost scales with workspace size),
render (shiki/markdown-it/diff SSR and output size), api (route latency and
response bytes), events (SSE fan-out, long-poll wakeups), process (startup,
idle RSS, and which imports the memory belongs to), and viewer (browser CPU,
layout work, heap, DOM size via CDP).

The design problem is deciding when a number moving is news, so each metric
carries a kind: bytes/count are deterministic and gated near-exactly, memory
and time get generous ratios plus an absolute floor. Timings are normalized by
a per-run machine index so a baseline recorded on one machine still means
something on another. CI runs with `--gate deterministic`, failing only on
machine-independent metrics — timings are still measured and printed, but a
flaky perf gate is one people learn to ignore.

Fixtures are pure functions of a seed, so a difference in the numbers is a
difference in the code. Write benchmarks use fixed iteration counts because
each write changes the thing being measured.

Tooling only; no runtime behavior changes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MGMuudU8anVrxB7sjY9hpt
The benchmark suite's process attribution showed a server holding ~132 MB
before serving a request, with ~72 MB of it attributable to importing
server/app.ts — mostly shiki, @pierre/diffs, markdown-it and
@mermaid-js/parser, loaded at boot whether or not the workspace ever used
them.

Two dynamic imports, both at the point of use:

- app.ts loads richRender.ts when it renders a markdown/code/diff/terminal
  surface. It sits after the html and mermaid branches, so an html-only
  workspace never loads any of it.
- postSurfaces.ts loads @pierre/diffs and @mermaid-js/parser when validating a
  published diff or mermaid surface.

Both halves were needed: leaving either static keeps the whole graph resident
and neither saves anything. Idle RSS 132 MB -> 102 MB, boot 486 ms -> 275 ms.

The mermaid parser and the diff parsers now load outside the try/catch that
reports parse failures, so a module-load error can't be reported to the user as
invalid mermaid (a 400 where it should be a 500).

A dynamic import is the kind of change that works on Node and fails only once
deployed, so test/workerIntegration now renders all four rich surface kinds on
real workerd and asserts on markup only the real renderers emit — the existing
html render deliberately never reaches that path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MGMuudU8anVrxB7sjY9hpt
Comment thread bench/run.ts Fixed
claude added 3 commits August 15, 2026 04:30
CodeQL flagged one high-severity alert on the benchmark suite: a post id parsed
out of a publish response flowed straight into the URL of the next request
(request forgery). The viewer suite did the same with a session id.

The bench only ever talks to a server it spawned itself, so this was not a live
vulnerability — but the shape genuinely was unverified, and the unchecked version
also fails confusingly: when a publish returns an error body instead of a post,
the next line 404s rather than reporting what actually went wrong. Both ids are
now checked against the url-safe base64 shape newId produces, and throw with the
offending value when they don't match.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MGMuudU8anVrxB7sjY9hpt
CodeQL's high-severity alert on this PR was regex injection at bench/run.ts:104
— a command-line argument going straight into the RegExp constructor.

--filter now takes comma-separated literal substrings, OR'd, case-insensitive.
That removes the taint flow, and it is the better CLI anyway: metric names are
full of regex metacharacters ("GET /s/:id code (cache hit)"), so the obvious
move — pasting a name off the results table — used to match nothing. Now it
works, and no hand-written pattern can backtrack a benchmark run into a hang.

Lowercasing happens inside the matcher rather than in the arg parser, so a
caller constructing a context directly still gets case-insensitive matching;
a test covers that seam along with the pasted-name and OR cases.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MGMuudU8anVrxB7sjY9hpt
The /s/:id benches asked for `?part=0` with no `&theme=&mode=`. The viewer
always sends both (Card.tsx builds every surface iframe src that way), and the
server renders differently when the mode is pinned — so the suite was measuring
a URL shape no client ever requests.

That is the failure mode a benchmark can't afford: profiling a shiki change
against these numbers showed no improvement at all, while the same change
through the real viewer URL was 47% faster and 30% smaller. A bench that misses
the win is worse than no bench, because it argues against the fix.

Baseline re-recorded for the corrected URLs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MGMuudU8anVrxB7sjY9hpt
@benvinegar
benvinegar merged commit 102990f into main Aug 15, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants