Skip to content

chore: add a chat rendering benchmark and compare route - #31278

Draft
mafredri wants to merge 22 commits into
mainfrom
chore/chatbench
Draft

mafredri wants to merge 22 commits into
mainfrom
chore/chatbench

Conversation

@mafredri

Copy link
Copy Markdown
Member

Warning

Not assumed to be mergeable. This is the benchmark used to measure #29596, kept here so the numbers can be checked and reproduced.

scripts/chatbench runs an embedded production coderd with built-in Postgres, a fake OpenAI-compatible provider (fakellm) that streams deterministic markdown, and a Playwright driver that measures focus and key latency in Chromium and WebKit while long chats stream. It also has trace and profile analyzers. /agents/compare/<chatId>/<chatId>/... renders one chat per pane; the driver uses it to run several chats on one main thread.

To run it:

bash scripts/chatbench/up.sh build
scripts/chatbench/seed.py
node scripts/chatbench/bench.mjs --browser chromium --scenario compare --label before
scripts/chatbench/bench-docker.sh --browser=webkit --scenario=compare --label=before

The bench does not depend on the fixes in #29596; it measures whatever build up.sh serves.

🤖 This PR was created with the help of Coder Agents, and will be reviewed by a human. 🏂🏻

@linear-code

linear-code Bot commented Oct 26, 2026

Copy link
Copy Markdown

CODAGT-1081

/agents/compare/:leftId/:rightId mounts two AgentChatPage instances,
each with its own MessageScroller provider, in one tree. This is the
scenario in which interaction lag was reported with two long streaming
chats, and it is what scripts/chatbench measures. AgentChatPage takes
an optional chatId so the route can supply it instead of the URL
parameter.

The route is not gated. The sidebar receives no selection and both
pages render a <title>.
Reproduces and measures the side-by-side streaming chat interaction
lag on the real production bundle, in real browsers, with real
profiles, so a fix can be judged by numbers instead of by eye.

up.sh runs an embedded coderd (built-in postgres, port 18080) next to
fakellm, an OpenAI-compatible server that streams deterministic
markdown at a fixed token rate. seed.py creates a user, provider,
model and two long chats. bench.mjs drives Chromium or WebKit with
Playwright against /agents/compare: loads the full history of both
chats, starts both streams, measures focus switching and typing
latency, and records Chromium CDP traces and CPU profiles, WebKit
Timeline and JS sampling profiles through the remote inspector, and
React DevTools commit profiles (react-devtools-inline, opt in). The
prep hook edits the live page before measuring (delete CSS rule
classes, for example) so a suspected cost can be attributed in one
run without a synthetic page. WebKit runs in the official Playwright
image because the host OS is newer than the WebKit build supports.

The trace-*, wk-* and react-summary analyzers turn those recordings
into per-task breakdowns, symbolicated JS stacks and React commit
tables.
The WebKit timeline shows that a frame forced layout but not which DOM
reads did it or how often. prep/geometry-probe.js wraps the geometry
getters and methods on the live page and reports counts and time per
phase through the new window.__bench.probe hook. --init runs scripts
before the app mounts so an API the app captures at mount can be
wrapped. wk-samples.mjs stacks ranks whole call stacks and keeps
native frames such as getBoundingClientRect visible. The history
loader stops on no progress instead of trusting the seeded turn count,
which every run outgrows.
The scroll-and-type scenario in scripts/chatbench mounts three chats.
The route now takes one path segment per chat instead of exactly two.
bench.mjs gains --panes, --stream=0 and --heap=1, the scrolltype
scenario, content-process memory readings, and browser events aligned
with long frames. wk-inspector.mjs records WebKit GC, memory pressure
and network events. heap-summary.mjs summarizes a V8 heap snapshot
without parsing it as one JSON value. seed.py adds login, add and
extend, and refreshes an expired session token.
Playwright 1.63 bundles its server into coreBundle.js, so the WKSession
patch can no longer load wkConnection.js by path. Find the WKSession
prototype from the first session that emits a protocol event instead,
and take the Docker image tag from the installed @playwright/test.
fakellm --chunk sends several tokens per delta (CHATBENCH_CHUNK in
up.sh), so a stream reaches 40k to 60k characters in minutes; with one
token per delta, chatd persisted only the first 2,474 tokens of a fast
stream. bench.mjs --warmup sets the streaming time before the streaming
phases, and each phase records the rendered length of the reply.
With one composer, focusSwitch clicked the same composer twice, so focus changed only on the first click. Blur it between clicks instead.
In WebKit a scroll to the top sometimes did not load the next history
page, and a run then measured only the first page (50 rows per pane)
with no error. Count the user and assistant messages through the API,
keep loading until one row per message renders, reload the page up to
twice when loading stalls, and fail the run if the transcript stays
partial. --history=page measures the first page on purpose.
When coderd dies without shutting down, its built-in PostgreSQL keeps
running, and every later start fails because the port is taken. up.sh
stop and build now stop the PostgreSQL recorded in postmaster.pid under
the bench root, after checking its command line.
… string length

Also log the probe counts over the whole recording, which includes the time to start and stop it.
--type-prefill and --type-chars fill the composer and then type filler
text one key at a time, so it grows line by line. On main each growth
changes the composer's mobile dropdown variables on <html>; the #29596
numbers for that case come from these flags.
…at caused them

wrap-summary.mjs recomputes the #29596 wrapping table from results.json:
the slowest keystroke before each growth, the other keystrokes' p50 and
the <html> style mutation records, all within the typing window.
…that wrapped

The slowest keystroke before a growth was often another key on the side
without root writes (14 of 18 in Chromium, 7 of 15 in WebKit). The
wrapping keystroke is now the last one before the growth's first <html>
style mutation record, or before the growth when there is none; the
positions match across all runs of one browser. A growth that pairs
with no keystroke makes the script exit 1.

This branch was previously deployed

1 inactive deployment
main — b244220e Deployed Nov 18, 2026 by mafredri via cla #259499
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant