Repository navigation
Conversation
mafredri
force-pushed
the
chore/chatbench
branch
2 times, most recently
from
October 27, 2026 14:25
519cb9c to
1e96b46
Compare
mafredri
force-pushed
the
chore/chatbench
branch
from
October 30, 2026 11:38
bf78dcc to
c2c49d2
Compare
mafredri
force-pushed
the
chore/chatbench
branch
2 times, most recently
from
November 4, 2026 11:01
02f7bb2 to
0969532
Compare
/agents/compare/:leftId/:rightId mounts two AgentChatPage instances, each with its own MessageScroller provider, in one tree. This is the scenario in which interaction lag was reported with two long streaming chats, and it is what scripts/chatbench measures. AgentChatPage takes an optional chatId so the route can supply it instead of the URL parameter. The route is not gated. The sidebar receives no selection and both pages render a <title>.
Reproduces and measures the side-by-side streaming chat interaction lag on the real production bundle, in real browsers, with real profiles, so a fix can be judged by numbers instead of by eye. up.sh runs an embedded coderd (built-in postgres, port 18080) next to fakellm, an OpenAI-compatible server that streams deterministic markdown at a fixed token rate. seed.py creates a user, provider, model and two long chats. bench.mjs drives Chromium or WebKit with Playwright against /agents/compare: loads the full history of both chats, starts both streams, measures focus switching and typing latency, and records Chromium CDP traces and CPU profiles, WebKit Timeline and JS sampling profiles through the remote inspector, and React DevTools commit profiles (react-devtools-inline, opt in). The prep hook edits the live page before measuring (delete CSS rule classes, for example) so a suspected cost can be attributed in one run without a synthetic page. WebKit runs in the official Playwright image because the host OS is newer than the WebKit build supports. The trace-*, wk-* and react-summary analyzers turn those recordings into per-task breakdowns, symbolicated JS stacks and React commit tables.
The WebKit timeline shows that a frame forced layout but not which DOM reads did it or how often. prep/geometry-probe.js wraps the geometry getters and methods on the live page and reports counts and time per phase through the new window.__bench.probe hook. --init runs scripts before the app mounts so an API the app captures at mount can be wrapped. wk-samples.mjs stacks ranks whole call stacks and keeps native frames such as getBoundingClientRect visible. The history loader stops on no progress instead of trusting the seeded turn count, which every run outgrows.
The scroll-and-type scenario in scripts/chatbench mounts three chats. The route now takes one path segment per chat instead of exactly two.
bench.mjs gains --panes, --stream=0 and --heap=1, the scrolltype scenario, content-process memory readings, and browser events aligned with long frames. wk-inspector.mjs records WebKit GC, memory pressure and network events. heap-summary.mjs summarizes a V8 heap snapshot without parsing it as one JSON value. seed.py adds login, add and extend, and refreshes an expired session token.
Playwright 1.63 bundles its server into coreBundle.js, so the WKSession patch can no longer load wkConnection.js by path. Find the WKSession prototype from the first session that emits a protocol event instead, and take the Docker image tag from the installed @playwright/test.
fakellm --chunk sends several tokens per delta (CHATBENCH_CHUNK in up.sh), so a stream reaches 40k to 60k characters in minutes; with one token per delta, chatd persisted only the first 2,474 tokens of a fast stream. bench.mjs --warmup sets the streaming time before the streaming phases, and each phase records the rendered length of the reply.
With one composer, focusSwitch clicked the same composer twice, so focus changed only on the first click. Blur it between clicks instead.
In WebKit a scroll to the top sometimes did not load the next history page, and a run then measured only the first page (50 rows per pane) with no error. Count the user and assistant messages through the API, keep loading until one row per message renders, reload the page up to twice when loading stalls, and fail the run if the transcript stays partial. --history=page measures the first page on purpose.
When coderd dies without shutting down, its built-in PostgreSQL keeps running, and every later start fails because the port is taken. up.sh stop and build now stop the PostgreSQL recorded in postmaster.pid under the bench root, after checking its command line.
… string length Also log the probe counts over the whole recording, which includes the time to start and stop it.
--type-prefill and --type-chars fill the composer and then type filler text one key at a time, so it grows line by line. On main each growth changes the composer's mobile dropdown variables on <html>; the #29596 numbers for that case come from these flags.
…at caused them wrap-summary.mjs recomputes the #29596 wrapping table from results.json: the slowest keystroke before each growth, the other keystrokes' p50 and the <html> style mutation records, all within the typing window.
…that wrapped The slowest keystroke before a growth was often another key on the side without root writes (14 of 18 in Chromium, 7 of 15 in WebKit). The wrapping keystroke is now the last one before the growth's first <html> style mutation record, or before the growth when there is none; the positions match across all runs of one browser. A growth that pairs with no keystroke makes the script exit 1.
mafredri
force-pushed
the
chore/chatbench
branch
from
November 18, 2026 13:55
0969532 to
b244220
Compare
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Warning
Not assumed to be mergeable. This is the benchmark used to measure #29596, kept here so the numbers can be checked and reproduced.
scripts/chatbenchruns an embedded production coderd with built-in Postgres, a fake OpenAI-compatible provider (fakellm) that streams deterministic markdown, and a Playwright driver that measures focus and key latency in Chromium and WebKit while long chats stream. It also has trace and profile analyzers./agents/compare/<chatId>/<chatId>/...renders one chat per pane; the driver uses it to run several chats on one main thread.To run it:
The bench does not depend on the fixes in #29596; it measures whatever build
up.shserves.