Skip to content

perf(coderd/x/chatd): reuse agent connection across steps - #30051

Closed
mafredri wants to merge 7 commits into
mainfrom
chatd-cache-workspace-home
Closed

mafredri wants to merge 7 commits into
mainfrom
chatd-cache-workspace-home

Conversation

@mafredri

@mafredri mafredri commented Sep 28, 2026 •

Copy link
Copy Markdown
Member

buildTurnEnvironment builds a turnWorkspaceContext per generation step and releases the step's workspace agent connection when the step ends, so every step acquires a new one (1 round trip for the reachability check). Root chats also run LS for the home directory on every step to build the plan path, and preparation blocks on it. A tool call is two steps, one that runs the tool and one for the next model request, so on main each tool call pays for two preparations. One tool call plus one reply on main: 4 preparations, 4 dials, 4 LS calls (2 model requests). With this PR: 1 dial, 1 LS, 1 release when the chat runner exits (TestAgentConnReusedAcrossGenerationSteps).

Measured at a 462 ms round trip time (setup and table in #30053), with 4 tool steps and a reply per turn: main acquired 10 connections per turn and a turn took 27.2 s; with this PR it acquired 1 and took 10.6 s, and 5.1 s together with #30053.

  • Steps run main's connection code and dial through the chat runner's runnerAgentConn. It returns the connection it holds if it is to the requested workspace agent and the workspace agent's row is not disconnected, and otherwise acquires one and holds it. It holds its lock while acquiring, so concurrent dials share one acquisition, and a dial after the runner closed returns an error.
  • It also keeps that workspace agent's home directory. workspaceHome gets the connection first and keys the home directory by it, so a cached home directory is used only behind the disconnected check every dial runs, and the dial bumps workspace usage.
  • The runner releases every connection it acquired when it exits, after waitForTasks. A connection it stops holding (another workspace agent, or a disconnected one) stays held until then, so no step uses a released connection. Queued turns run on the same runner, so they share the connection.
  • The root chat plan path calls forgetStaleBindingConn. It reads the latest build's chat agent through workspaceAgentIDForConn, as main's plan path does, and when the chat is bound to another workspace agent, the runner forgets its connection to it, so each step dials it again through dialWithLazyValidation, as on main.
  • Server.workspaceAgentDisconnected holds the status check that the step's cache-hit path and the runner both use.

Behavior change: a step that gets the runner's connection skips ServerTailnet.AgentConn's reachability check, and the stale-binding validation that runs when a dial is slow. Subagent chats skip the plan path lookup, so for them a rebuilt workspace is detected by the database status check of the old workspace agent. A connection the runner stops holding stays acquired until the runner exits.

ARCHITECTURE.md gets a TODO in the runner section.

Supersedes #30052. Related: #30053.

Plan

Round trips between coderd and the workspace agent per tool call in a root chat, measured as the time between model requests at a 462 ms round trip time (spans show where they go):

read_file execute
main 11 (5.2 s) 14 (6.5 s)
#30051 3 (1.5 s) 6 (2.9 s)
#30051 + #30053 1 (0.57 s) 2 (1.0 s)

On main, each of the 2 preparations per tool call costs 4 round trips (acquisition 1, LS on a new connection 3), and the tool's requests cost 3 each on a new connection.

Findings:

  • buildTurnEnvironment creates a workspace context per step. StartGeneration returns after each action, so a task is one step. The runner spans tasks and already holds per-chat state (stopNudges, sessionStart).
  • The plan path block dials and runs LS every step, uncached.
  • A canceled task can still run while its successor starts (cancelActiveTask does not wait), so state shared between steps must not hold a chat snapshot, and a connection one step stops using may still be in use by another.
  • dialWithLazyValidation can return before its first dial ends and releases that dial's connection later, so a dial can finish after the runner closed.

Decisions:

  • The runner replaces only the dial; step code stays as on main.
  • No reference counting: the runner keeps every connection it acquired until it exits.
  • The runner's lock is held across the acquisition, which makes concurrent dials share one and keeps a late acquisition from replacing a newer one or undoing a forget.
  • The stale-binding check stays for root chats, in forgetStaleBindingConn, which reads the latest build's chat agent through workspaceAgentIDForConn.
  • One name: runnerAgentConn.
  • Reviewed by GPT 6.1 Sol and Claude Opus 5.5. Applied: the lock held across the acquisition, dials failing after close, the cached home directory behind the disconnected check, the overstated binding comment, and test cleanups (one chat ID per runner, the tool call required to run, assert off the test goroutine).

Steps:

  1. Runner-owned workspace agent connection and home directory (perf(coderd/x/chatd): reuse agent connection across steps #30051).
  2. Reuse workspace agent API connections for chatd (perf: reuse workspace agent API connections in chatd #30053).
  3. Later: start and wait for execute in one request, after feat: run chat tool calls once per tool call ID on the agent #29946.
  4. Later: the workspace agent reports its home directory with the context it pushes, removing the LS.

🤖 This PR was created with the help of Coder Agents, and will be reviewed by a human. 🏂🏻

@mafredri
mafredri requested a balanced review from Copilot September 28, 2026 17:18
@mafredri

This comment has been minimized.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-30T12:47:47.278987Z 3ef31fa Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector[bot]

This comment was marked as outdated.

This comment was marked as outdated.

@mafredri
mafredri force-pushed the chatd-cache-workspace-home branch 2 times, most recently from b80cb3f to 0387af1 Compare September 29, 2026 12:37
@mafredri mafredri changed the title perf(coderd/x/chatd): cache workspace home for plan path perf(coderd/x/chatd): read workspace home once per run Sep 29, 2026
@mafredri
mafredri force-pushed the chatd-cache-workspace-home branch from 0387af1 to e404e13 Compare September 30, 2026 08:04
@mafredri mafredri changed the title perf(coderd/x/chatd): read workspace home once per run perf(coderd/x/chatd): reuse workspace context across steps Sep 30, 2026
@mafredri
mafredri force-pushed the chatd-cache-workspace-home branch from e404e13 to 9269314 Compare September 30, 2026 08:43
@mafredri

This comment has been minimized.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9269314d1e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/ARCHITECTURE.md Outdated
- a list of goroutines it has spawned to perform side effects, each identified by a unique ID, together with cancellation handles and go channels that the goroutines use to notify they have finished.
- the ID of the currently active goroutine, if there is one.

TODO: the runner also owns the chat's workspace context, which caches the workspace agent, its connection, and its home directory across generation goroutines, and releases the connection when the runner exits. Describe this here.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Replace the architecture TODO with final documentation

Replace this TODO with the final description of the runner-owned workspace context before merging. The repository guidance treats chatd architecture TODOs as handoff notes for the human PR author and explicitly requires reviewers to flag any added by a PR; leaving it unresolved means the documented runner lifecycle omits the new connection ownership and release invariant.

AGENTS.md reference: AGENTS.md:L74-L75

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't need reminders about TODO's will fix before merge. Focus on real issues.

@mafredri
mafredri force-pushed the chatd-cache-workspace-home branch 3 times, most recently from ab89e49 to 3ef31fa Compare September 30, 2026 10:41
@mafredri

This comment has been minimized.

@chatgpt-codex-connector

This comment has been minimized.

@linear-code

linear-code Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

CODAGT-1261

CODAGT-1625

@mafredri
mafredri force-pushed the chatd-cache-workspace-home branch 2 times, most recently from 543cebd to 590181d Compare October 9, 2026 11:00
@mafredri mafredri changed the title perf(coderd/x/chatd): reuse workspace context across steps perf(coderd/x/chatd): reuse agent connection across steps Oct 9, 2026
@mafredri
mafredri force-pushed the chatd-cache-workspace-home branch 4 times, most recently from 2cd9809 to aa90713 Compare October 13, 2026 11:33
@mafredri
mafredri force-pushed the chatd-cache-workspace-home branch from aa90713 to c7cf4e3 Compare October 21, 2026 08:40
@github-actions github-actions Bot added the stale This issue is like stale bread. label Nov 4, 2026
prepareGeneration created a turnWorkspaceContext per step, so the
cached agent and connection were dropped after every step. Root chats
also dialed the agent and ran LS for the home directory on every step
to build the plan path. One tool call plus one reply: 4 steps, 4 dials,
4 LS calls.

- The chat runner owns the workspace context and closes it on exit.
- The context caches the agent's home directory. setAgentLocked and
  clearAgentLocked are the only writers of the cached agent and clear
  derived state.
- chatStateMu and currentChat are plain fields.
- Prepare bumps workspace activity once per step instead of relying on
  the dial.

The agent is re-resolved from the database on first use or when the
chat's workspace changes, not every step. The cached connection still
re-reads the agent row on use and is dropped when the agent is
disconnected.
prepareGeneration requires the runner's workspace context since the
previous commit. TestPrepareGenerationMemory, added on main after this
branch started, called it without one and failed with "workspace
context is required".
The previous commit shared one workspace context across the steps of a
turn. A step whose task was canceled can still run while the next step
starts, and both wrote the shared chat snapshot. The cache was checked
against the workspace ID only, so a chat rebound to another workspace
agent of the same workspace kept the old connection and home directory,
and a workspace change could hand out the previous workspace's
connection.

Each step now builds its own workspace context from its own chat
snapshot, as on main. The runner keeps only the connection and the home
directory, keyed by workspace agent ID, and a step borrows them when its
resolved workspace agent matches. The connection is released after the
last borrowing step closes, so dropping it in one step does not close it
under another step's tool calls. A cached home directory still bumps
workspace usage, as the dial it replaces did.
The previous commit cached the connection twice, per step and per
runner, and kept the two in sync with borrow counts and a list of
connections to release at step close. It copied the status check, and
the plan path code had to drop the runner's connection itself.

Steps now run main's connection code and only dial through the runner.
The runner returns the connection it holds when it is to the requested
workspace agent and that workspace agent is not disconnected, and
otherwise acquires one. It releases every connection it acquired when
it exits, after its steps have ended, so no step uses a released
connection and nothing needs counting. The lookup that finds the latest
build's chat agent now also forgets a connection to a stale binding.
workspaceAgentIDForConn reads the latest build's chat agent, and also
made the runner forget its connection to a stale binding. The plan path
now calls checkWorkspaceAgentBinding, which does the lookup and the
forget, so the lookup has no new side effect and the plan path still
does not reach into the runner's connection.
Reviews found that runnerAgentConn acquired outside its lock:

- Concurrent first dials each acquired a connection and the runner
  held all of them until it exited.
- A late acquisition could replace a newer connection, or undo forget.
- A dial that finished after close added a release nobody ran, so the
  coordinator ticket leaked.

Holding mu across the acquisition makes concurrent dials share one,
and forget and close wait for it. A dial after close returns an error
instead of acquiring.

workspaceHome now gets the connection first and keys the cached home
by it, so a cached home is used only behind the disconnected check
that every dial runs. Before, a cached home skipped that check.

checkWorkspaceAgentBinding is now forgetStaleBindingConn: forgetting
makes each step dial again through dialWithLazyValidation, which
validates only when the dial is slow or fails, so "validates the
binding" overstated it.
The test is now named for what it checks. StartProcess and ProcessOutput
must run once, so a failed tool call cannot pass it, and the release is
checked directly: WaitUntilIdleForTest returns after the runner exits.
The agent ID check uses assert because it runs off the test goroutine.
@mafredri
mafredri force-pushed the chatd-cache-workspace-home branch from c7cf4e3 to c3bb1c2 Compare November 4, 2026 10:55
@github-actions github-actions Bot removed the stale This issue is like stale bread. label Nov 5, 2026
@github-actions github-actions Bot added the stale This issue is like stale bread. label Nov 13, 2026
@github-actions github-actions Bot closed this Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stale This issue is like stale bread.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants