Skip to content

perf(coderd/x/chatd): keep agent conn across steps - #30052

Closed
mafredri wants to merge 2 commits into
chatd-cache-workspace-homefrom
chatd-keep-agent-conn
Closed

mafredri wants to merge 2 commits into
chatd-cache-workspace-homefrom
chatd-keep-agent-conn

Conversation

@mafredri

@mafredri mafredri commented Sep 28, 2026 •

Copy link
Copy Markdown
Member

Each generation step builds a new workspace context and releases its agent connection when the step ends. A turn with one tool call runs several steps, so each step that touched the workspace acquired a new AgentConn. On coderd that costs another reachability ping to the agent before the step's first request. In the new test, one tool call and one reply dialed the agent twice on top of #30051.

The runner workspace state from #30051 now also holds the connection while the runner owns the chat. Each step still resolves the agent from the database, so the checks for deleted workspaces and changed agent bindings are unchanged. The step reuses the held connection only for the same workspace and agent, and only when the agent row it just read does not show the agent as disconnected. Error paths that clear the step's cached connection also drop the held one, and the runner releases it after its tasks finish.

The runner TODO in ARCHITECTURE.md from #30051 now mentions the connection too.

Depends on #30051. This is PR 2 of 3 to reduce round trips per Coder Agents tool call. The next PR reuses TCP connections to the agent through the existing per-agent pool in ServerTailnet.

Implementation plan

Problem

A tool step in a root chat costs about 7 round trips between coderd and the agent for read_file and 10 for execute. About 4 of those happen before the tool runs.

Findings

  • Every generation step builds a new turnWorkspaceContext and releases it at the end. StartGeneration returns after each action, so a task lasts one step. The chat runner outlives tasks and already keeps per-chat state across them.
  • Root chats resolved the plan path on every step by dialing the agent and sending LS (fixed in perf(coderd/x/chatd): reuse agent connection across steps #30051).
  • Each agent API request opens a new TCP connection: a ping, a TCP handshake, then the request.
  • ServerTailnet already pools keep-alive connections per agent for workspace apps.
  • The agent's API server closes idle connections after 20 s: IdleTimeout is unset and falls back to ReadTimeout.

Steps

  1. Keep the workspace home directory in the chat runner, keyed by agent ID (perf(coderd/x/chatd): reuse agent connection across steps #30051).
  2. Keep the agent connection across generation steps while the runner owns the chat. (This PR.)
  3. Send agent API requests through the existing per-agent pool in ServerTailnet, set an idle timeout on the agent's API server, and remove the double ping on dial.
  4. Later, after feat: run chat tool calls once per tool call ID on the agent #29946: start and wait for execute in one request.

Expected round trips per tool step

After step read_file execute
Today 7 10
1 4 7
2 3 6
3 1 2

🤖 This PR was created with the help of Coder Agents, and will be reviewed by a human. 🏂🏻

Every generation step of a root chat built the plan path by dialing
the workspace agent and sending an LS for the home directory.
Preparation waits for it, so each step paid a ping plus a request on
a new TCP connection before the model or tool ran. One tool call and
one reply ran four steps and four LS calls.

The chat runner now keeps the home directory, keyed by agent ID, for
as long as it owns the chat, the same way it keeps stop-nudge state
across tasks. A new build gets a new agent ID, so a rebuilt workspace
is read again.

The dial was also the only per-step caller of trackWorkspaceUsage, so
the plan path block now bumps workspace activity itself. Without that,
steps that run no tools would stop extending the autostop deadline.
@mafredri
mafredri force-pushed the chatd-cache-workspace-home branch from b80cb3f to 0387af1 Compare September 29, 2026 12:37
Each generation step builds a new workspace context and releases its
agent connection when the step ends, so every step that touched the
workspace acquired a new AgentConn. On coderd that means another
reachability ping to the agent before the first request of the step.

The chat runner's workspace state now also holds the connection while
the runner owns the chat. Each step still resolves the agent from the
database, then reuses the held connection only for the same workspace
and agent, and only when the agent row it just read does not show the
agent as disconnected. Error paths that clear the step's cached
connection also drop the held one, so a bad connection is not reused.
The runner releases the connection after its tasks finish.
@mafredri

Copy link
Copy Markdown
Member Author

Folded into #30051, which now reuses the whole workspace context across steps instead of adding a separate holder for the connection.

🤖 This comment was generated with Coder Agents.

@mafredri mafredri closed this Sep 30, 2026
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 30, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant