Skip to content

perf: reuse workspace agent API connections in chatd - #30053

Closed
mafredri wants to merge 12 commits into
mainfrom
agent-api-conn-pool
Closed

mafredri wants to merge 12 commits into
mainfrom
agent-api-conn-pool

Conversation

@mafredri

@mafredri mafredri commented Sep 28, 2026 •

Copy link
Copy Markdown
Member

Every workspace agent API request from coderd opens a new TCP connection over the tailnet: a reachability check, a handshake and the request, 3 round trips. A chatd read_file is 1 request and execute is 2, so at a 462 ms round trip time they take 1.4 s and 2.8 s before the workspace agent does any work. This PR keeps chatd's connections to the workspace agent's HTTP API server open between requests, so a request on a kept connection takes 1 round trip.

  • AgentConnOptions.APIIdleConnTimeout sends every request of the AgentConn through one keep-alive http.Client, which closes connections idle for longer than the timeout. Without the option nothing changes: each request gets its own client with keep-alives disabled, as on main.
  • Close ends the client's dials and closes its connections; a connection in use closes when its response ends. http.Transport keeps a dial running after its request is canceled.
  • decodeAgentJSON reads up to 4 KiB after the decoded value. http.Transport reuses a connection only after the response body was read to EOF, and the JSON decoder can stop before the end of a chunked body.
  • coderd passes chatd ServerTailnet.keepAliveAgentConn, which sets the option to 15 s. Like AgentConn, it logs and counts a workspace agent that does not answer (feat: add ServerTailnet logs and metrics for unreachable agents #30133). Releasing a connection from ServerTailnet.AgentConn or keepAliveAgentConn now closes it. For other callers this changes nothing, because their connections keep no idle connections.
  • The workspace agent's API server sets IdleTimeout to workspacesdk.AgentHTTPAPIServerIdleTimeout, 20 s. Every workspace agent already used 20 s, because Go uses ReadTimeout when IdleTimeout is unset. A test checks that coderd's 15 s stays below it.

Measured at a 462 ms round trip time: a ServerTailnet connected over dogfood's DERP to a workspace agent in the same workspace, with tc qdisc add dev eth0 root netem delay 225ms, so both directions are delayed. 3 turns, each acquires a connection, sends 5 read_file and 5 execute calls, and releases it.

main this PR
read_file (1 request), p50 1,386 ms 462 ms
execute (2 requests), p50 2,776 ms 925 ms
Requests on a reused connection 0 of 45 42 of 45

Each turn's first request dials and takes 1,388 ms. With #30051, chatd holds one connection per chat runner, so a turn's tool calls share it; without it, one per generation step.

End to end, with the same ServerTailnet and round trip time: a chatd server with a scripted model running 4 tool steps per turn (read_file, execute, read_file, execute) and a reply, all builds on the same main commit. Medians of 3 turns after a warm-up turn; a step is the time between two model requests, so the tool call plus the next step's preparation.

turn read_file step execute step
main 27.2 s 5.2 s 6.5 s
this PR 21.7 s 4.3 s 4.7 s
#30051 10.6 s 1.5 s 2.9 s
this PR and #30051 5.1 s 0.57 s 1.0 s

In every build, the first model request of a turn came 1.9 s after the chat was created. Spans show where it goes: the acquisition's reachability check (464 ms), then an LS on a new connection (1,388 ms).

Behavior changes:

  • If the workspace agent closes a kept connection, which happens when it crashes and restarts with the same ID, a request sent on it can fail. With a real coder agent killed with SIGKILL and restarted, 6 of 6 POSTs failed and none ran, 3 of 3 GETs succeeded because http.Transport resent them, and the next request succeeded on a new connection. With SIGTERM the workspace agent closes its connections on shutdown, and 9 of 9 requests succeeded.
  • A gap of more than 15 s between two requests costs a new connection. On dogfood, 20.0% of the gaps between workspace tool calls in a chat were longer (24,646 gaps over 74 chats of one user since 2026-05-27).
  • An idle API connection holds 35 to 50 KB in coderd and 22 to 26 KB in the workspace agent. A connection from keepAliveAgentConn keeps up to 2, http.Transport's default per host, for example after parallel tool calls.

Related: #30051.

Decision log
  • Scope: chatd only. An earlier revision moved all workspace agent HTTP traffic, including workspace apps and wsproxy, to per-workspace-agent transports. It was dropped as too large a change for the goal.
  • One keep-alive client for all requests. Earlier revisions sent only the requests that http.Transport resends after a reused connection fails on the kept connection, and every other request on a new one. That needed a copy of net/http's resend rule, an Idempotency-Key header the workspace agent never reads, and the request context passed to the dial as a context value. It kept POSTs working across a workspace agent crash; this revision accepts one failed request there instead.
  • No per-method retry in the SDK. It would bring back two client paths and per-method routing, and tool-call deduplication in the workspace agent is in memory, so a retry after a restart could still run a tool call twice.
  • 15 s in coderd, 20 s in the workspace agent: a request sent on a connection the workspace agent already closed can fail, so coderd must close idle connections first for every workspace agent version, and all of them use 20 s. A request reaches the workspace agent after coderd's idle time plus one round trip, so the 5 s margin covers round trips up to 5 s. A longer coderd timeout would need a workspace agent version check.
  • 15 s is a literal, not AgentHTTPAPIServerIdleTimeout - 5*time.Second: a derived value would rise with the constant, past the 20 s older workspace agents use. A test fails if the constant drops to 15 s or less.
  • Measured with a real workspace agent at its 20 s idle timeout: after gaps of 14.5 s the next POST reused the connection, after 15.5 s to 25 s it opened a new one, and all 15 POSTs succeeded and ran once. Reconnecting PTY, SSH and the WatchGit websocket on a keep-alive connection that held an idle connection succeeded 9 of 9.
  • Dials on the keep-alive client end when the AgentConn closes instead of when their request ends, so no request context has to reach the dial.
  • HTTP/2 was not considered further.
  • Reviewed by GPT 6.1 Sol and Claude Opus 5.5. Applied: the option doc no longer states net/http's resend rule, newAPIClient builds its own transport, the timeout comment says why it is a literal, a test that Close ends a dial a request still waits on (CloseIdleConnections cancels only dials no request waits on), a deterministic test of the drain, and the removal of tests that asserted net/http's resend rule or the rejected routing. Not applied: that Close leaves in-use connections open (CloseIdleConnections also closes connections that become idle later, net/http/transport.go 902 and 1134), and that the nolint on the drain is unneeded (CI's new-code lint gate fails without it).

🤖 This PR was created with the help of Coder Agents, and will be reviewed by a human. 🏂🏻

@mafredri

This comment has been minimized.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-30T09:15:47.485978Z c9caa57 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

This comment has been minimized.

@mafredri
mafredri force-pushed the agent-api-conn-pool branch 5 times, most recently from 0db9f77 to fad9c05 Compare September 30, 2026 13:10
@linear-code

linear-code Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

CODAGT-1261

CODAGT-1625

@mafredri mafredri changed the title perf: pool agent API connections in coderd perf: reuse agent API connections through per-agent transports Sep 30, 2026
@mafredri
mafredri force-pushed the agent-api-conn-pool branch 4 times, most recently from caa389e to c78e68a Compare October 7, 2026 13:16
@mafredri
mafredri force-pushed the agent-api-conn-pool branch 2 times, most recently from 415b428 to c90d8bc Compare October 9, 2026 13:10
@mafredri mafredri changed the title perf: reuse agent API connections through per-agent transports perf: reuse workspace agent API connections in chatd Oct 9, 2026
@mafredri
mafredri force-pushed the agent-api-conn-pool branch 5 times, most recently from 77f4783 to 579f037 Compare October 13, 2026 11:33
@mafredri
mafredri force-pushed the agent-api-conn-pool branch from 579f037 to c438655 Compare October 21, 2026 08:21
@github-actions github-actions Bot added the stale This issue is like stale bread. label Nov 4, 2026
Every workspace agent API request opens a new TCP connection over the
tailnet: a reachability check, a handshake and the request, 3 round
trips. A caller that holds an AgentConn for many requests, as chatd does
for a chat turn, pays that on each one.

AgentConnOptions.APIIdleConnTimeout keeps one connection open between
requests. Only requests that http.Transport resends on a new connection
when a reused one fails use it, so the agent closing the connection
while it is idle fails no request. Tool-call requests and CancelToolCall
send Idempotency-Key, which the agent's dedup makes safe, and LS sends
the key with no value. Close closes the idle connection. Without the
option nothing changes.
chatd holds a workspace agent connection for a chat turn, so its tool
calls can share one API connection. Its connections now keep one idle
API connection for up to 5 minutes, and release closes it. Other
callers of ServerTailnet.AgentConn are unchanged.
http.Transport reuses a connection only after the response body was
read to EOF. For a chunked body, the JSON decoder can return before
the terminating chunk is read, and the connection is then closed. In
a local test with 20 KB and 200 KB chunked responses, 4 or 5 of the
next 9 requests had to dial again. TestServerTailnet_ChatAgentConn
failed once in CI on a request that dialed again; its log does not
show whether that response was chunked.

decodeAgentJSON now reads up to 4 KiB past the decoded value.
The agent's HTTP API server set no IdleTimeout, so Go used the 20 s
ReadTimeout and closed a connection chatd keeps between tool calls
whenever the model took longer than that. On dogfood, 13% of the gaps
between workspace tool calls in a chat turn were longer than 20 s, and
0.04% longer than 5 minutes.

The agent now keeps idle API connections for 5 minutes, and chatd uses
the same value. Other clients disable keep-alive, so the agent closes
their connections after each response as before. Checked with real
agent processes: a request 25 s or 60 s after the previous one reused
the connection, where an agent built from main had closed it.
Each request built a new http.Client and, unless it went on the kept
connection, a new http.Transport, and the request context was passed
through a struct field into a context value for the dial. The conn now
builds one client and both transports once, and the transport links the
dial to the request's own context.

Resendable requests now send Idempotency-Key with no value, which
http.Transport documents as idempotent and does not send. Tool calls
sent the tool call ID twice, and the agent reads neither header copy:
its dedup uses Coder-Tool-Call-Id.
ServerTailnet now closes an AgentConn when it is released, so the
connection that keeps an idle API connection needs no wrapper, and is
named for what it does instead of for chatd.

It keeps the idle connection for 2 minutes, below the workspace
agent's 5, so coderd normally closes it first and resends happen only
on agent restarts and older agents. On dogfood, 0.3% of the gaps
between workspace tool calls in a chat turn were longer than 2 minutes.
A connection with APIIdleConnTimeout sends every request through one
keep-alive client. Picking which requests may use a kept connection
meant copying net/http's resend rule, marking requests with an
Idempotency-Key the agent never reads, and passing the request context
to the dial as a context value.

When a kept connection was closed by the agent, http.Transport resends
a GET on a new connection and a POST returns the error. The timeout is
set below the agent's idle timeout, so that leaves an agent restart.
Measured with a real agent restarted by SIGKILL with the same ID: 6 of
6 POSTs failed and none ran; the next request succeeded.

Dials on the keep-alive client end when the conn closes instead of
when their request ends, because http.Transport keeps a dial running
after its request is canceled.
Every workspace agent built so far closes idle API connections after
20 s: Go used the 20 s ReadTimeout because IdleTimeout was unset. A
POST sent on a connection the agent already closed returns an error,
so coderd must close idle connections first, for every agent version.

The agent keeps 20 s, now as AgentHTTPAPIServerIdleTimeout, so the
value is defined once and a test fails if it drops to or below the 15
s coderd keeps.

On dogfood, 80% of the gaps between workspace tool calls in a chat are
shorter than 15 s (24,646 gaps over 74 chats). Measured against a real
agent: after gaps of 14.5 s the next POST reused the connection, after
15.5 s to 25 s it opened a new one, and all 15 succeeded.
The new-code lint gate flags ignored errors. A failed drain only stops
http.Transport from reusing the connection, and the response is
already decoded, so returning the error would fail a request that
succeeded.
newAPIClient took a transport and overwrote its DialContext. It now
builds the transport from the idle timeout.

The option doc no longer states http.Transport's resend rule: it says a
request on a connection the workspace agent closed can fail, and that
Close ends the client's dials and connections. The constraint from
older workspace agents moves to coderd, which picks the timeout.
- CloseEndsKeepAliveDial closes while the request still waits on its
  dial. http.Transport.CloseIdleConnections cancels only dials no
  request waits on, so the old test, which closed after the request
  ended, passed without endKeepAliveDials.
- A decodeAgentJSON test with a body read in chunks replaces the
  repeated chunked requests over tailnet, so the drain is tested without
  depending on how much the decoder reads ahead.
- DroppedIdleAPIConn is gone: it asserted net/http's resend rule, which
  the SDK no longer depends on. So is the tool call case, left from the
  design that routed requests by it.
A value derived from workspacesdk.AgentHTTPAPIServerIdleTimeout would
rise with it past the 20 s that every existing workspace agent build
uses. The test that it stays below the constant moves into the internal
test file, which drops an export.
@mafredri
mafredri force-pushed the agent-api-conn-pool branch from f108b23 to 5e365f7 Compare November 4, 2026 10:59
@github-actions github-actions Bot removed the stale This issue is like stale bread. label Nov 5, 2026
@github-actions github-actions Bot added the stale This issue is like stale bread. label Nov 13, 2026
@github-actions github-actions Bot closed this Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stale This issue is like stale bread.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant