Repository navigation
Conversation
mafredri
force-pushed
the
agent-api-conn-pool
branch
from
September 30, 2026 08:52
9f1f2b5 to
c9caa57
Compare
This comment has been minimized.
This comment has been minimized.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This comment has been minimized.
This comment has been minimized.
mafredri
force-pushed
the
agent-api-conn-pool
branch
5 times, most recently
from
September 30, 2026 13:10
0db9f77 to
fad9c05
Compare
mafredri
force-pushed
the
agent-api-conn-pool
branch
from
September 30, 2026 16:39
fad9c05 to
a07db20
Compare
mafredri
force-pushed
the
agent-api-conn-pool
branch
4 times, most recently
from
October 7, 2026 13:16
caa389e to
c78e68a
Compare
mafredri
force-pushed
the
agent-api-conn-pool
branch
2 times, most recently
from
October 9, 2026 13:10
415b428 to
c90d8bc
Compare
mafredri
force-pushed
the
agent-api-conn-pool
branch
5 times, most recently
from
October 13, 2026 11:33
77f4783 to
579f037
Compare
mafredri
force-pushed
the
agent-api-conn-pool
branch
from
October 21, 2026 08:21
579f037 to
c438655
Compare
Every workspace agent API request opens a new TCP connection over the tailnet: a reachability check, a handshake and the request, 3 round trips. A caller that holds an AgentConn for many requests, as chatd does for a chat turn, pays that on each one. AgentConnOptions.APIIdleConnTimeout keeps one connection open between requests. Only requests that http.Transport resends on a new connection when a reused one fails use it, so the agent closing the connection while it is idle fails no request. Tool-call requests and CancelToolCall send Idempotency-Key, which the agent's dedup makes safe, and LS sends the key with no value. Close closes the idle connection. Without the option nothing changes.
chatd holds a workspace agent connection for a chat turn, so its tool calls can share one API connection. Its connections now keep one idle API connection for up to 5 minutes, and release closes it. Other callers of ServerTailnet.AgentConn are unchanged.
http.Transport reuses a connection only after the response body was read to EOF. For a chunked body, the JSON decoder can return before the terminating chunk is read, and the connection is then closed. In a local test with 20 KB and 200 KB chunked responses, 4 or 5 of the next 9 requests had to dial again. TestServerTailnet_ChatAgentConn failed once in CI on a request that dialed again; its log does not show whether that response was chunked. decodeAgentJSON now reads up to 4 KiB past the decoded value.
The agent's HTTP API server set no IdleTimeout, so Go used the 20 s ReadTimeout and closed a connection chatd keeps between tool calls whenever the model took longer than that. On dogfood, 13% of the gaps between workspace tool calls in a chat turn were longer than 20 s, and 0.04% longer than 5 minutes. The agent now keeps idle API connections for 5 minutes, and chatd uses the same value. Other clients disable keep-alive, so the agent closes their connections after each response as before. Checked with real agent processes: a request 25 s or 60 s after the previous one reused the connection, where an agent built from main had closed it.
Each request built a new http.Client and, unless it went on the kept connection, a new http.Transport, and the request context was passed through a struct field into a context value for the dial. The conn now builds one client and both transports once, and the transport links the dial to the request's own context. Resendable requests now send Idempotency-Key with no value, which http.Transport documents as idempotent and does not send. Tool calls sent the tool call ID twice, and the agent reads neither header copy: its dedup uses Coder-Tool-Call-Id.
ServerTailnet now closes an AgentConn when it is released, so the connection that keeps an idle API connection needs no wrapper, and is named for what it does instead of for chatd. It keeps the idle connection for 2 minutes, below the workspace agent's 5, so coderd normally closes it first and resends happen only on agent restarts and older agents. On dogfood, 0.3% of the gaps between workspace tool calls in a chat turn were longer than 2 minutes.
A connection with APIIdleConnTimeout sends every request through one keep-alive client. Picking which requests may use a kept connection meant copying net/http's resend rule, marking requests with an Idempotency-Key the agent never reads, and passing the request context to the dial as a context value. When a kept connection was closed by the agent, http.Transport resends a GET on a new connection and a POST returns the error. The timeout is set below the agent's idle timeout, so that leaves an agent restart. Measured with a real agent restarted by SIGKILL with the same ID: 6 of 6 POSTs failed and none ran; the next request succeeded. Dials on the keep-alive client end when the conn closes instead of when their request ends, because http.Transport keeps a dial running after its request is canceled.
Every workspace agent built so far closes idle API connections after 20 s: Go used the 20 s ReadTimeout because IdleTimeout was unset. A POST sent on a connection the agent already closed returns an error, so coderd must close idle connections first, for every agent version. The agent keeps 20 s, now as AgentHTTPAPIServerIdleTimeout, so the value is defined once and a test fails if it drops to or below the 15 s coderd keeps. On dogfood, 80% of the gaps between workspace tool calls in a chat are shorter than 15 s (24,646 gaps over 74 chats). Measured against a real agent: after gaps of 14.5 s the next POST reused the connection, after 15.5 s to 25 s it opened a new one, and all 15 succeeded.
The new-code lint gate flags ignored errors. A failed drain only stops http.Transport from reusing the connection, and the response is already decoded, so returning the error would fail a request that succeeded.
newAPIClient took a transport and overwrote its DialContext. It now builds the transport from the idle timeout. The option doc no longer states http.Transport's resend rule: it says a request on a connection the workspace agent closed can fail, and that Close ends the client's dials and connections. The constraint from older workspace agents moves to coderd, which picks the timeout.
- CloseEndsKeepAliveDial closes while the request still waits on its dial. http.Transport.CloseIdleConnections cancels only dials no request waits on, so the old test, which closed after the request ended, passed without endKeepAliveDials. - A decodeAgentJSON test with a body read in chunks replaces the repeated chunked requests over tailnet, so the drain is tested without depending on how much the decoder reads ahead. - DroppedIdleAPIConn is gone: it asserted net/http's resend rule, which the SDK no longer depends on. So is the tool call case, left from the design that routed requests by it.
A value derived from workspacesdk.AgentHTTPAPIServerIdleTimeout would rise with it past the 20 s that every existing workspace agent build uses. The test that it stays below the constant moves into the internal test file, which drops an export.
mafredri
force-pushed
the
agent-api-conn-pool
branch
from
November 4, 2026 10:59
f108b23 to
5e365f7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every workspace agent API request from coderd opens a new TCP connection over the tailnet: a reachability check, a handshake and the request, 3 round trips. A chatd
read_fileis 1 request andexecuteis 2, so at a 462 ms round trip time they take 1.4 s and 2.8 s before the workspace agent does any work. This PR keeps chatd's connections to the workspace agent's HTTP API server open between requests, so a request on a kept connection takes 1 round trip.AgentConnOptions.APIIdleConnTimeoutsends every request of theAgentConnthrough one keep-alivehttp.Client, which closes connections idle for longer than the timeout. Without the option nothing changes: each request gets its own client with keep-alives disabled, as on main.Closeends the client's dials and closes its connections; a connection in use closes when its response ends.http.Transportkeeps a dial running after its request is canceled.decodeAgentJSONreads up to 4 KiB after the decoded value.http.Transportreuses a connection only after the response body was read to EOF, and the JSON decoder can stop before the end of a chunked body.ServerTailnet.keepAliveAgentConn, which sets the option to 15 s. LikeAgentConn, it logs and counts a workspace agent that does not answer (feat: add ServerTailnet logs and metrics for unreachable agents #30133). Releasing a connection fromServerTailnet.AgentConnorkeepAliveAgentConnnow closes it. For other callers this changes nothing, because their connections keep no idle connections.IdleTimeouttoworkspacesdk.AgentHTTPAPIServerIdleTimeout, 20 s. Every workspace agent already used 20 s, because Go usesReadTimeoutwhenIdleTimeoutis unset. A test checks that coderd's 15 s stays below it.Measured at a 462 ms round trip time: a
ServerTailnetconnected over dogfood's DERP to a workspace agent in the same workspace, withtc qdisc add dev eth0 root netem delay 225ms, so both directions are delayed. 3 turns, each acquires a connection, sends 5read_fileand 5executecalls, and releases it.read_file(1 request), p50execute(2 requests), p50Each turn's first request dials and takes 1,388 ms. With #30051, chatd holds one connection per chat runner, so a turn's tool calls share it; without it, one per generation step.
End to end, with the same
ServerTailnetand round trip time: a chatd server with a scripted model running 4 tool steps per turn (read_file,execute,read_file,execute) and a reply, all builds on the same main commit. Medians of 3 turns after a warm-up turn; a step is the time between two model requests, so the tool call plus the next step's preparation.read_filestepexecutestepIn every build, the first model request of a turn came 1.9 s after the chat was created. Spans show where it goes: the acquisition's reachability check (464 ms), then an
LSon a new connection (1,388 ms).Behavior changes:
coder agentkilled with SIGKILL and restarted, 6 of 6 POSTs failed and none ran, 3 of 3 GETs succeeded becausehttp.Transportresent them, and the next request succeeded on a new connection. With SIGTERM the workspace agent closes its connections on shutdown, and 9 of 9 requests succeeded.keepAliveAgentConnkeeps up to 2,http.Transport's default per host, for example after parallel tool calls.Related: #30051.
Decision log
http.Transportresends after a reused connection fails on the kept connection, and every other request on a new one. That needed a copy of net/http's resend rule, anIdempotency-Keyheader the workspace agent never reads, and the request context passed to the dial as a context value. It kept POSTs working across a workspace agent crash; this revision accepts one failed request there instead.AgentHTTPAPIServerIdleTimeout - 5*time.Second: a derived value would rise with the constant, past the 20 s older workspace agents use. A test fails if the constant drops to 15 s or less.WatchGitwebsocket on a keep-alive connection that held an idle connection succeeded 9 of 9.AgentConncloses instead of when their request ends, so no request context has to reach the dial.newAPIClientbuilds its own transport, the timeout comment says why it is a literal, a test thatCloseends a dial a request still waits on (CloseIdleConnectionscancels only dials no request waits on), a deterministic test of the drain, and the removal of tests that asserted net/http's resend rule or the rejected routing. Not applied: thatCloseleaves in-use connections open (CloseIdleConnectionsalso closes connections that become idle later,net/http/transport.go902 and 1134), and that thenolinton the drain is unneeded (CI's new-code lint gate fails without it).