Repository navigation
fix: log workspace shutdown reasons for SSH connections - #30515
Conversation
Workspace stops and deletes closed active SSH connections with generic shutdown or process-exit reasons. This obscured the workspace action that ended a connection and made related logs harder to correlate. We added a bounded lookup in the workspace agent so coderd can provide the shutdown build's reason before the agent's SSH server closes active connections. We changed the agent's SSH logging to include that reason and the build details alongside the client session ID for ordinary sessions and connections used only for port forwarding. Earlier client or network disconnects retained their original attribution. We changed coderd's connection monitor to keep the preceding successful start build's agent connected while a stop or delete job is pending or running. This preserves the connection for the lookup and final logs, and changes the agent's connection status during stopping. Completed builds remain usable only until the monitor retires the connection; unavailable or unknown reasons fall back to server_shutdown. We added regression coverage for shutdown attribution, fallback behavior, and REST stop and delete requests. We also advanced the agent API to version 2.13, which new agents now require from coderd. - resolves https://linear.app/codercom/issue/CLIENT-665/close-workspace-tunnel-with-a-reason
Background agent logs could reach the test logger after cleanup completed, causing the shutdown test to panic during the flake check. We changed the shared log-capture helper to keep test-log writes and cleanup under the same lock. Cleanup now waits for active writes, while later logs remain available for assertions without writing to a completed test. We added regression coverage for both behaviors.
The SSH shutdown regression could leave a sleep child holding an output pipe on macOS, delaying server closure until the test context expired. We changed the SSH test commands to replace the shell with the waiting process. This keeps shutdown signals directed at the tracked process and preserves coverage for stdin EOF and channels closed before shutdown.
The workspace shutdown branch was behind main. We merged the latest main changes to keep the PR current.
code-asher
left a comment
There was a problem hiding this comment.
I am still getting familiar with all the networking code so I feel uncertain about what I am thinking 😆 but I feel like instead of the agent reaching out to coderd to get the workspace status when a connection closes, we should be able to use the more general reason by disconnecting the peers, so coderd could disconnect all the peers with "workspace stopping" before stopping the workspace.
I think Spike is on PTO but looking at people who have been in this code, @deansheather or maybe @ethanndickson would you have some insight on which would be the best way for the agent to know that the workspace stopping, or if maybe there is some other better way? 🙏
The only other thing I can think of is to send the agent a specific signal when stopping the workspace so it knows that is why it is shutting down, but idk if this is possible.
|
Yeah, sure we can definetly say that we are closing the ssh connection because we are shutting down the agent, but don't we want to go into a more granular reason: like the fact that we are shutting down the agent because a user requested it? I agree with you that coderd should proactively tell the agent why the workspace is stopping, but I don't necessarily know where to start. |
Determining workspace shutdown reasons in the agent required an extra control-plane request during shutdown, even though coderd and the CLI already had access to workspace build state. We moved reason enrichment to coderd when it handles connection reports. Coderd translated generic server shutdowns into workspace shutdowns for running or successful stop and delete builds and logged the build context alongside the client session ID. Existing client and network reasons were preserved, and unavailable or unrelated build state retained the generic reason. We changed the CLI to log shutdown context from its existing workspace watcher once per build while preserving graceful SSH closure. We retained the agent's connection tracking so it reported server shutdowns without replacing earlier client or network closures. We removed the shutdown-context RPC, its API version bump, and the connection-monitor allowance because the workspace context was resolved by coderd and the CLI. We added coverage for reason enrichment, session ID propagation, and PostgreSQL persistence with and without a session ID. - resolves [https://linear.app/codercom/issue/CLIENT-665/close-workspace-tunnel-with-a-reason](https://linear.app/codercom/issue/CLIENT-665/close-workspace-tunnel-with-a-reason)
The workspace shutdown branch was behind main, including changes to the connection log format. We merged the latest main changes and updated the connection-log tests to use the connection method and application fields. We preserved the shutdown reason and client session ID assertions.
The agent's SSH server decided whether a connection or session ended because of a shutdown by recording the first termination on every connection and channel. It wrapped reads and writes on the SSH transport and on JetBrains forwarding channels, and it tracked each session channel separately, so that a later shutdown never replaced an earlier client or transport close. That precision required a lot of machinery for a reason that coderd and the CLI already refined from workspace build state. The CLI also reported a workspace shutdown as soon as a stop or delete build was queued, while coderd only attributed a disconnect to a running or completed build, so the two disagreed when a queued build was canceled or failed. We simplify the agent's SSH server so that it marks each open connection when the server closes, and the session, connection, and JetBrains logs report a server shutdown from that mark together with the client session ID. A connection that the client closes before the shutdown keeps its original reason, because it leaves the server before the shutdown marks the open connections. A client disconnect that races with the shutdown, or a command that outlives its closed session channel, is now reported as a server shutdown, which we accept in exchange for the simpler model. We remove the tests that covered the dropped tracking. We change the CLI's workspace watcher to log the shutdown reason only once the stop or delete job is running or has succeeded, which matches coderd, and we strengthen its test to show that a queued build is not logged. We also restructure the shutdown checks in the CLI and in coderd's connection log handling into named conditions without changing their behavior.
code-asher
left a comment
There was a problem hiding this comment.
We should do this for reconnecting pty connections as well!
When the agent shut down, the reconnecting PTY server closed each PTY and its attached connections, and the connection handler then returned without an error. The server therefore logged and reported those connections as graceful closes. Its fallback for a forced shutdown reported a free-form reason that coderd did not recognize, so coderd never refined either case into a workspace shutdown. We change the reconnecting PTY server so that it reports any connection that ends after the agent begins shutting down as a server shutdown initiated by the agent, and its log keeps the client session ID. The forced shutdown fallback now reports the same standard reason, which lets coderd translate it into a workspace stop or delete in the connection log, as it already does for SSH. A client that closes its connection before the shutdown still keeps its graceful reason. We add a test for the reconnecting PTY server that covers an agent shutdown and a client close before the shutdown, and we extend the coderd connection log tests with a reconnecting PTY shutdown case.
Coderd's agent connection monitor closed an agent's RPC connection as outdated as soon as a stop or delete build existed, often before the workspace was torn down and its SSH sessions closed. The agent dropped any connection report that was in flight when the RPC closed, and the agent authentication query only let the previous agent reconnect during a stop, so the server shutdown reports that coderd refines into workspace stops and deletes could be lost. The enterprise connection log test connected without the monitor, so it did not catch this. The agent's SSH server also stored a shutdown flag on every connection, the CLI logged a message that suggested it had requested the shutdown, and coderd logged each refined disconnect at info level, unlike every other closed connection. We change the connection monitor to keep the previous successful start build's agent connected while the next build is a pending or running stop or delete, and we extend the agent authentication query to the same rule, so the agent can deliver its final reports and reconnect for a delete as it already could for a stop. The monitor retires the connection once the shutdown job finishes. We change the enterprise test to connect as a real agent and to report only after the monitor has checked the shutdown build, and we add monitor and authentication coverage for stop and delete builds. We simplify the agent's SSH server so that it reports a server shutdown from its own closing state instead of a per-connection flag, and it now waits for JetBrains forwarding channels to report their close, because their handlers run outside the connection tracking. We rename the CLI message to say that the workspace is shutting down, and we remove coderd's info log, since the agent and CLI logs together with the connection log table already carry the reason and the client session ID.
The linter for new code rejected the agent's SSH server because the new channel tracking method suppressed a revive warning without explaining why, which failed the CI lint job. We add the explanation to the directive. The method takes a control flag to add or remove a channel, matching the existing connection and session tracking methods, so the suppression stays.
Currently, coderd attributes an agent's server shutdown disconnect to a workspace shutdown only when the latest build is a stop or delete. The agent connection monitor and the agent authentication query already keep a previous agent connected only while a newer build stops or deletes its workspace, so the transition check repeats that rule and misses a shutdown if a restart's builds are ever queued together. The shutdown tests for the agent's SSH and reconnecting PTY servers also duplicate the existing agent connection report tests and rely on their own fake connections and reporters. This commit makes coderd compare the latest build with the build that created the agent, which the agent API already receives, while keeping the timing and job status checks. With this change, any newer build whose job is running or has succeeded counts as the cause of the shutdown. This commit also moves the shutdown coverage into the existing agent tests for SSH sessions, reconnecting PTY connections, and JetBrains port forwarding, which now check both the disconnect report and the agent's log line, and it removes the separate reconnecting PTY test and the JetBrains unit test along with their fakes. The SSH server keeps registering JetBrains forwarding channels until their close is reported, because their handlers outlive the connection tracking, and that registration now sits next to the server's other tracking methods. The reconnecting PTY server's comment now explains why a shutdown reaches its handler without an error.
Currently, the agent waits for its logs to be sent during shutdown but not for its connection reports. Reconnecting PTY connections also finish on their own after the shutdown begins, so when a shutdown completes quickly the agent stops its report loop before the disconnect reports for the connections it closes reach coderd, and coderd cannot record why those connections closed. The reconnecting PTY server can also log and report the same close twice when its forced shutdown path and its connection handler both finish a connection. This commit makes the agent's connection reporter track connections that have not reported their disconnect and reports that are still queued or being sent, and the agent now waits for both during shutdown before it stops reporting. The wait stays bounded by the agent's existing shutdown deadline, and a connection reports its disconnect only once. This commit also makes the reconnecting PTY server log and report each close once, whichever path finishes the connection first. With this change, the disconnects caused by an agent shutdown reach coderd even when the shutdown is fast.
…-reasons # Conflicts: # agent/agent_test.go # agent/agentssh/agentssh.go # agent/agentssh/jetbrainstrack.go # agent/reconnectingpty/server.go # agent/upgrade_test.go # coderd/agentapi/connectionlog_test.go
Move the agentssh shutdown reason tests into agentssh_test.go and drop the AgentShutdown case, which TestAgent_SSHUpgrade now covers by also asserting the shutdown fields on the SSH connection log. Wait for shell output before shutting the agent down so the disconnect report always records transmitted bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the negative disconnect code in connection report assertions with a wantNonZeroCode constant, matching the existing byte constants. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Workspace stops and deletes closed active SSH connections with generic shutdown or process-exit reasons. This obscured the workspace action that ended a connection and made related logs harder to correlate.
The agent’s SSH tracking and logging now identify connections closed by the SSH server as server_shutdown, while preserving earlier client or network disconnects. The agent includes the client session ID when available for ordinary sessions and connections used only for port forwarding.
Coderd now enriches shutdown reasons when it processes connection reports. It translates server_shutdown into workspace_stopped when the latest build is a running or successful stop or delete requested before the connection closed. Coderd persists the enriched reason and logs the build reason, transition, and build ID alongside the client session ID. Unavailable or unrelated build state retains the generic shutdown reason.
The CLI now logs shutdown context once per stop or delete build through its existing workspace watcher. This includes the client session ID when available and preserves the existing graceful SSH shutdown behavior.
Regarding the PTY connections, when the agent shut down, the reconnecting PTY server closed each PTY
and its attached connections, and the connection handler then returned without an error. The server therefore logged and reported those connections as graceful closes. Its fallback for a forced shutdown
reported a free-form reason that coderd did not recognize, so coderd never refined either case into a workspace shutdown.
We change the reconnecting PTY server so that it reports any connection that ends after the agent begins shutting down as a server shutdown initiated by the agent, and its log keeps the client session ID. The forced shutdown fallback now reports the same standard reason, which lets coderd translate it into a workspace stop or delete in the connection log, as it already does for SSH. A client that closes its connection before the shutdown still keeps its graceful reason.
Regression tests cover shutdown attribution, fallback behavior, CLI logging, and PostgreSQL persistence with and without a client session ID. The test logging helper also synchronizes with cleanup to prevent logging after a test completes