Skip to content

fix: log workspace shutdown reasons for SSH connections - #30515

Merged
fioan89 merged 21 commits into
mainfrom
workspace-shutdown-reasons
Oct 19, 2026
Merged

fioan89 merged 21 commits into
mainfrom
workspace-shutdown-reasons

Conversation

@fioan89

@fioan89 fioan89 commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Workspace stops and deletes closed active SSH connections with generic shutdown or process-exit reasons. This obscured the workspace action that ended a connection and made related logs harder to correlate.

The agent’s SSH tracking and logging now identify connections closed by the SSH server as server_shutdown, while preserving earlier client or network disconnects. The agent includes the client session ID when available for ordinary sessions and connections used only for port forwarding.

Coderd now enriches shutdown reasons when it processes connection reports. It translates server_shutdown into workspace_stopped when the latest build is a running or successful stop or delete requested before the connection closed. Coderd persists the enriched reason and logs the build reason, transition, and build ID alongside the client session ID. Unavailable or unrelated build state retains the generic shutdown reason.

The CLI now logs shutdown context once per stop or delete build through its existing workspace watcher. This includes the client session ID when available and preserves the existing graceful SSH shutdown behavior.

Regarding the PTY connections, when the agent shut down, the reconnecting PTY server closed each PTY
and its attached connections, and the connection handler then returned without an error. The server therefore logged and reported those connections as graceful closes. Its fallback for a forced shutdown
reported a free-form reason that coderd did not recognize, so coderd never refined either case into a workspace shutdown.

We change the reconnecting PTY server so that it reports any connection that ends after the agent begins shutting down as a server shutdown initiated by the agent, and its log keeps the client session ID. The forced shutdown fallback now reports the same standard reason, which lets coderd translate it into a workspace stop or delete in the connection log, as it already does for SSH. A client that closes its connection before the shutdown still keeps its graceful reason.

Regression tests cover shutdown attribution, fallback behavior, CLI logging, and PostgreSQL persistence with and without a client session ID. The test logging helper also synchronizes with cleanup to prevent logging after a test completes

Workspace stops and deletes closed active SSH connections with generic
shutdown or process-exit reasons. This obscured the workspace action that
ended a connection and made related logs harder to correlate.

We added a bounded lookup in the workspace agent so coderd can provide
the shutdown build's reason before the agent's SSH server closes active
connections. We changed the agent's SSH logging to include that reason
and the build details alongside the client session ID for ordinary
sessions and connections used only for port forwarding. Earlier client
or network disconnects retained their original attribution.

We changed coderd's connection monitor to keep the preceding successful
start build's agent connected while a stop or delete job is pending or
running. This preserves the connection for the lookup and final logs,
and changes the agent's connection status during stopping. Completed
builds remain usable only until the monitor retires the connection;
unavailable or unknown reasons fall back to server_shutdown.

We added regression coverage for shutdown attribution, fallback behavior,
and REST stop and delete requests. We also advanced the agent API to
version 2.13, which new agents now require from coderd.

- resolves https://linear.app/codercom/issue/CLIENT-665/close-workspace-tunnel-with-a-reason
@fioan89
fioan89 requested review from code-asher and removed request for johnstcn and spikecurtis October 8, 2026 17:44
@linear-code

linear-code Bot commented Oct 8, 2026

Copy link
Copy Markdown

CLIENT-665

@github-actions github-actions Bot added the community Pull Requests and issues created by the community. label Oct 8, 2026
Background agent logs could reach the test logger after cleanup completed,
causing the shutdown test to panic during the flake check.

We changed the shared log-capture helper to keep test-log writes and cleanup
under the same lock. Cleanup now waits for active writes, while later logs
remain available for assertions without writing to a completed test.
We added regression coverage for both behaviors.
The SSH shutdown regression could leave a sleep child holding an output
pipe on macOS, delaying server closure until the test context expired.

We changed the SSH test commands to replace the shell with the waiting
process. This keeps shutdown signals directed at the tracked process and
preserves coverage for stdin EOF and channels closed before shutdown.
The workspace shutdown branch was behind main.

We merged the latest main changes to keep the PR current.

@code-asher code-asher left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am still getting familiar with all the networking code so I feel uncertain about what I am thinking 😆 but I feel like instead of the agent reaching out to coderd to get the workspace status when a connection closes, we should be able to use the more general reason by disconnecting the peers, so coderd could disconnect all the peers with "workspace stopping" before stopping the workspace.

I think Spike is on PTO but looking at people who have been in this code, @deansheather or maybe @ethanndickson would you have some insight on which would be the best way for the agent to know that the workspace stopping, or if maybe there is some other better way? 🙏

The only other thing I can think of is to send the agent a specific signal when stopping the workspace so it knows that is why it is shutting down, but idk if this is possible.

@fioan89

fioan89 commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator Author

Yeah, sure we can definetly say that we are closing the ssh connection because we are shutting down the agent, but don't we want to go into a more granular reason: like the fact that we are shutting down the agent because a user requested it?

I agree with you that coderd should proactively tell the agent why the workspace is stopping, but I don't necessarily know where to start.
Currently, the provisioner changes the workspace infrastructure, which causes the agent to terminate, but that termination doesn’t carry the workspace’s shutdown reason. Since we already have an RPC connection between the agent and coderd, I thought a single best-effort lookup during agent shutdown was a reasonable, small change for this logging requirement.

Determining workspace shutdown reasons in the agent required an extra
control-plane request during shutdown, even though coderd and the CLI
already had access to workspace build state.

We moved reason enrichment to coderd when it handles connection reports.
Coderd translated generic server shutdowns into workspace shutdowns for
running or successful stop and delete builds and logged the build context
alongside the client session ID. Existing client and network reasons were
preserved, and unavailable or unrelated build state retained the generic
reason.

We changed the CLI to log shutdown context from its existing workspace
watcher once per build while preserving graceful SSH closure. We retained
the agent's connection tracking so it reported server shutdowns without
replacing earlier client or network closures.

We removed the shutdown-context RPC, its API version bump, and the
connection-monitor allowance because the workspace context was resolved
by coderd and the CLI. We added coverage for reason enrichment, session ID
propagation, and PostgreSQL persistence with and without a session ID.

- resolves [https://linear.app/codercom/issue/CLIENT-665/close-workspace-tunnel-with-a-reason](https://linear.app/codercom/issue/CLIENT-665/close-workspace-tunnel-with-a-reason)
The workspace shutdown branch was behind main, including changes to the
connection log format.

We merged the latest main changes and updated the connection-log tests to
use the connection method and application fields. We preserved the
shutdown reason and client session ID assertions.
The agent's SSH server decided whether a connection or session ended
because of a shutdown by recording the first termination on every
connection and channel. It wrapped reads and writes on the SSH transport
and on JetBrains forwarding channels, and it tracked each session channel
separately, so that a later shutdown never replaced an earlier client or
transport close. That precision required a lot of machinery for a reason
that coderd and the CLI already refined from workspace build state. The
CLI also reported a workspace shutdown as soon as a stop or delete build
was queued, while coderd only attributed a disconnect to a running or
completed build, so the two disagreed when a queued build was canceled
or failed.

We simplify the agent's SSH server so that it marks each open connection
when the server closes, and the session, connection, and JetBrains logs
report a server shutdown from that mark together with the client session
ID. A connection that the client closes before the shutdown keeps its
original reason, because it leaves the server before the shutdown marks
the open connections. A client disconnect that races with the shutdown,
or a command that outlives its closed session channel, is now reported
as a server shutdown, which we accept in exchange for the simpler model.
We remove the tests that covered the dropped tracking.

We change the CLI's workspace watcher to log the shutdown reason only
once the stop or delete job is running or has succeeded, which matches
coderd, and we strengthen its test to show that a queued build is not
logged. We also restructure the shutdown checks in the CLI and in
coderd's connection log handling into named conditions without changing
their behavior.
@fioan89
fioan89 requested a review from code-asher October 14, 2026 21:16

@code-asher code-asher left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should do this for reconnecting pty connections as well!

Comment thread agent/agentssh/agentssh.go Resolved
Comment thread cli/ssh.go Resolved
Comment thread coderd/agentapi/connectionlog.go Resolved
Comment thread coderd/agentapi/connectionlog.go Resolved
Comment thread coderd/agentapi/connectionlog.go Resolved
Comment thread coderd/agentapi/connectionlog.go Resolved
When the agent shut down, the reconnecting PTY server closed each PTY
and its attached connections, and the connection handler then returned
without an error. The server therefore logged and reported those
connections as graceful closes. Its fallback for a forced shutdown
reported a free-form reason that coderd did not recognize, so coderd
never refined either case into a workspace shutdown.

We change the reconnecting PTY server so that it reports any connection
that ends after the agent begins shutting down as a server shutdown
initiated by the agent, and its log keeps the client session ID. The
forced shutdown fallback now reports the same standard reason, which
lets coderd translate it into a workspace stop or delete in the
connection log, as it already does for SSH. A client that closes its
connection before the shutdown still keeps its graceful reason.

We add a test for the reconnecting PTY server that covers an agent
shutdown and a client close before the shutdown, and we extend the
coderd connection log tests with a reconnecting PTY shutdown case.
Coderd's agent connection monitor closed an agent's RPC connection as
outdated as soon as a stop or delete build existed, often before the
workspace was torn down and its SSH sessions closed. The agent dropped
any connection report that was in flight when the RPC closed, and the
agent authentication query only let the previous agent reconnect during
a stop, so the server shutdown reports that coderd refines into
workspace stops and deletes could be lost. The enterprise connection
log test connected without the monitor, so it did not catch this. The
agent's SSH server also stored a shutdown flag on every connection, the
CLI logged a message that suggested it had requested the shutdown, and
coderd logged each refined disconnect at info level, unlike every other
closed connection.

We change the connection monitor to keep the previous successful start
build's agent connected while the next build is a pending or running
stop or delete, and we extend the agent authentication query to the
same rule, so the agent can deliver its final reports and reconnect
for a delete as it already could for a stop. The monitor retires the
connection once the shutdown job finishes. We change the enterprise
test to connect as a real agent and to report only after the monitor
has checked the shutdown build, and we add monitor and authentication
coverage for stop and delete builds.

We simplify the agent's SSH server so that it reports a server shutdown
from its own closing state instead of a per-connection flag, and it
now waits for JetBrains forwarding channels to report their close,
because their handlers run outside the connection tracking. We rename
the CLI message to say that the workspace is shutting down, and we
remove coderd's info log, since the agent and CLI logs together with
the connection log table already carry the reason and the client
session ID.
The linter for new code rejected the agent's SSH server because the new
channel tracking method suppressed a revive warning without explaining
why, which failed the CI lint job.

We add the explanation to the directive. The method takes a control
flag to add or remove a channel, matching the existing connection and
session tracking methods, so the suppression stays.

@code-asher code-asher left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Been testing it out and it is looking good! Both rpty and ssh connection when stopping or restarting the workspace are logging workspace_stopped everywhere 🎉

Comment thread agent/reconnectingpty/server_test.go Resolved
Comment thread agent/agentssh/agentssh.go Resolved
Comment thread agent/agentssh/shutdown.go Resolved
Comment thread agent/reconnectingpty/server.go Resolved
Comment thread agent/reconnectingpty/server.go Resolved
Comment thread coderd/agentapi/connectionlog.go Resolved
Currently, coderd attributes an agent's server shutdown disconnect to a
workspace shutdown only when the latest build is a stop or delete. The
agent connection monitor and the agent authentication query already keep
a previous agent connected only while a newer build stops or deletes its
workspace, so the transition check repeats that rule and misses a
shutdown if a restart's builds are ever queued together. The shutdown
tests for the agent's SSH and reconnecting PTY servers also duplicate
the existing agent connection report tests and rely on their own fake
connections and reporters.

This commit makes coderd compare the latest build with the build that
created the agent, which the agent API already receives, while keeping
the timing and job status checks. With this change, any newer build
whose job is running or has succeeded counts as the cause of the
shutdown.

This commit also moves the shutdown coverage into the existing agent
tests for SSH sessions, reconnecting PTY connections, and JetBrains port
forwarding, which now check both the disconnect report and the agent's
log line, and it removes the separate reconnecting PTY test and the
JetBrains unit test along with their fakes. The SSH server keeps
registering JetBrains forwarding channels until their close is reported,
because their handlers outlive the connection tracking, and that
registration now sits next to the server's other tracking methods. The
reconnecting PTY server's comment now explains why a shutdown reaches
its handler without an error.
Currently, the agent waits for its logs to be sent during shutdown but
not for its connection reports. Reconnecting PTY connections also finish
on their own after the shutdown begins, so when a shutdown completes
quickly the agent stops its report loop before the disconnect reports
for the connections it closes reach coderd, and coderd cannot record why
those connections closed. The reconnecting PTY server can also log and
report the same close twice when its forced shutdown path and its
connection handler both finish a connection.

This commit makes the agent's connection reporter track connections that
have not reported their disconnect and reports that are still queued or
being sent, and the agent now waits for both during shutdown before it
stops reporting. The wait stays bounded by the agent's existing shutdown
deadline, and a connection reports its disconnect only once. This commit
also makes the reconnecting PTY server log and report each close once,
whichever path finishes the connection first.

With this change, the disconnects caused by an agent shutdown reach
coderd even when the shutdown is fast.
…-reasons

# Conflicts:
#	agent/agent_test.go
#	agent/agentssh/agentssh.go
#	agent/agentssh/jetbrainstrack.go
#	agent/reconnectingpty/server.go
#	agent/upgrade_test.go
#	coderd/agentapi/connectionlog_test.go
Comment thread agent/agent.go Resolved
Comment thread agent/agent_test.go Resolved
Comment thread agent/agentssh/shutdown_test.go Resolved
fioan89 and others added 3 commits October 19, 2026 22:29
Move the agentssh shutdown reason tests into agentssh_test.go and drop
the AgentShutdown case, which TestAgent_SSHUpgrade now covers by also
asserting the shutdown fields on the SSH connection log.

Wait for shell output before shutting the agent down so the disconnect
report always records transmitted bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the negative disconnect code in connection report assertions
with a wantNonZeroCode constant, matching the existing byte constants.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@fioan89
fioan89 merged commit 1a65385 into main Oct 19, 2026
29 checks passed
@fioan89
fioan89 deleted the workspace-shutdown-reasons branch October 19, 2026 20:06
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 19, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

community Pull Requests and issues created by the community.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants