Skip to content

docs: add guide for tracing workspace connections - #30707

Closed
coder[bot] wants to merge 13 commits into
mainfrom
superblink/client-1032-trace-connections
Closed

coder[bot] wants to merge 13 commits into
mainfrom
superblink/client-1032-trace-connections

Conversation

@coder

@coder coder Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Adds a how-to guide under Monitoring for platform teams: find a user's client_session_id, collect client, coderd, and agent logs, correlate them, and read disconnect_reason and build_reason. Also links it from the connection logs service-log section and adds it to the manifest.

Resolves CLIENT-1032. Written ahead of merge under the multi-PR launch exception in the content guidelines. It describes behavior from these PRs, so it should merge after the open ones:

PR Status (2026-10-27) Covered in the guide
#30467 Merged 2026-10-14 Agent-reported app names (AppNameOrPort, such as cursor)
#30515 Merged 2026-10-19 (1a65385) The agent logs server_shutdown with disconnect_initiator on ssh connection complete; coderd records workspace_stopped in the connection log; build_id, build_reason, transition appear only on the coder ssh log line workspace is shutting down
#30389 Merged 2026-10-16 RxBytes / TxBytes on disconnected entries for SSH and web terminal connections
#30665 Closed without merging 2026-10-15 Removed: connection path logs and ConnectPath / DisconnectPath. @code-asher said on 2026-10-27 to leave the connection path out because it needs a new solution, so it isn't part of this guide.
coder/coder-jetbrains-toolbox#372 Merged 2026-10-16, unreleased (JET-34 Done) Toolbox Collect logs including coder ssh logs. Ships in plugin v0.9.5 (confirmed by @fioan89); the guide names v0.9.5 and keeps "ask for the files separately" for earlier versions.
coder/coder-jetbrains-toolbox#373 Merged 2026-10-19, unreleased The plugin now picks the coder ssh log directory itself, passes it to coder ssh with --log-dir, and adds coder-ssh*-<pid>.log files from the last three days (each cut to its last 50 MB) straight into the support bundle. Ships in v0.9.5; the JetBrains Toolbox section now describes the new log directory order and where the logs land in the bundle.
coder/vscode-coder#1164 Open, approved by @matifali and @code-asher (2026-10-27); ships in v1.17.2 (coder/vscode-coder#1165, open). Latest published release is v1.17.1 (2026-10-15). Changes the Coder output channel prefix from [session <first 8 characters>] to [client_session_id=<full 32-character ID>]. The guide now requires v1.17.2 and describes the full-ID prefix (c98046a, caf1bb7).
#30759 Ready for review since 2026-10-21, approved by @code-asher 2026-10-26, unmerged Not covered yet: client_session_id in the connection log API, a client_session_id: filter, and View related connections in the UI.
#31289 Merged 2026-10-26 (@fioan89, JET-11 Done) Toolbox support bundle requirements in docs/support/support-bundle.md: plugin 0.9.5 and JetBrains Toolbox 3.9.0; plugin 0.9.4 and earlier collect Toolbox logs without a Coder support bundle. "Before you begin" and the JetBrains Toolbox section now match and link it.

No 2.39 release branch or tag yet. Latest published Toolbox plugin release is still v0.9.4 (2026-08-26); the v0.9.5 draft release was re-created 2026-10-19 after #373 merged but isn't published. @fioan89 confirmed 2026-10-21 that v0.9.5 is the release with all the connection log work. Still a draft on 2026-10-27. The plugin changelog on main lists JetBrains Toolbox 3.7.2 as its minimum, while #31289 documents 3.9.0 for v0.9.5. coder/coder-jetbrains-toolbox#374 (ready for review since 2026-10-22, no approvals yet) adds the Toolbox connection log buffer (JET-4); the guide will cover it once it merges and ships.

Changes since the first draft

  • Removed connection path content (the ConnectPath / DisconnectPath fields, the path log messages, and the "Read connection paths" section) because feat: log and record DERP/direct connection path per agent connection #30665 closed without merging.
  • Toolbox: Collect logs doesn't include the coder ssh log files, so the guide asks for them separately. Tracked in JET-34.
  • 2026-10-16: kept the RxBytes / TxBytes row. @code-asher confirmed bytes reach the connection log service logs (not the API), so they can be ingested from there. No file changes today.
  • 2026-10-16: applied Nick's review: "earlier versions" wording, split sentences, and a link to the coder ssh log directory from the Toolbox section. Fixed the agent log location (coder-agent.log in CODER_AGENT_LOG_DIR, otherwise the system temp directory: $TMPDIR or /tmp on Linux and macOS, %TEMP% on Windows) and documented the VS Code log directory fallback (coder.proxyLogDirectory, then CODER_SSH_LOG_DIR, then the extension's global storage). Checked against cli/agent.go and vscode-coder pathResolver.ts.
  • 2026-10-16: dropped the Coder 2.39 version pins, since the docs are versioned. The guide now says to update agents to the server's version, because agents can lag behind the server. Client versions (VS Code extension, Toolbox plugin) stay because they ship separately.
  • 2026-10-17: feat: add byte tracking to ssh and pty connections #30389 merged. The RxBytes / TxBytes row now says the agent sets them only on disconnected entries for SSH and web terminal connections, and that older agents leave them empty (per coderd/agentapi/connectionlog.go and the migration column comments). "Before you begin" now lists bytes among the fields earlier agents don't record. Commit 32af7d2.
  • 2026-10-17: resolved the render question. cdr.dev/slog/v3 calls Value() on any driver.Valuer before encoding (map.go), so sql.NullInt64 renders as a plain number, or null when unset, not a {Int64, Valid} object.
  • 2026-10-17: applied Nick's two factual corrections from his 2026-10-16 review, checked against fix: log workspace shutdown reasons for SSH connections #30515 head 01a3b79f06. The agent log table now says ssh connection complete gets disconnect_reason, disconnect_expected, and disconnect_initiator only when the agent is shutting down (agent/agentssh/agentssh.go, shutdown.go). "Read disconnect reasons" now says SSH connections (as opposed to sessions) aren't reported to the connection log, and that the agent log (server_shutdown) and connection log (workspace_stopped) can differ, with the conditions from workspaceShutdownReason in coderd/agentapi/connectionlog.go. build_id, build_reason, and transition are now attributed to the coder ssh log's workspace is shutting down line (cli/ssh.go). Commit 55c8c16.
  • 2026-10-20: fix: log workspace shutdown reasons for SSH connections #30515 merged on 2026-10-19. The merged agent, CLI, and coderd changes match what the guide was checked against on 2026-10-17, so no file changes. The fix: log workspace shutdown reasons for SSH connections #30515 description says coderd logs the build reason, transition, and build ID, but merged coderd/agentapi/connectionlog.go only persists the enriched reason and doesn't log those fields, so the guide keeps them on the coder ssh line.
  • 2026-10-21: @fioan89 confirmed v0.9.5 is the Toolbox plugin release with all the connection log work. "Before you begin" now names v0.9.5 for session IDs and Collect logs (replacing the TODO). The JetBrains Toolbox section now gives the v0.9.5 coder ssh log directory order (plugin setting, then CODER_SSH_LOG_DIR, then ssh-logs in the plugin data directory) and says recent coder ssh logs land in the support bundle under toolbox-logs/ssh/, per the plugin README and changelog on main. Earlier versions still need the files requested separately. Commit 2653d37.
  • 2026-10-22: applied Nick's 2026-10-21 review (checked against main by a Coder Agents run). CODER_LOGGING_JSON takes a file path, with a note that human-readable logs also default to /dev/stderr. Only 5xx request failures log at warn; 4xx need debug logging (field notes and correlate step 2). ConnectionID is per SSH session and matches the agent log id. New ssh session failed row; SSH sessions don't record network_error (only the web terminal does), so the guide points to ssh session failed. unknown appears only in the agent log; the connection log leaves DisconnectReason empty. JetBrains Toolbox note: no ssh session closed or per-session connection log entries, one connection for the forwarded channel, normal close reason, debug-only watcher lines. disconnect_expected is agent-log only. Also closed the older open threads: coder-agent*.log glob plus rotation (5 MB, 19 backups), autodelete build reason, VS Code minimums (1.16.4 for the log buffer, 1.17.1 for editor names, since 1.17.0 has a "Premature close" regression), and Toolbox lines carrying the full 32-character ID (CoderLogger.withSessionId on plugin main). Commit 6e3b712.
  • 2026-10-27: docs(docs): document Toolbox support bundle requirements #31289 merged on 2026-10-26 with the Toolbox support bundle requirements. "Before you begin" now says plugin 0.9.5 requires JetBrains Toolbox 3.9.0 or later. The JetBrains Toolbox section now says only 0.9.5 or later adds the nested coder-support.zip (with coder ssh logs under toolbox-logs/ssh/), earlier plugin versions collect only Toolbox logs, and points to the CLI bundle and the new Toolbox section in the support bundle guide. The support-bundle.md#jetbrains-toolbox anchor exists on main but not on this branch yet, so it resolves after a main merge or on merge. Commit e5a37d4.
  • 2026-10-27 (05:00 UTC): description only, no file changes. @code-asher said to leave the connection path out of 2.39, so the feat: log and record DERP/direct connection path per agent connection #30665 row and its open question no longer expect a replacement. Added feat(logging): prefix output lines with the full client_session_id vscode-coder#1164 (full session ID prefix in the VS Code output channel, approved, unmerged). On @code-asher's review note that Go logs show ClientSessionID=: on main, coderd, the agent, and the CLI log the field as client_session_id (coderd/tracing/httpmw.go, agent/agentssh/agentssh.go, cli/root.go), and ClientSessionID appears only on connection_log entries, which is what the guide already says.
  • 2026-10-27 (05:20 UTC): the VS Code prerequisite is now Coder Remote 1.17.2 or later (feat(logging): prefix output lines with the full client_session_id vscode-coder#1164, shipping via chore: bump cronstrue from 2.2.0 to 2.4.0 in /site #1165), and the VS Code section describes the [client_session_id=<session-id>] prefix with the full 32-character ID. Earlier versions show only the first 8 characters, and versions before 1.17.1 don't report editor names. Dropped the "Starting with version 1.16.4" qualifier on the log buffer, since 1.17.2 is now the minimum. "Read disconnect reasons" now says that when the agent is shutting down, SSH sessions report server_shutdown and the agent log keeps the original message in disconnect_detail (agent/agentssh/agentssh.go on main). Replied to every open review thread, including Nick's mixed-agent question: agents only update when a workspace rebuilds, so a new server routinely serves earlier agents, and the prerequisite lists what earlier agents don't record. Commits c98046a, caf1bb7.

Open questions

CLI and agent defaults (CODER_FLIGHT_RECORDER_SIZE 10, CODER_AGENT_FLIGHT_RECORDER_SIZE 1000, 50 retained coder ssh logs, XDG state directory) come from cli/flightrecorder.go, cli/root.go, and cli/agent.go on main. The commands in the page weren't run against a live 2.39 deployment.

This description is refreshed daily at 05:00 UTC (10:00 PKT) from the Linear project, the dependency PRs, and #proj-connection-troubleshooting.

@linear-code

linear-code Bot commented Oct 13, 2026

Copy link
Copy Markdown

CLIENT-1032

@coder coder Bot assigned matifali Oct 13, 2026
@github-actions github-actions Bot added the community Pull Requests and issues created by the community. label Oct 13, 2026
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Docs preview

Check off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here.

@matifali matifali removed the community Pull Requests and issues created by the community. label Oct 14, 2026
For details, refer to [Generate a support bundle](../../support/support-bundle.md#vs-code).

The extension starts `coder ssh` with debug logging and writes its logs to a `log` directory in the extension's global storage, unless `coder.proxyLogDirectory` is set.
In the **Coder** output channel, each line starts with `[session` and the first 8 characters of the session ID.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In Toolbox the lines contain "client_session_id". They don't really start with that, that's the nature of the how loggers works with TBX but I wonder, shouldn't we use the same consistent discriminator across our clients? Here's a sample log line from TBX:

3.8.1.88030 41702 2026-10-14 22:25:21.930 INFO  Dispatcher-worker-28 CoderRemoteEnvironment         Plugin com.coder.toolbox > client_session_id=bcca9d4ff38bcbab0120e20fa13cbab4 Toolbox is disconnecting SSH from bobiverse-bob.dev without an explicit user disconnect. Latest known Coder state: environment=Ready, workspace=RUNNING, agent=CONNECTED, agentLifecycle=READY

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah the cli, coderd, and agent behave like Toolbox as well so it would make sense to update the VS Code extension to match.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc: @EhabY

@EhabY EhabY Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know this makes sense for uniformity but the logs here are shown in the output channel and should be more readable than logs in a txt file.

image

Like if we add this then each line prefix would be huge and hard to read.

Note, the timestamp and the level is printed by default (not us), so we can only control [session XYZ] and after

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see the issue now, we want the full ID to correlate... I wonder if we can save it some other way

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant if we can save it besides just on every log message, like we could correlate using the first 8 chars but that's finicky so yeah we have to put the whole thing I guess 😞

Do we use the same semantics: [client_session_id=<32 HEX VALUE>]?

@matifali matifali Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@code-asher do you know?

@EhabY EhabY Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose this depends on what we want, do we want isolated logs to be correlated? Then yes we need to put the client session ID fully. If we want it only correlated on support bundles then can maintain some mapping of short -> full ID like:

{
  "client_session_id": "bcca9d4ff38bcbab0120e20fa13cbab4",
  "display_session_id": "bcca9d4f"
}

I'm guessing it's the former so we def need to put it fully but now do we add it as a prefix or suffix? Do we need to do [client_session_id=<32 HEX VALUE>] or can we just put it as is without the variable name?

@code-asher code-asher Oct 27, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah actually in our Go projects it shows up as ClientSessionID=<32 hex value> (which shows up after the main text, so not a prefix, something like [info] a thing happened ClientSessionID=1234...

I think just the ID is sufficient, otherwise we would already run into trouble with client_session_id vs ClientSessionID, and the IDs should be unique enough. Although it could make sense to label it anyway, in case logs ever gain other IDs.

edit: oh that is just the connection log yeah we do use the underscore in logs

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in c98046a. The guide now requires Coder Remote 1.17.2, where coder/vscode-coder#1164 prefixes output channel lines with [client_session_id=<id>] and the full 32-character ID, matching Toolbox, the CLI, and the agent.

@code-asher on ClientSessionID=: on main, coderd (coderd/tracing/httpmw.go), the agent (agent/agentssh, agent/reconnectingpty), and the CLI (cli/root.go) all log the key as client_session_id. ClientSessionID only appears on connection_log service log entries, because those export the stored record's Go field names. The guide uses each name where it appears.

Toolbox starts `coder ssh`, which writes its logs to the default directory for `coder ssh` unless an SSH log directory is set in the Coder plugin settings.
For the Toolbox logs, ask the user to select **Collect logs** from the workspace action menu.
For details, refer to [JetBrains Toolbox troubleshooting](../../user-guides/workspace-access/jetbrains/toolbox.md#troubleshooting).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could also detail that logs related to an ssh session will include the client_session_id= and the full 32 chars

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Covered: the Toolbox section says session log lines include the full 32-character client_session_id, and as of c98046a the VS Code section does too (1.17.2+).

For details, refer to [Generate a support bundle](../../support/support-bundle.md#vs-code).

The extension starts `coder ssh` with debug logging and writes its logs to a `log` directory in the extension's global storage, unless `coder.proxyLogDirectory` is set.
In the **Coder** output channel, each line starts with `[session` and the first 8 characters of the session ID.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

again different from TBX where we include the full 32 chars.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc: @EhabY

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c98046a: VS Code 1.17.2+ logs the full 32-character ID, same as Toolbox.

For details, refer to [Generate a support bundle](../../support/support-bundle.md#vs-code).

The extension starts `coder ssh` with debug logging and writes its logs to a `log` directory in the extension's global storage, unless `coder.proxyLogDirectory` is set.
In the **Coder** output channel, each line starts with `[session` and the first 8 characters of the session ID.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see the issue now, we want the full ID to correlate... I wonder if we can save it some other way

| `accepted conn`, `reconnecting pty closed` | A web terminal connection opened or closed. The closing entry includes `disconnect_reason`. |
| `disconnected from coderd` | The agent lost its connection to `coderd`. The buffered debug entries are written just before this line. |

## Correlate a session

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR actually adds more info like the duration of a session (when we can know it) and it adds a way in the UI to correlate from a single connection log: #30759

Image

This might simplify a lot from the entry itself

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. #30759 is still a draft, so the guide doesn't cover it yet. Once it merges, the duration and the session ID filter can replace part of the service-log jq step.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction to my reply above: #30759 isn't a draft. It's been ready for review since 2026-10-21 and @code-asher approved it on 2026-10-26 (head 7f5bf0991). At that head, the API adds client_session_id to ssh_info and a client_session_id: filter, and the UI derives duration from connect and disconnect times (ssh_info.duration isn't in codersdk anymore). This guide doesn't cover it yet. If #30759 merges before the 2.39 freeze, I'll add the filter and the View related connections link, which can replace part of the service-log jq step.

@nickvigilante nickvigilante left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a mixture of my own review and a Coder Agents review.

## Before you begin

- Run Coder server and workspace agents on version 2.39 or later.
Older agents keep working, but they don't record client session IDs, app names, or workspace shutdown reasons.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Older agents keep working, but they don't record client session IDs, app names, or workspace shutdown reasons.
Agents from earlier versions keep working, but they don't record client session IDs, app names, or workspace shutdown reasons.

- Run Coder server and workspace agents on version 2.39 or later.
Older agents keep working, but they don't record client session IDs, app names, or workspace shutdown reasons.
- Collect `coderd` logs with a log management tool such as Splunk, Datadog, or Grafana Loki.
JSON output is easiest to query; enable it with [`CODER_LOGGING_JSON`](../../reference/cli/server/index.md#--log-json).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
JSON output is easiest to query; enable it with [`CODER_LOGGING_JSON`](../../reference/cli/server/index.md#--log-json).
JSON output is easiest to query.
Turn on JSON output with [`CODER_LOGGING_JSON`](../../reference/cli/server/index.md#--log-json).

Correct the agent log location (system temporary directory, CODER_AGENT_LOG_DIR), document the VS Code coder.proxyLogDirectory fallback, and apply wording suggestions.

@nickvigilante nickvigilante left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My robot found a few things worth addressing. I'll review again after that, but I think this LGTM pending one Eng review.

|--------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `handling ssh session` | An SSH session started. |
| `ssh session closed` | An SSH session ended. Coder includes `disconnect_reason`, `disconnect_expected`, and `exit_code`. |
| `ssh connection complete` | An SSH connection closed. When a workspace stop or delete closed it, Coder also includes `disconnect_reason`, `build_id`, `build_reason`, and `transition`. |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't match the current head of #30515 (52928429bd).
On a workspace stop or delete, ssh connection complete only gets serverShutdownFields(): disconnect_reason=server_shutdown, disconnect_expected=true, and disconnect_initiator=agent (agent/agentssh/agentssh.go:1206-1211, agent/agentssh/shutdown.go:9-15).
No agent log line carries build_id, build_reason, or transition.
Those three fields are only in the coder ssh client log, on a workspace is shutting down line (cli/ssh.go:1022-1031).

Outside shutdown, this line has no disconnect_reason at all.

Suggested change
| `ssh connection complete` | An SSH connection closed. When a workspace stop or delete closed it, Coder also includes `disconnect_reason`, `build_id`, `build_reason`, and `transition`. |
| `ssh connection complete` | An SSH connection closed. If the agent was shutting down, Coder also includes `disconnect_reason`, `disconnect_expected`, and `disconnect_initiator`. |

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 55c8c16 and rechecked against main now that #30515 merged: ssh connection complete lists disconnect_reason, disconnect_expected, disconnect_initiator only during shutdown (serverShutdownFields()), and the build fields point to the coder ssh log.


## Read disconnect reasons

When an SSH session, SSH connection, or web terminal connection closes, the agent logs `disconnect_reason` and `disconnect_expected`, and reports the same reason to the connection log as `DisconnectReason`.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With #30515, the agent log and the connection log can disagree.
The agent logs server_shutdown, and coderd rewrites it to workspace_stopped before writing the connection log entry, but only when the latest build is a running or succeeded stop or delete requested before the connection closed (coderd/agentapi/connectionlog.go:94-96, 149-189).
SSH connections (as opposed to sessions) also aren't reported to the connection log.

Suggested change
When an SSH session, SSH connection, or web terminal connection closes, the agent logs `disconnect_reason` and `disconnect_expected`, and reports the same reason to the connection log as `DisconnectReason`.
When an SSH session or web terminal connection closes, the agent logs `disconnect_reason` and `disconnect_expected` and reports the reason to the connection log as `DisconnectReason`.
The two can differ after a workspace stop or delete: the agent logs `server_shutdown`, and `coderd` records `workspace_stopped` in the connection log.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 55c8c16: SSH connections aren't counted as reported, and the section explains the agent server_shutdown vs connection log workspace_stopped difference and when coderd rewrites it.

Comment on lines +196 to +197
| `server_shutdown` | The agent closed the connection because the agent was shutting down. |
| `workspace_stopped` | A workspace stop or delete closed the connection. `build_reason` shows what started the build. |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

workspace_stopped only appears in the connection log (set by coderd) and in the coder ssh client log.
build_reason isn't in agent logs, coderd logs, or the connection log; the only place it's logged is the coder ssh client log (cli/ssh.go:1029).
Web terminal sessions have no equivalent.

The #30515 description says coderd "logs the build reason, transition, and build ID", but I couldn't find that log call in its diff.
Worth confirming with the #30515 author which is intended, and re-checking this section right before merge since that PR is still moving.

The new descriptions are longer than the existing column, so the table may need reformatting after you apply this.

Suggested change
| `server_shutdown` | The agent closed the connection because the agent was shutting down. |
| `workspace_stopped` | A workspace stop or delete closed the connection. `build_reason` shows what started the build. |
| `server_shutdown` | The agent closed the connection because the agent was shutting down. In the connection log, `coderd` can replace it with `workspace_stopped`. |
| `workspace_stopped` | Connection log only. A workspace stop or delete closed the connection. The `coder ssh` client log records `build_reason`. |

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 55c8c16: build_reason now points only to the coder ssh log's workspace is shutting down line, confirmed on main in cli/ssh.go after #30515 merged.

| `workspace_stopped` | A workspace stop or delete closed the connection. `build_reason` shows what started the build. |
| `unknown` | The agent couldn't determine why the connection closed. |

For SSH sessions, the reason can also be a specific message, such as `process exited with error status: 1` when the command exited with a nonzero status.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the free-form reasons also include file transfer blocked, unsupported subsystem: <name>, x11 handler failed, and raw SFTP or session error strings (agent/agentssh/agentssh.go:603-688).
If the agent is shutting down when the session ends, the reason becomes server_shutdown and the original text moves to disconnect_detail (agentssh.go:581-586).

Suggested change
For SSH sessions, the reason can also be a specific message, such as `process exited with error status: 1` when the command exited with a nonzero status.
For SSH sessions, the reason can also be a specific message, such as `process exited with error status: 1` when the command exited with a nonzero status, or `file transfer blocked`.
If the agent was shutting down, the reason is `server_shutdown`, and the specific message is in `disconnect_detail`.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. The raw error text case was added earlier, and c98046a adds the shutdown case: the reason becomes server_shutdown and the specific message moves to disconnect_detail (agent/agentssh/agentssh.go, ssh session closed defer).

For SSH sessions, the reason can also be a specific message, such as `process exited with error status: 1` when the command exited with a nonzero status.

`disconnect_expected` is `true` for `graceful`, `server_shutdown`, and `workspace_stopped`, and `false` for every other reason.
Filter on `disconnect_expected=false` to find the disconnects worth investigating.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

disconnect_expected is only in agent and coder ssh logs; connection_log entries don't have it.
A reader following the earlier jq example would filter connection_log entries and get nothing.

Suggested change
Filter on `disconnect_expected=false` to find the disconnects worth investigating.
Filter agent logs on `disconnect_expected=false` to find the disconnects worth investigating.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 6e3b712: the filter is scoped to agent logs, with a note that connection_log entries don't include disconnect_expected.

Comment on lines +205 to +211
When `disconnect_reason` is `workspace_stopped`, `build_reason` shows what stopped the workspace:

| `build_reason` | The workspace stopped because |
|----------------|-----------------------------------------------|
| `initiator` | A user or API client requested the build. |
| `autostop` | The workspace reached its autostop schedule. |
| `dormancy` | The workspace became dormant from inactivity. |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stop and delete builds can also have failedstop and autodelete (coderd/database/models.go:1410-1411, set in coderd/autobuild/lifecycle_executor.go:673).
codersdk.BuildReason doesn't define constants for them, but the raw value passes through.
Dashboard and CLI stops use initiator; dashboard and cli are only sent for starts (site/src/api/api.ts:1409).

This also points the reader at the coder ssh log, per the comment on line 197.

Suggested change
When `disconnect_reason` is `workspace_stopped`, `build_reason` shows what stopped the workspace:
| `build_reason` | The workspace stopped because |
|----------------|-----------------------------------------------|
| `initiator` | A user or API client requested the build. |
| `autostop` | The workspace reached its autostop schedule. |
| `dormancy` | The workspace became dormant from inactivity. |
When the `coder ssh` client log shows `disconnect_reason=workspace_stopped`, `build_reason` shows what stopped the workspace:
| `build_reason` | The workspace stopped because |
|----------------|----------------------------------------------------------------------------------|
| `initiator` | A user or API client requested the build. |
| `autostop` | The workspace reached its autostop schedule. |
| `dormancy` | The workspace became dormant from inactivity. |
| `failedstop` | The workspace's last build failed and the template's failure cleanup stopped it. |
| `autodelete` | The workspace was dormant long enough for the template to delete it. |

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 6e3b712: added autodelete; left out failedstop since nothing sets it.

1. Search the agent log for the session ID:

```sh
grep '<session-id>' <agent-log-dir>/coder-agent.log

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The agent log rotates.
lumberjack rolls coder-agent.log at 5 MB and keeps 19 backups named coder-agent-<timestamp>.log (cli/agent.go:167-171), so this grep can miss the earlier part of a session.
Consider also adding a sentence near line 129 that older entries are in rotated coder-agent-*.log files.

Suggested change
grep '<session-id>' <agent-log-dir>/coder-agent.log
grep '<session-id>' <agent-log-dir>/coder-agent*.log

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 6e3b712: the grep uses coder-agent*.log and the Agent logs section describes rotation at 5 MB with 19 older files.

It writes the `coder ssh` logs to the directory in the `coder.proxyLogDirectory` VS Code setting.
If the setting is empty, the extension uses `CODER_SSH_LOG_DIR`, and if that isn't set either, a `log` directory in the extension's global storage.
In the **Coder** output channel, each line starts with `[session` and the first 8 characters of the session ID.
The extension keeps the 1,000 most recent entries below the channel's log level and writes them to the channel when a connection fails or the user collects a support bundle.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: coder.connectionLogBuffer.size and this buffer shipped in v1.16.4 (vscode-coder CHANGELOG), but the prerequisite on line 19 says 1.16.1 or later.
Either call out 1.16.4 here or raise the minimum in the prerequisite.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded in c98046a: the prerequisite is now Coder Remote 1.17.2 or later, which covers the log buffer, so the 1.16.4 call-out is gone.

Turn on JSON output with [`CODER_LOGGING_JSON`](../../reference/cli/server/index.md#--log-json).
- The `connection_log` entries in this guide require a [Premium license](https://coder.com/pricing#compare-plans).
Client and agent logs don't.
- For VS Code, use the Coder Remote extension version 1.16.1 or later to send session IDs, and version 1.17.0 or later to report the editor name, such as `cursor`.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: v1.17.0 has a "Premature close" regression in Positron, VSCodium, and other editors that v1.17.1 fixed (vscode-coder CHANGELOG).
Pointing at 1.17.1 avoids sending readers to a broken build.

Suggested change
- For VS Code, use the Coder Remote extension version 1.16.1 or later to send session IDs, and version 1.17.0 or later to report the editor name, such as `cursor`.
- For VS Code, use the Coder Remote extension version 1.16.1 or later to send session IDs, and version 1.17.1 or later to report the editor name, such as `cursor`.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded in c98046a: the minimum is now 1.17.2, which is past the 1.17.0 regression.


One client session often spans several connections.
For example, an IDE that reconnects after a network interruption opens a new SSH connection each time.
Each connection gets its own connection log entry, with its own `ConnectionID` and the same `ClientSessionID`.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this reads as one entry per connection, but line 107 says each SSH or web terminal connection produces a connected and a disconnected service log entry.

Suggested change
Each connection gets its own connection log entry, with its own `ConnectionID` and the same `ClientSessionID`.
Each connection has its own `ConnectionID`, shared by its `connected` and `disconnected` entries, and the same `ClientSessionID` as the other connections in the session.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 6e3b712 with your wording (one ConnectionID shared by the connected and disconnected entries).

@matifali matifali added the experimental Changes that might not necessarily be merged, until its approved to proceed with. label Oct 16, 2026
default and others added 3 commits October 17, 2026 00:06
… connections guide

The agent logs server_shutdown with disconnect_initiator on ssh connection complete, and only the coder ssh log carries build_id, build_reason, and transition. coderd rewrites server_shutdown to workspace_stopped in the connection log when a stop or delete build caused it.
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@coder
coder Bot force-pushed the superblink/client-1032-trace-connections branch from 4b31769 to 2653d37 Compare October 21, 2026 05:41

@nickvigilante nickvigilante left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of the Trace workspace connections guide

I checked the page against origin/main (45e392b8be) at head 2653d37.
Most claims hold; the inline comments cover four important issues and four nits.

Important

  • JetBrains Toolbox sessions don't produce ssh session closed lines, and their normal close reason is normal close (line 148).
  • SSH sessions never record network_error, so the filter misses SSH transport failures (line 204).
  • CODER_LOGGING_JSON takes a file path, not an on/off value (line 16).
  • Only 5xx failures are logged at warn; 4xx failures need debug logging (lines 124-125 and 171).

Still open from earlier threads

These threads are unresolved and the current text still has the issue:

  • Line 176: the grep only reads coder-agent.log, but the agent rotates at 5 MB and keeps 19 backups (cli/agent.go:167-171); use coder-agent*.log.
  • Lines 206-207: connection_log entries don't have disconnect_expected.
  • Lines 213-217: the build_reason table is missing autodelete (coderd/autobuild/lifecycle_executor.go:673); failedstop is in the enum but nothing sets it.
  • Lines 19 and 84: version minimums (1.16.4 for the log buffer, 1.17.1 instead of 1.17.0).
  • Line 45: one connection produces a connected and a disconnected entry.
  • fioan89's Toolbox threads about the full 32-character ID.

Verified against code

Session ID format and CODER_TRACE_SESSION_ID, coder ssh log directories, file names, retention, and flight recorder size, agent log location and flight recorder behavior, agent log messages, the server_shutdown to workspace_stopped rewrite, the workspace is shutting down fields, web terminal console errors, the jq example, the method:tunnel filter, and doc link anchors.
I didn't verify VS Code or Toolbox extension behavior, since that code lives in other repositories.


This review was generated by Coder Agents on behalf of @nickvigilante.


| Message | What it records |
|--------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------|
| `handling ssh session` | An SSH session started. |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important: JetBrains Toolbox sessions don't follow the pattern this table describes.

  • The agent sets reportSession = false for the JetBrains app family (agent/agentssh/agentssh.go:553-558), so there's no ssh session closed line and no per-session connection log entry for those sessions.
  • Toolbox connection log entries come from JetbrainsChannelWatcher, which tracks the single persistent forwarded channel (agent/agentssh/jetbrainstrack.go:92-130).
  • On a normal close, that path reports DisconnectReason as normal close, which isn't in the reasons table.
  • The agent logs JetBrains watcher accepted channel and JetBrains channel closed at debug level, so they aren't in the agent log by default.
  • handling ssh session still fires for each of the many sessions JetBrains opens, so this search is noisy for Toolbox users.

Since the guide lists Toolbox as a supported client, could we add a short JetBrains note here and in "Read disconnect reasons"?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 6e3b712: the Agent logs section now notes that Toolbox sessions are frequent handling ssh session lines with no ssh session closed or per-session connection log entry, that the forwarded channel is reported as one connection with JetBrains watcher accepted channel / JetBrains channel closed at debug level, and normal close is in the reasons table.

| `workspace_stopped` | A workspace stop or delete closed the connection. Only the connection log and the `coder ssh` log record this reason. |
| `unknown` | The agent couldn't determine why the connection closed. |

For SSH sessions, the reason can also be a specific message, such as `process exited with error status: 1` when the command exited with a nonzero status.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important: SSH sessions never record network_error (line 199).
Only the web terminal uses it (agent/reconnectingpty/server.go:139).
When an SSH session fails, the agent records the raw error text as the reason (closeCause(err.Error()), agent/agentssh/agentssh.go:633,689), so filtering on network_error misses every SSH transport failure.

Suggested change
For SSH sessions, the reason can also be a specific message, such as `process exited with error status: 1` when the command exited with a nonzero status.
For SSH sessions, the reason can also be a specific message, such as `process exited with error status: 1` when the command exited with a nonzero status, or the raw error text when the session failed.
SSH sessions don't record `network_error`, so search the agent log for `ssh session failed` to find SSH transport failures.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 6e3b712: network_error is scoped to the web terminal, and the text points to ssh session failed for SSH transport failures.

Agents from earlier versions keep working, but they don't record client session IDs, app names, workspace shutdown reasons, or bytes transferred.
- Collect `coderd` logs with a log management tool such as Splunk, Datadog, or Grafana Loki.
JSON output is easiest to query.
Turn on JSON output with [`CODER_LOGGING_JSON`](../../reference/cli/server/index.md#--log-json).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important: --log-json / CODER_LOGGING_JSON is a string flag that takes a file path ("Output JSON logs to a given file", docs/reference/cli/server/index.md:881-889), not an on/off switch.
"Turn on JSON output with" suggests setting it to true, which writes the log to a file named true.

Suggested change
Turn on JSON output with [`CODER_LOGGING_JSON`](../../reference/cli/server/index.md#--log-json).
To write JSON logs, set [`CODER_LOGGING_JSON`](../../reference/cli/server/index.md#--log-json) to a file path, such as `/dev/stderr`.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 6e3b712: it now says to set CODER_LOGGING_JSON to a file path such as /dev/stderr, and to point CODER_LOGGING_HUMAN at /dev/null to keep only JSON there.

Comment on lines +124 to +125
Failed requests with a `5xx` status are logged at the warn level.
Successful requests are logged at the debug level, so they appear only when debug logging is on.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important: coderd/httpmw/loggermw/logger.go:182-185 logs 5xx at warn and every other status at debug, including 401, 403, and 404.
"Successful requests" undersells what's hidden, and step 2 of "Correlate a session" (line 171) misses the most common client failures unless debug logging is on.

Suggested change
Failed requests with a `5xx` status are logged at the warn level.
Successful requests are logged at the debug level, so they appear only when debug logging is on.
Requests that fail with a `5xx` status are logged at the warn level.
All other requests, including `4xx` failures such as `401`, `403`, and `404`, are logged at the debug level, so they appear only when debug logging is on.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 6e3b712 with your wording: only 5xx is warn; 4xx and successes are debug.

Note the time of each `connected` and `disconnected` entry and its `DisconnectReason`.

1. Search the `coderd` logs for `client_session_id` set to the same value.
Matching warn-level entries show failed requests from the client.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up to the 5xx comment above: warn-level entries only cover 5xx failures.

Suggested change
Matching warn-level entries show failed requests from the client.
Matching warn-level entries show requests that failed with a `5xx` status.
To see `4xx` failures, turn on debug logging.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 6e3b712: step 2 now says warn entries cover 5xx only and to turn on debug logging for 4xx.

| Field | Description |
|----------------------|-------------------------------------------------------------------------------------------------------------------------------------------|
| `ClientSessionID` | The client session ID. Empty for workspace app, port forwarding, and tunnel events, and for connections from older clients or agents. |
| `ConnectionID` | Identifies one SSH or web terminal connection. |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: ConnectionID is per SSH session, not per SSH connection.
sessionHandler creates it (agent/agentssh/agentssh.go:490), so one SSH connection that carries several sessions produces several entries.
The same UUID is the id field on the agent's handling ssh session and ssh session closed lines, which gives readers a direct join between the agent log and the connection log.
Line 45 has the same wording.

Suggested change
| `ConnectionID` | Identifies one SSH or web terminal connection. |
| `ConnectionID` | Identifies one SSH session or web terminal connection. For SSH, matches the `id` field in the agent log. |

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 6e3b712: ConnectionID is per SSH session or web terminal connection and matches the agent's id field; the intro paragraph uses the same wording.

| Message | What it records |
|--------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------|
| `handling ssh session` | An SSH session started. |
| `ssh session closed` | An SSH session ended. Coder includes `disconnect_reason`, `disconnect_expected`, and `exit_code`. |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: ssh session failed (warn level, agent/agentssh/agentssh.go:686) is the line logged right before an error-text reason, so it's worth listing.

Suggested change
| `ssh session closed` | An SSH session ended. Coder includes `disconnect_reason`, `disconnect_expected`, and `exit_code`. |
| `ssh session closed` | An SSH session ended. Coder includes `disconnect_reason`, `disconnect_expected`, and `exit_code`. |
| `ssh session failed` | An SSH session failed with an error. The error text becomes the `disconnect_reason`. |

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added ssh session failed to the agent log table in 6e3b712.

| `network_error` | The transport between the client and the agent failed, for example with an EOF, a read or write error, or a timeout. |
| `server_shutdown` | The agent closed the connection because the agent was shutting down. |
| `workspace_stopped` | A workspace stop or delete closed the connection. Only the connection log and the `coder ssh` log record this reason. |
| `unknown` | The agent couldn't determine why the connection closed. |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: unknown only appears in the agent log.
codersdk.DisconnectReason.SlogField() renders the empty reason as unknown (codersdk/disconnect.go:24-26), but the agent reports "" to coderd, which stores it as NULL, so DisconnectReason in the connection log is empty instead.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 6e3b712: the unknown row says only the agent log shows it and DisconnectReason is empty in the connection log.


For SSH sessions, the reason can also be a specific message, such as `process exited with error status: 1` when the command exited with a nonzero status.

`disconnect_expected` is `true` for `graceful`, `server_shutdown`, and `workspace_stopped`, and `false` for every other reason.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Expected() also returns true for client_closed (codersdk/disconnect.go:104-110).
Nothing emits it today, so this is only a risk if a client starts using it, but "false for every other reason" won't hold if one does.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reworded in 6e3b712 to list the false cases explicitly (network_error, unknown, error messages) instead of "every other reason", so it stays true if a client starts emitting client_closed.

Plugin 0.9.5 requires JetBrains Toolbox 3.9.0, and earlier plugin versions collect only Toolbox logs without a Coder support bundle. Link the Toolbox support bundle section.
@matifali matifali closed this Oct 27, 2026
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 27, 2026
@matifali

Copy link
Copy Markdown
Contributor

Replaced by #31342 under @matifali, preserving the existing branch and commits. The replacement remains a draft with outstanding documentation review findings.

🤖 This response was generated by Coder Agents.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

experimental Changes that might not necessarily be merged, until its approved to proceed with.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants