Repository navigation
docs: add guide for tracing workspace connections - #30707
coder[bot] wants to merge 13 commits into
Conversation
Docs previewCheck off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here. |
| For details, refer to [Generate a support bundle](../../support/support-bundle.md#vs-code). | ||
|
|
||
| The extension starts `coder ssh` with debug logging and writes its logs to a `log` directory in the extension's global storage, unless `coder.proxyLogDirectory` is set. | ||
| In the **Coder** output channel, each line starts with `[session` and the first 8 characters of the session ID. |
There was a problem hiding this comment.
In Toolbox the lines contain "client_session_id". They don't really start with that, that's the nature of the how loggers works with TBX but I wonder, shouldn't we use the same consistent discriminator across our clients? Here's a sample log line from TBX:
3.8.1.88030 41702 2026-10-14 22:25:21.930 INFO Dispatcher-worker-28 CoderRemoteEnvironment Plugin com.coder.toolbox > client_session_id=bcca9d4ff38bcbab0120e20fa13cbab4 Toolbox is disconnecting SSH from bobiverse-bob.dev without an explicit user disconnect. Latest known Coder state: environment=Ready, workspace=RUNNING, agent=CONNECTED, agentLifecycle=READY
There was a problem hiding this comment.
Yeah the cli, coderd, and agent behave like Toolbox as well so it would make sense to update the VS Code extension to match.
There was a problem hiding this comment.
I know this makes sense for uniformity but the logs here are shown in the output channel and should be more readable than logs in a txt file.
Like if we add this then each line prefix would be huge and hard to read.
Note, the timestamp and the level is printed by default (not us), so we can only control [session XYZ] and after
There was a problem hiding this comment.
I see the issue now, we want the full ID to correlate... I wonder if we can save it some other way
There was a problem hiding this comment.
I meant if we can save it besides just on every log message, like we could correlate using the first 8 chars but that's finicky so yeah we have to put the whole thing I guess 😞
Do we use the same semantics: [client_session_id=<32 HEX VALUE>]?
There was a problem hiding this comment.
I suppose this depends on what we want, do we want isolated logs to be correlated? Then yes we need to put the client session ID fully. If we want it only correlated on support bundles then can maintain some mapping of short -> full ID like:
{
"client_session_id": "bcca9d4ff38bcbab0120e20fa13cbab4",
"display_session_id": "bcca9d4f"
}I'm guessing it's the former so we def need to put it fully but now do we add it as a prefix or suffix? Do we need to do [client_session_id=<32 HEX VALUE>] or can we just put it as is without the variable name?
There was a problem hiding this comment.
Ah actually in our Go projects it shows up as ClientSessionID=<32 hex value> (which shows up after the main text, so not a prefix, something like [info] a thing happened ClientSessionID=1234...
I think just the ID is sufficient, otherwise we would already run into trouble with client_session_id vs ClientSessionID, and the IDs should be unique enough. Although it could make sense to label it anyway, in case logs ever gain other IDs.
edit: oh that is just the connection log yeah we do use the underscore in logs
There was a problem hiding this comment.
Updated in c98046a. The guide now requires Coder Remote 1.17.2, where coder/vscode-coder#1164 prefixes output channel lines with [client_session_id=<id>] and the full 32-character ID, matching Toolbox, the CLI, and the agent.
@code-asher on ClientSessionID=: on main, coderd (coderd/tracing/httpmw.go), the agent (agent/agentssh, agent/reconnectingpty), and the CLI (cli/root.go) all log the key as client_session_id. ClientSessionID only appears on connection_log service log entries, because those export the stored record's Go field names. The guide uses each name where it appears.
| Toolbox starts `coder ssh`, which writes its logs to the default directory for `coder ssh` unless an SSH log directory is set in the Coder plugin settings. | ||
| For the Toolbox logs, ask the user to select **Collect logs** from the workspace action menu. | ||
| For details, refer to [JetBrains Toolbox troubleshooting](../../user-guides/workspace-access/jetbrains/toolbox.md#troubleshooting). | ||
|
|
There was a problem hiding this comment.
we could also detail that logs related to an ssh session will include the client_session_id= and the full 32 chars
There was a problem hiding this comment.
Covered: the Toolbox section says session log lines include the full 32-character client_session_id, and as of c98046a the VS Code section does too (1.17.2+).
| For details, refer to [Generate a support bundle](../../support/support-bundle.md#vs-code). | ||
|
|
||
| The extension starts `coder ssh` with debug logging and writes its logs to a `log` directory in the extension's global storage, unless `coder.proxyLogDirectory` is set. | ||
| In the **Coder** output channel, each line starts with `[session` and the first 8 characters of the session ID. |
There was a problem hiding this comment.
again different from TBX where we include the full 32 chars.
There was a problem hiding this comment.
Fixed in c98046a: VS Code 1.17.2+ logs the full 32-character ID, same as Toolbox.
| For details, refer to [Generate a support bundle](../../support/support-bundle.md#vs-code). | ||
|
|
||
| The extension starts `coder ssh` with debug logging and writes its logs to a `log` directory in the extension's global storage, unless `coder.proxyLogDirectory` is set. | ||
| In the **Coder** output channel, each line starts with `[session` and the first 8 characters of the session ID. |
There was a problem hiding this comment.
I see the issue now, we want the full ID to correlate... I wonder if we can save it some other way
| | `accepted conn`, `reconnecting pty closed` | A web terminal connection opened or closed. The closing entry includes `disconnect_reason`. | | ||
| | `disconnected from coderd` | The agent lost its connection to `coderd`. The buffered debug entries are written just before this line. | | ||
|
|
||
| ## Correlate a session |
There was a problem hiding this comment.
This PR actually adds more info like the duration of a session (when we can know it) and it adds a way in the UI to correlate from a single connection log: #30759
This might simplify a lot from the entry itself
There was a problem hiding this comment.
Thanks. #30759 is still a draft, so the guide doesn't cover it yet. Once it merges, the duration and the session ID filter can replace part of the service-log jq step.
There was a problem hiding this comment.
Correction to my reply above: #30759 isn't a draft. It's been ready for review since 2026-10-21 and @code-asher approved it on 2026-10-26 (head 7f5bf0991). At that head, the API adds client_session_id to ssh_info and a client_session_id: filter, and the UI derives duration from connect and disconnect times (ssh_info.duration isn't in codersdk anymore). This guide doesn't cover it yet. If #30759 merges before the 2.39 freeze, I'll add the filter and the View related connections link, which can replace part of the service-log jq step.
nickvigilante
left a comment
There was a problem hiding this comment.
This is a mixture of my own review and a Coder Agents review.
| ## Before you begin | ||
|
|
||
| - Run Coder server and workspace agents on version 2.39 or later. | ||
| Older agents keep working, but they don't record client session IDs, app names, or workspace shutdown reasons. |
There was a problem hiding this comment.
| Older agents keep working, but they don't record client session IDs, app names, or workspace shutdown reasons. | |
| Agents from earlier versions keep working, but they don't record client session IDs, app names, or workspace shutdown reasons. |
| - Run Coder server and workspace agents on version 2.39 or later. | ||
| Older agents keep working, but they don't record client session IDs, app names, or workspace shutdown reasons. | ||
| - Collect `coderd` logs with a log management tool such as Splunk, Datadog, or Grafana Loki. | ||
| JSON output is easiest to query; enable it with [`CODER_LOGGING_JSON`](../../reference/cli/server/index.md#--log-json). |
There was a problem hiding this comment.
| JSON output is easiest to query; enable it with [`CODER_LOGGING_JSON`](../../reference/cli/server/index.md#--log-json). | |
| JSON output is easiest to query. | |
| Turn on JSON output with [`CODER_LOGGING_JSON`](../../reference/cli/server/index.md#--log-json). |
Correct the agent log location (system temporary directory, CODER_AGENT_LOG_DIR), document the VS Code coder.proxyLogDirectory fallback, and apply wording suggestions.
nickvigilante
left a comment
There was a problem hiding this comment.
My robot found a few things worth addressing. I'll review again after that, but I think this LGTM pending one Eng review.
| |--------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------| | ||
| | `handling ssh session` | An SSH session started. | | ||
| | `ssh session closed` | An SSH session ended. Coder includes `disconnect_reason`, `disconnect_expected`, and `exit_code`. | | ||
| | `ssh connection complete` | An SSH connection closed. When a workspace stop or delete closed it, Coder also includes `disconnect_reason`, `build_id`, `build_reason`, and `transition`. | |
There was a problem hiding this comment.
This doesn't match the current head of #30515 (52928429bd).
On a workspace stop or delete, ssh connection complete only gets serverShutdownFields(): disconnect_reason=server_shutdown, disconnect_expected=true, and disconnect_initiator=agent (agent/agentssh/agentssh.go:1206-1211, agent/agentssh/shutdown.go:9-15).
No agent log line carries build_id, build_reason, or transition.
Those three fields are only in the coder ssh client log, on a workspace is shutting down line (cli/ssh.go:1022-1031).
Outside shutdown, this line has no disconnect_reason at all.
| | `ssh connection complete` | An SSH connection closed. When a workspace stop or delete closed it, Coder also includes `disconnect_reason`, `build_id`, `build_reason`, and `transition`. | | |
| | `ssh connection complete` | An SSH connection closed. If the agent was shutting down, Coder also includes `disconnect_reason`, `disconnect_expected`, and `disconnect_initiator`. | |
|
|
||
| ## Read disconnect reasons | ||
|
|
||
| When an SSH session, SSH connection, or web terminal connection closes, the agent logs `disconnect_reason` and `disconnect_expected`, and reports the same reason to the connection log as `DisconnectReason`. |
There was a problem hiding this comment.
With #30515, the agent log and the connection log can disagree.
The agent logs server_shutdown, and coderd rewrites it to workspace_stopped before writing the connection log entry, but only when the latest build is a running or succeeded stop or delete requested before the connection closed (coderd/agentapi/connectionlog.go:94-96, 149-189).
SSH connections (as opposed to sessions) also aren't reported to the connection log.
| When an SSH session, SSH connection, or web terminal connection closes, the agent logs `disconnect_reason` and `disconnect_expected`, and reports the same reason to the connection log as `DisconnectReason`. | |
| When an SSH session or web terminal connection closes, the agent logs `disconnect_reason` and `disconnect_expected` and reports the reason to the connection log as `DisconnectReason`. | |
| The two can differ after a workspace stop or delete: the agent logs `server_shutdown`, and `coderd` records `workspace_stopped` in the connection log. |
There was a problem hiding this comment.
Addressed in 55c8c16: SSH connections aren't counted as reported, and the section explains the agent server_shutdown vs connection log workspace_stopped difference and when coderd rewrites it.
| | `server_shutdown` | The agent closed the connection because the agent was shutting down. | | ||
| | `workspace_stopped` | A workspace stop or delete closed the connection. `build_reason` shows what started the build. | |
There was a problem hiding this comment.
workspace_stopped only appears in the connection log (set by coderd) and in the coder ssh client log.
build_reason isn't in agent logs, coderd logs, or the connection log; the only place it's logged is the coder ssh client log (cli/ssh.go:1029).
Web terminal sessions have no equivalent.
The #30515 description says coderd "logs the build reason, transition, and build ID", but I couldn't find that log call in its diff.
Worth confirming with the #30515 author which is intended, and re-checking this section right before merge since that PR is still moving.
The new descriptions are longer than the existing column, so the table may need reformatting after you apply this.
| | `server_shutdown` | The agent closed the connection because the agent was shutting down. | | |
| | `workspace_stopped` | A workspace stop or delete closed the connection. `build_reason` shows what started the build. | | |
| | `server_shutdown` | The agent closed the connection because the agent was shutting down. In the connection log, `coderd` can replace it with `workspace_stopped`. | | |
| | `workspace_stopped` | Connection log only. A workspace stop or delete closed the connection. The `coder ssh` client log records `build_reason`. | |
| | `workspace_stopped` | A workspace stop or delete closed the connection. `build_reason` shows what started the build. | | ||
| | `unknown` | The agent couldn't determine why the connection closed. | | ||
|
|
||
| For SSH sessions, the reason can also be a specific message, such as `process exited with error status: 1` when the command exited with a nonzero status. |
There was a problem hiding this comment.
Nit: the free-form reasons also include file transfer blocked, unsupported subsystem: <name>, x11 handler failed, and raw SFTP or session error strings (agent/agentssh/agentssh.go:603-688).
If the agent is shutting down when the session ends, the reason becomes server_shutdown and the original text moves to disconnect_detail (agentssh.go:581-586).
| For SSH sessions, the reason can also be a specific message, such as `process exited with error status: 1` when the command exited with a nonzero status. | |
| For SSH sessions, the reason can also be a specific message, such as `process exited with error status: 1` when the command exited with a nonzero status, or `file transfer blocked`. | |
| If the agent was shutting down, the reason is `server_shutdown`, and the specific message is in `disconnect_detail`. |
There was a problem hiding this comment.
Done. The raw error text case was added earlier, and c98046a adds the shutdown case: the reason becomes server_shutdown and the specific message moves to disconnect_detail (agent/agentssh/agentssh.go, ssh session closed defer).
| For SSH sessions, the reason can also be a specific message, such as `process exited with error status: 1` when the command exited with a nonzero status. | ||
|
|
||
| `disconnect_expected` is `true` for `graceful`, `server_shutdown`, and `workspace_stopped`, and `false` for every other reason. | ||
| Filter on `disconnect_expected=false` to find the disconnects worth investigating. |
There was a problem hiding this comment.
disconnect_expected is only in agent and coder ssh logs; connection_log entries don't have it.
A reader following the earlier jq example would filter connection_log entries and get nothing.
| Filter on `disconnect_expected=false` to find the disconnects worth investigating. | |
| Filter agent logs on `disconnect_expected=false` to find the disconnects worth investigating. |
There was a problem hiding this comment.
Addressed in 6e3b712: the filter is scoped to agent logs, with a note that connection_log entries don't include disconnect_expected.
| When `disconnect_reason` is `workspace_stopped`, `build_reason` shows what stopped the workspace: | ||
|
|
||
| | `build_reason` | The workspace stopped because | | ||
| |----------------|-----------------------------------------------| | ||
| | `initiator` | A user or API client requested the build. | | ||
| | `autostop` | The workspace reached its autostop schedule. | | ||
| | `dormancy` | The workspace became dormant from inactivity. | |
There was a problem hiding this comment.
Stop and delete builds can also have failedstop and autodelete (coderd/database/models.go:1410-1411, set in coderd/autobuild/lifecycle_executor.go:673).
codersdk.BuildReason doesn't define constants for them, but the raw value passes through.
Dashboard and CLI stops use initiator; dashboard and cli are only sent for starts (site/src/api/api.ts:1409).
This also points the reader at the coder ssh log, per the comment on line 197.
| When `disconnect_reason` is `workspace_stopped`, `build_reason` shows what stopped the workspace: | |
| | `build_reason` | The workspace stopped because | | |
| |----------------|-----------------------------------------------| | |
| | `initiator` | A user or API client requested the build. | | |
| | `autostop` | The workspace reached its autostop schedule. | | |
| | `dormancy` | The workspace became dormant from inactivity. | | |
| When the `coder ssh` client log shows `disconnect_reason=workspace_stopped`, `build_reason` shows what stopped the workspace: | |
| | `build_reason` | The workspace stopped because | | |
| |----------------|----------------------------------------------------------------------------------| | |
| | `initiator` | A user or API client requested the build. | | |
| | `autostop` | The workspace reached its autostop schedule. | | |
| | `dormancy` | The workspace became dormant from inactivity. | | |
| | `failedstop` | The workspace's last build failed and the template's failure cleanup stopped it. | | |
| | `autodelete` | The workspace was dormant long enough for the template to delete it. | |
There was a problem hiding this comment.
Addressed in 6e3b712: added autodelete; left out failedstop since nothing sets it.
| 1. Search the agent log for the session ID: | ||
|
|
||
| ```sh | ||
| grep '<session-id>' <agent-log-dir>/coder-agent.log |
There was a problem hiding this comment.
The agent log rotates.
lumberjack rolls coder-agent.log at 5 MB and keeps 19 backups named coder-agent-<timestamp>.log (cli/agent.go:167-171), so this grep can miss the earlier part of a session.
Consider also adding a sentence near line 129 that older entries are in rotated coder-agent-*.log files.
| grep '<session-id>' <agent-log-dir>/coder-agent.log | |
| grep '<session-id>' <agent-log-dir>/coder-agent*.log |
There was a problem hiding this comment.
Addressed in 6e3b712: the grep uses coder-agent*.log and the Agent logs section describes rotation at 5 MB with 19 older files.
| It writes the `coder ssh` logs to the directory in the `coder.proxyLogDirectory` VS Code setting. | ||
| If the setting is empty, the extension uses `CODER_SSH_LOG_DIR`, and if that isn't set either, a `log` directory in the extension's global storage. | ||
| In the **Coder** output channel, each line starts with `[session` and the first 8 characters of the session ID. | ||
| The extension keeps the 1,000 most recent entries below the channel's log level and writes them to the channel when a connection fails or the user collects a support bundle. |
There was a problem hiding this comment.
Nit: coder.connectionLogBuffer.size and this buffer shipped in v1.16.4 (vscode-coder CHANGELOG), but the prerequisite on line 19 says 1.16.1 or later.
Either call out 1.16.4 here or raise the minimum in the prerequisite.
There was a problem hiding this comment.
Superseded in c98046a: the prerequisite is now Coder Remote 1.17.2 or later, which covers the log buffer, so the 1.16.4 call-out is gone.
| Turn on JSON output with [`CODER_LOGGING_JSON`](../../reference/cli/server/index.md#--log-json). | ||
| - The `connection_log` entries in this guide require a [Premium license](https://coder.com/pricing#compare-plans). | ||
| Client and agent logs don't. | ||
| - For VS Code, use the Coder Remote extension version 1.16.1 or later to send session IDs, and version 1.17.0 or later to report the editor name, such as `cursor`. |
There was a problem hiding this comment.
Nit: v1.17.0 has a "Premature close" regression in Positron, VSCodium, and other editors that v1.17.1 fixed (vscode-coder CHANGELOG).
Pointing at 1.17.1 avoids sending readers to a broken build.
| - For VS Code, use the Coder Remote extension version 1.16.1 or later to send session IDs, and version 1.17.0 or later to report the editor name, such as `cursor`. | |
| - For VS Code, use the Coder Remote extension version 1.16.1 or later to send session IDs, and version 1.17.1 or later to report the editor name, such as `cursor`. |
There was a problem hiding this comment.
Superseded in c98046a: the minimum is now 1.17.2, which is past the 1.17.0 regression.
|
|
||
| One client session often spans several connections. | ||
| For example, an IDE that reconnects after a network interruption opens a new SSH connection each time. | ||
| Each connection gets its own connection log entry, with its own `ConnectionID` and the same `ClientSessionID`. |
There was a problem hiding this comment.
Nit: this reads as one entry per connection, but line 107 says each SSH or web terminal connection produces a connected and a disconnected service log entry.
| Each connection gets its own connection log entry, with its own `ConnectionID` and the same `ClientSessionID`. | |
| Each connection has its own `ConnectionID`, shared by its `connected` and `disconnected` entries, and the same `ClientSessionID` as the other connections in the session. |
There was a problem hiding this comment.
Addressed in 6e3b712 with your wording (one ConnectionID shared by the connected and disconnected entries).
… connections guide The agent logs server_shutdown with disconnect_initiator on ssh connection complete, and only the coder ssh log carries build_id, build_reason, and transition. coderd rewrites server_shutdown to workspace_stopped in the connection log when a stop or delete build caused it.
|
All contributors have signed the CLA ✍️ ✅ |
4b31769 to
2653d37
Compare
nickvigilante
left a comment
There was a problem hiding this comment.
Review of the Trace workspace connections guide
I checked the page against origin/main (45e392b8be) at head 2653d37.
Most claims hold; the inline comments cover four important issues and four nits.
Important
- JetBrains Toolbox sessions don't produce
ssh session closedlines, and their normal close reason isnormal close(line 148). - SSH sessions never record
network_error, so the filter misses SSH transport failures (line 204). CODER_LOGGING_JSONtakes a file path, not an on/off value (line 16).- Only
5xxfailures are logged at warn;4xxfailures need debug logging (lines 124-125 and 171).
Still open from earlier threads
These threads are unresolved and the current text still has the issue:
- Line 176: the
greponly readscoder-agent.log, but the agent rotates at 5 MB and keeps 19 backups (cli/agent.go:167-171); usecoder-agent*.log. - Lines 206-207:
connection_logentries don't havedisconnect_expected. - Lines 213-217: the
build_reasontable is missingautodelete(coderd/autobuild/lifecycle_executor.go:673);failedstopis in the enum but nothing sets it. - Lines 19 and 84: version minimums (1.16.4 for the log buffer, 1.17.1 instead of 1.17.0).
- Line 45: one connection produces a
connectedand adisconnectedentry. - fioan89's Toolbox threads about the full 32-character ID.
Verified against code
Session ID format and CODER_TRACE_SESSION_ID, coder ssh log directories, file names, retention, and flight recorder size, agent log location and flight recorder behavior, agent log messages, the server_shutdown to workspace_stopped rewrite, the workspace is shutting down fields, web terminal console errors, the jq example, the method:tunnel filter, and doc link anchors.
I didn't verify VS Code or Toolbox extension behavior, since that code lives in other repositories.
This review was generated by Coder Agents on behalf of @nickvigilante.
|
|
||
| | Message | What it records | | ||
| |--------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------| | ||
| | `handling ssh session` | An SSH session started. | |
There was a problem hiding this comment.
Important: JetBrains Toolbox sessions don't follow the pattern this table describes.
- The agent sets
reportSession = falsefor the JetBrains app family (agent/agentssh/agentssh.go:553-558), so there's nossh session closedline and no per-session connection log entry for those sessions. - Toolbox connection log entries come from
JetbrainsChannelWatcher, which tracks the single persistent forwarded channel (agent/agentssh/jetbrainstrack.go:92-130). - On a normal close, that path reports
DisconnectReasonasnormal close, which isn't in the reasons table. - The agent logs
JetBrains watcher accepted channelandJetBrains channel closedat debug level, so they aren't in the agent log by default. handling ssh sessionstill fires for each of the many sessions JetBrains opens, so this search is noisy for Toolbox users.
Since the guide lists Toolbox as a supported client, could we add a short JetBrains note here and in "Read disconnect reasons"?
There was a problem hiding this comment.
Addressed in 6e3b712: the Agent logs section now notes that Toolbox sessions are frequent handling ssh session lines with no ssh session closed or per-session connection log entry, that the forwarded channel is reported as one connection with JetBrains watcher accepted channel / JetBrains channel closed at debug level, and normal close is in the reasons table.
| | `workspace_stopped` | A workspace stop or delete closed the connection. Only the connection log and the `coder ssh` log record this reason. | | ||
| | `unknown` | The agent couldn't determine why the connection closed. | | ||
|
|
||
| For SSH sessions, the reason can also be a specific message, such as `process exited with error status: 1` when the command exited with a nonzero status. |
There was a problem hiding this comment.
Important: SSH sessions never record network_error (line 199).
Only the web terminal uses it (agent/reconnectingpty/server.go:139).
When an SSH session fails, the agent records the raw error text as the reason (closeCause(err.Error()), agent/agentssh/agentssh.go:633,689), so filtering on network_error misses every SSH transport failure.
| For SSH sessions, the reason can also be a specific message, such as `process exited with error status: 1` when the command exited with a nonzero status. | |
| For SSH sessions, the reason can also be a specific message, such as `process exited with error status: 1` when the command exited with a nonzero status, or the raw error text when the session failed. | |
| SSH sessions don't record `network_error`, so search the agent log for `ssh session failed` to find SSH transport failures. |
There was a problem hiding this comment.
Addressed in 6e3b712: network_error is scoped to the web terminal, and the text points to ssh session failed for SSH transport failures.
| Agents from earlier versions keep working, but they don't record client session IDs, app names, workspace shutdown reasons, or bytes transferred. | ||
| - Collect `coderd` logs with a log management tool such as Splunk, Datadog, or Grafana Loki. | ||
| JSON output is easiest to query. | ||
| Turn on JSON output with [`CODER_LOGGING_JSON`](../../reference/cli/server/index.md#--log-json). |
There was a problem hiding this comment.
Important: --log-json / CODER_LOGGING_JSON is a string flag that takes a file path ("Output JSON logs to a given file", docs/reference/cli/server/index.md:881-889), not an on/off switch.
"Turn on JSON output with" suggests setting it to true, which writes the log to a file named true.
| Turn on JSON output with [`CODER_LOGGING_JSON`](../../reference/cli/server/index.md#--log-json). | |
| To write JSON logs, set [`CODER_LOGGING_JSON`](../../reference/cli/server/index.md#--log-json) to a file path, such as `/dev/stderr`. |
There was a problem hiding this comment.
Addressed in 6e3b712: it now says to set CODER_LOGGING_JSON to a file path such as /dev/stderr, and to point CODER_LOGGING_HUMAN at /dev/null to keep only JSON there.
| Failed requests with a `5xx` status are logged at the warn level. | ||
| Successful requests are logged at the debug level, so they appear only when debug logging is on. |
There was a problem hiding this comment.
Important: coderd/httpmw/loggermw/logger.go:182-185 logs 5xx at warn and every other status at debug, including 401, 403, and 404.
"Successful requests" undersells what's hidden, and step 2 of "Correlate a session" (line 171) misses the most common client failures unless debug logging is on.
| Failed requests with a `5xx` status are logged at the warn level. | |
| Successful requests are logged at the debug level, so they appear only when debug logging is on. | |
| Requests that fail with a `5xx` status are logged at the warn level. | |
| All other requests, including `4xx` failures such as `401`, `403`, and `404`, are logged at the debug level, so they appear only when debug logging is on. |
There was a problem hiding this comment.
Addressed in 6e3b712 with your wording: only 5xx is warn; 4xx and successes are debug.
| Note the time of each `connected` and `disconnected` entry and its `DisconnectReason`. | ||
|
|
||
| 1. Search the `coderd` logs for `client_session_id` set to the same value. | ||
| Matching warn-level entries show failed requests from the client. |
There was a problem hiding this comment.
Follow-up to the 5xx comment above: warn-level entries only cover 5xx failures.
| Matching warn-level entries show failed requests from the client. | |
| Matching warn-level entries show requests that failed with a `5xx` status. | |
| To see `4xx` failures, turn on debug logging. |
There was a problem hiding this comment.
Addressed in 6e3b712: step 2 now says warn entries cover 5xx only and to turn on debug logging for 4xx.
| | Field | Description | | ||
| |----------------------|-------------------------------------------------------------------------------------------------------------------------------------------| | ||
| | `ClientSessionID` | The client session ID. Empty for workspace app, port forwarding, and tunnel events, and for connections from older clients or agents. | | ||
| | `ConnectionID` | Identifies one SSH or web terminal connection. | |
There was a problem hiding this comment.
Nit: ConnectionID is per SSH session, not per SSH connection.
sessionHandler creates it (agent/agentssh/agentssh.go:490), so one SSH connection that carries several sessions produces several entries.
The same UUID is the id field on the agent's handling ssh session and ssh session closed lines, which gives readers a direct join between the agent log and the connection log.
Line 45 has the same wording.
| | `ConnectionID` | Identifies one SSH or web terminal connection. | | |
| | `ConnectionID` | Identifies one SSH session or web terminal connection. For SSH, matches the `id` field in the agent log. | |
There was a problem hiding this comment.
Addressed in 6e3b712: ConnectionID is per SSH session or web terminal connection and matches the agent's id field; the intro paragraph uses the same wording.
| | Message | What it records | | ||
| |--------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------| | ||
| | `handling ssh session` | An SSH session started. | | ||
| | `ssh session closed` | An SSH session ended. Coder includes `disconnect_reason`, `disconnect_expected`, and `exit_code`. | |
There was a problem hiding this comment.
Nit: ssh session failed (warn level, agent/agentssh/agentssh.go:686) is the line logged right before an error-text reason, so it's worth listing.
| | `ssh session closed` | An SSH session ended. Coder includes `disconnect_reason`, `disconnect_expected`, and `exit_code`. | | |
| | `ssh session closed` | An SSH session ended. Coder includes `disconnect_reason`, `disconnect_expected`, and `exit_code`. | | |
| | `ssh session failed` | An SSH session failed with an error. The error text becomes the `disconnect_reason`. | |
There was a problem hiding this comment.
Added ssh session failed to the agent log table in 6e3b712.
| | `network_error` | The transport between the client and the agent failed, for example with an EOF, a read or write error, or a timeout. | | ||
| | `server_shutdown` | The agent closed the connection because the agent was shutting down. | | ||
| | `workspace_stopped` | A workspace stop or delete closed the connection. Only the connection log and the `coder ssh` log record this reason. | | ||
| | `unknown` | The agent couldn't determine why the connection closed. | |
There was a problem hiding this comment.
Nit: unknown only appears in the agent log.
codersdk.DisconnectReason.SlogField() renders the empty reason as unknown (codersdk/disconnect.go:24-26), but the agent reports "" to coderd, which stores it as NULL, so DisconnectReason in the connection log is empty instead.
There was a problem hiding this comment.
Addressed in 6e3b712: the unknown row says only the agent log shows it and DisconnectReason is empty in the connection log.
|
|
||
| For SSH sessions, the reason can also be a specific message, such as `process exited with error status: 1` when the command exited with a nonzero status. | ||
|
|
||
| `disconnect_expected` is `true` for `graceful`, `server_shutdown`, and `workspace_stopped`, and `false` for every other reason. |
There was a problem hiding this comment.
Nit: Expected() also returns true for client_closed (codersdk/disconnect.go:104-110).
Nothing emits it today, so this is only a risk if a client starts using it, but "false for every other reason" won't hold if one does.
There was a problem hiding this comment.
Reworded in 6e3b712 to list the false cases explicitly (network_error, unknown, error messages) instead of "every other reason", so it stays true if a client starts emitting client_closed.
Plugin 0.9.5 requires JetBrains Toolbox 3.9.0, and earlier plugin versions collect only Toolbox logs without a Coder support bundle. Link the Toolbox support bundle section.
Adds a how-to guide under Monitoring for platform teams: find a user's
client_session_id, collect client,coderd, and agent logs, correlate them, and readdisconnect_reasonandbuild_reason. Also links it from the connection logs service-log section and adds it to the manifest.Resolves CLIENT-1032. Written ahead of merge under the multi-PR launch exception in the content guidelines. It describes behavior from these PRs, so it should merge after the open ones:
AppNameOrPort, such ascursor)1a65385)server_shutdownwithdisconnect_initiatoronssh connection complete;coderdrecordsworkspace_stoppedin the connection log;build_id,build_reason,transitionappear only on thecoder sshlog lineworkspace is shutting downRxBytes/TxBytesondisconnectedentries for SSH and web terminal connectionsConnectPath/DisconnectPath. @code-asher said on 2026-10-27 to leave the connection path out because it needs a new solution, so it isn't part of this guide.coder sshlogs. Ships in plugin v0.9.5 (confirmed by @fioan89); the guide names v0.9.5 and keeps "ask for the files separately" for earlier versions.coder sshlog directory itself, passes it tocoder sshwith--log-dir, and addscoder-ssh*-<pid>.logfiles from the last three days (each cut to its last 50 MB) straight into the support bundle. Ships in v0.9.5; the JetBrains Toolbox section now describes the new log directory order and where the logs land in the bundle.[session <first 8 characters>]to[client_session_id=<full 32-character ID>]. The guide now requires v1.17.2 and describes the full-ID prefix (c98046a, caf1bb7).client_session_idin the connection log API, aclient_session_id:filter, and View related connections in the UI.docs/support/support-bundle.md: plugin 0.9.5 and JetBrains Toolbox 3.9.0; plugin 0.9.4 and earlier collect Toolbox logs without a Coder support bundle. "Before you begin" and the JetBrains Toolbox section now match and link it.No 2.39 release branch or tag yet. Latest published Toolbox plugin release is still v0.9.4 (2026-08-26); the v0.9.5 draft release was re-created 2026-10-19 after #373 merged but isn't published. @fioan89 confirmed 2026-10-21 that v0.9.5 is the release with all the connection log work. Still a draft on 2026-10-27. The plugin changelog on
mainlists JetBrains Toolbox 3.7.2 as its minimum, while #31289 documents 3.9.0 for v0.9.5. coder/coder-jetbrains-toolbox#374 (ready for review since 2026-10-22, no approvals yet) adds the Toolbox connection log buffer (JET-4); the guide will cover it once it merges and ships.Changes since the first draft
ConnectPath/DisconnectPathfields, the path log messages, and the "Read connection paths" section) because feat: log and record DERP/direct connection path per agent connection #30665 closed without merging.coder sshlog files, so the guide asks for them separately. Tracked in JET-34.RxBytes/TxBytesrow. @code-asher confirmed bytes reach the connection log service logs (not the API), so they can be ingested from there. No file changes today.coder sshlog directory from the Toolbox section. Fixed the agent log location (coder-agent.loginCODER_AGENT_LOG_DIR, otherwise the system temp directory:$TMPDIRor/tmpon Linux and macOS,%TEMP%on Windows) and documented the VS Code log directory fallback (coder.proxyLogDirectory, thenCODER_SSH_LOG_DIR, then the extension's global storage). Checked againstcli/agent.goand vscode-coderpathResolver.ts.RxBytes/TxBytesrow now says the agent sets them only ondisconnectedentries for SSH and web terminal connections, and that older agents leave them empty (percoderd/agentapi/connectionlog.goand the migration column comments). "Before you begin" now lists bytes among the fields earlier agents don't record. Commit 32af7d2.cdr.dev/slog/v3callsValue()on anydriver.Valuerbefore encoding (map.go), sosql.NullInt64renders as a plain number, ornullwhen unset, not a{Int64, Valid}object.01a3b79f06. The agent log table now saysssh connection completegetsdisconnect_reason,disconnect_expected, anddisconnect_initiatoronly when the agent is shutting down (agent/agentssh/agentssh.go,shutdown.go). "Read disconnect reasons" now says SSH connections (as opposed to sessions) aren't reported to the connection log, and that the agent log (server_shutdown) and connection log (workspace_stopped) can differ, with the conditions fromworkspaceShutdownReasonincoderd/agentapi/connectionlog.go.build_id,build_reason, andtransitionare now attributed to thecoder sshlog'sworkspace is shutting downline (cli/ssh.go). Commit 55c8c16.coderdchanges match what the guide was checked against on 2026-10-17, so no file changes. The fix: log workspace shutdown reasons for SSH connections #30515 description sayscoderdlogs the build reason, transition, and build ID, but mergedcoderd/agentapi/connectionlog.goonly persists the enriched reason and doesn't log those fields, so the guide keeps them on thecoder sshline.coder sshlog directory order (plugin setting, thenCODER_SSH_LOG_DIR, thenssh-logsin the plugin data directory) and says recentcoder sshlogs land in the support bundle undertoolbox-logs/ssh/, per the plugin README and changelog onmain. Earlier versions still need the files requested separately. Commit 2653d37.mainby a Coder Agents run).CODER_LOGGING_JSONtakes a file path, with a note that human-readable logs also default to/dev/stderr. Only5xxrequest failures log at warn;4xxneed debug logging (field notes and correlate step 2).ConnectionIDis per SSH session and matches the agent logid. Newssh session failedrow; SSH sessions don't recordnetwork_error(only the web terminal does), so the guide points tossh session failed.unknownappears only in the agent log; the connection log leavesDisconnectReasonempty. JetBrains Toolbox note: nossh session closedor per-session connection log entries, one connection for the forwarded channel,normal closereason, debug-only watcher lines.disconnect_expectedis agent-log only. Also closed the older open threads:coder-agent*.logglob plus rotation (5 MB, 19 backups),autodeletebuild reason, VS Code minimums (1.16.4 for the log buffer, 1.17.1 for editor names, since 1.17.0 has a "Premature close" regression), and Toolbox lines carrying the full 32-character ID (CoderLogger.withSessionIdon pluginmain). Commit 6e3b712.coder-support.zip(withcoder sshlogs undertoolbox-logs/ssh/), earlier plugin versions collect only Toolbox logs, and points to the CLI bundle and the new Toolbox section in the support bundle guide. Thesupport-bundle.md#jetbrains-toolboxanchor exists onmainbut not on this branch yet, so it resolves after amainmerge or on merge. Commit e5a37d4.ClientSessionID=: onmain,coderd, the agent, and the CLI log the field asclient_session_id(coderd/tracing/httpmw.go,agent/agentssh/agentssh.go,cli/root.go), andClientSessionIDappears only onconnection_logentries, which is what the guide already says.[client_session_id=<session-id>]prefix with the full 32-character ID. Earlier versions show only the first 8 characters, and versions before 1.17.1 don't report editor names. Dropped the "Starting with version 1.16.4" qualifier on the log buffer, since 1.17.2 is now the minimum. "Read disconnect reasons" now says that when the agent is shutting down, SSH sessions reportserver_shutdownand the agent log keeps the original message indisconnect_detail(agent/agentssh/agentssh.goonmain). Replied to every open review thread, including Nick's mixed-agent question: agents only update when a workspace rebuilds, so a new server routinely serves earlier agents, and the prerequisite lists what earlier agents don't record. Commits c98046a, caf1bb7.Open questions
ssh_info.durationfrom the API, so the UI derives duration from connect and disconnect times, and the API addsclient_session_idplus aclient_session_id:filter. If it merges before the 2.39 freeze, add the filter and the View related connections link to the guide?CLI and agent defaults (
CODER_FLIGHT_RECORDER_SIZE10,CODER_AGENT_FLIGHT_RECORDER_SIZE1000, 50 retainedcoder sshlogs, XDG state directory) come fromcli/flightrecorder.go,cli/root.go, andcli/agent.goonmain. The commands in the page weren't run against a live 2.39 deployment.This description is refreshed daily at 05:00 UTC (10:00 PKT) from the Linear project, the dependency PRs, and #proj-connection-troubleshooting.