Skip to content

feat: buffer CLI session logs and flush on connection failure - #29752

Merged
aqandrew merged 9 commits into
mainfrom
cli-4-session-log-buffer
Sep 29, 2026
Merged

aqandrew merged 9 commits into
mainfrom
cli-4-session-log-buffer

Conversation

@aqandrew

@aqandrew aqandrew commented Sep 22, 2026 •

Copy link
Copy Markdown
Collaborator

What

Adds in-memory log buffering for CLI diagnostics so the debug detail leading up to a failure is available without logging debug during normal operation. Below-level entries are kept in a slog flight recorder (v3.2.0) and emitted only when a command returns an error. This implements CLI-4 (connection-log RFC req 13). Depends on the cdr.dev/slog/v3 v3.2.0 bump (already on main).

How

  • Root command (cli/root.go):
    • New global --log-buffer-size / CODER_LOG_BUFFER_SIZE (default 10, 0 disables, clamped to 10000), shared by all commands instead of duplicated per command.
    • bufferedLoggerMiddleware runs the invocation logger at Info (or Debug with -v) with a flight recorder, and on a returned error calls Flush to replay the buffered history to stderr. Flush does not add a duplicate error line; the returned error is still rendered by the normal formatter.
    • Opt-in via an annotationBufferedLogger annotation so it does not clobber the ~17 commands that build their own logger on inv.Logger (which would otherwise double-log). Future commands opt in with one line.
  • start / stop / update: opt into the middleware and just client.SetLogger(inv.Logger) so codersdk request/response debug is captured and flushed to stderr on failure. They no longer write session log files and dropped their per-command --log-dir / --log-buffer-size.
  • ssh: keeps its own file logger (writing to stderr would corrupt its stdio stream), now using the global buffer size and the shared bufferedLogger helper. It retains --log-dir / CODER_SSH_LOG_DIR, stdio/PID filename handling, and file pruning (50 most recent).
  • restart is excluded, matching the client_session_id opt-in set (DEVEX-664).

Default behavior change

start / stop / update now emit buffered diagnostics (recent below-level entries, e.g. sdk request/response) to stderr when they fail; on success they stay quiet. coder ssh continues to write a per-connection diagnostic file by default under the XDG state dir (~/.local/state/coder/logs), pruned to the 50 most recent. Opt out of buffering with --log-buffer-size 0.

Review changes (Asher)

  • Shared logger construction: the level is computed once and the flight recorder is always attached, in a single bufferedLogger helper used by ssh and the workspace commands.
  • Workspace commands print to normal output instead of a log file, and the default buffer is 10 to avoid flooding the terminal on failure.
  • The buffer lives on the root command / logger (global option + middleware) rather than being wired into each command.

Alignment with #29671 (agent)

Same core model: Info default with a flight recorder, -v for full debug, and the shared --log-buffer-size semantics (0 disables). Differences: the agent writes to a file and defaults the buffer to 1000, while the CLI prints to stderr and defaults to 10; both now rely on slog's auto-flush-on-error (the CLI additionally flushes via the middleware on a returned error).

Note

The agent PR (#29671) adds an agent-scoped --log-buffer-size / CODER_AGENT_LOG_BUFFER_SIZE on the hidden coder agent command. Once both land, the global --log-buffer-size here and the agent's own flag will need reconciling on coder agent. Flagging for whichever merges second.

Testing

cli.TestClampLogBufferSize, TestPruneSessionLogs, TestDefaultSessionLogDir, and TestBufferedLogger_* cover clamping, ssh log retention, the default state dir, buffer-until-error, the middleware's flush path, and the verbose escape hatch. start/stop/update/ssh command tests pass. Golden --help output and docs/reference/cli regenerated. (TestSSH/Env is a pre-existing interactive-PTY test that does not run in all sandboxes; it fails identically on main here.)

Rollout context

Cross-repo sequence: coder/slog flight recorder (released as v3.2.0) -> go.mod bump in coder/coder (#29600, merged) -> CLIENT-672 (agent, #29671) and this PR (CLI-4), which are independent consumers and can merge in any order.


Opened by Coder Agents on behalf of @aqandrew.

@linear-code

linear-code Bot commented Sep 22, 2026

Copy link
Copy Markdown

CLI-4

Comment thread cli/testdata/coder_start_--help.golden Resolved
Comment thread cli/ssh.go Resolved
Comment thread cli/logbuffer.go Resolved
@spikecurtis
spikecurtis removed their request for review September 28, 2026 07:13
Implements CLI-4 (connection-log RFC req 13) for the session commands:
ssh, start, stop, and update. Adds a shared helper (cli/logbuffer.go) that
builds a per-invocation file logger which, unless --verbose (-v) is set,
runs at Info and keeps a rolling in-memory history of debug entries via a
slog flight recorder. The history is written to the log file automatically
when the command logs its terminal error on exit, so the detail leading up
to a failure is available without logging debug during normal operation.

- New shared --log-buffer-size / CODER_LOG_BUFFER_SIZE (default 1000, 0
  disables, clamped to 10000).
- Logs default to the XDG state dir (~/.local/state/coder/logs) via
  github.com/adrg/xdg; the most recent 50 files are retained.
- start/stop/update gain a --log-dir (CODER_LOG_DIR) and set the client
  logger so HTTP request detail is captured; ssh keeps its existing
  --log-dir / CODER_SSH_LOG_DIR and now logs by default.
- restart is excluded to match the client_session_id opt-in set.
@aqandrew
aqandrew force-pushed the cli-4-session-log-buffer branch from 111aae0 to d703984 Compare September 28, 2026 22:17
…n error

Adds a global --log-buffer-size / CODER_LOG_BUFFER_SIZE option (default 10)
and a root bufferedLoggerMiddleware that keeps below-level entries in a flight
recorder and emits them to stderr only when a command returns an error.
Workspace commands (start, stop, update) opt in and no longer write session
log files; ssh keeps its file logger to avoid corrupting its stdio stream.
@aqandrew
aqandrew requested a review from code-asher September 29, 2026 00:56

@code-asher code-asher left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool! I think we should be consistent with the "flight recorder" name but otherwise it looks good to go to me.

Comment thread cli/logbuffer.go Resolved
Comment thread cli/root.go Resolved
Comment thread cli/root.go Resolved
Collect glob, stat, and remove failures with errors.Join and return them
instead of silently ignoring them, and log them at Warn in the ssh session
log file. Pruning stays best effort and never fails the command.
Rename the buffered-logger naming to flight recorder for consistency with the
underlying slog API: --flight-recorder-size / CODER_FLIGHT_RECORDER_SIZE,
flightRecorderMiddleware, the flightRecorder helper, the annotationFlightRecorder
annotation, and cli/flightrecorder.go. No behavior change.
aqandrew added a commit that referenced this pull request Sep 29, 2026
## What

Runs the workspace agent logger at `Info` with a slog flight recorder
(`v3.2.0`) instead of unconditionally at `Debug`. Entries below the
current log level are kept in a rolling in-memory ring and flushed to
the sinks when the agent loses its connection to coderd (and before any
`Error`+ log), so the detail leading up to a failure is available
without writing debug logs during normal operation.

Implements CLIENT-672 (connection-log RFC req 13) for the workspace
agent. Depends on the `cdr.dev/slog/v3` `v3.2.0` bump (already on
`main`).

## How

- `cli/agent.go`:
- The main agent logger is
`AppendSinks(...).Leveled(level).FlightRecorder(flightRecorderSize)`,
where `level` is `Info` by default and `Debug` under the global `-v` /
`CODER_VERBOSE`. The recorder is always attached; at `Debug` it is
inert, which keeps the setup correct if the configured level changes in
the future. The reaper logger is unchanged.
- New `--agent-flight-recorder-size` /
`CODER_AGENT_FLIGHT_RECORDER_SIZE` (default `1000`, `0` disables the
flight recorder), so the ring size is configurable per RFC req 13. The
agent flag is distinct from CLI-4's global `--flight-recorder-size`
(#29752) so the two don't collide when `coder agent` inherits the root
global.
- `workspaceAgent` is a `RootCmd` method so it reuses the root command's
global verbose flag instead of defining an agent-specific one.
- `agent/agent.go` (`runLoop`): on a lost coderd connection (the
`io.EOF` disconnect and the generic run error) the agent calls
`logger.Flush(ctx)` to emit the recorded history and then logs at
`Warn`. These are `Warn` rather than `Error` because reconnectable
disconnects are expected and `slogtest` fails tests on any `Error` log;
the manual flush is what surfaces the recorded detail. Graceful shutdown
paths (`ctx` canceled, agent closed) are not flushed.
- Tests: `agent.TestAgent_FlushFlightRecorderOnDisconnect` covers the
flush-on-disconnect path; `cli.TestWorkspaceAgent/DisabledServers` runs
with verbose since its debug messages are otherwise recorded. Golden
`--help` output regenerated.

## Default log verbosity change

This changes the default agent-log verbosity. Today `coder-agent.log`
receives `Debug` unconditionally; with this change, normal operation
logs `Info`, and recorded below-level entries are emitted only when the
agent loses its coderd connection (via a manual flush) or before any
`Error`+ log, or always under `-v` / `CODER_VERBOSE`. This is the
intended behavior of the feature: a quiet steady state with full detail
retained for diagnosing failures. Operators who want the previous
always-debug output can run with `-v` / `CODER_VERBOSE`.

## Alignment with #29752 (CLI)

This mirrors the CLI-4 (CLI log buffer) design so the two
implementations agree:

- **Level model:** Info by default, debug available on demand via the
global `-v` / `CODER_VERBOSE`.
- **Flight recorder size:** default `1000`, `0` disables. CLI-4 exposes
a global `--flight-recorder-size`; the agent uses a distinct
`--agent-flight-recorder-size` / `CODER_AGENT_FLIGHT_RECORDER_SIZE` (the
`CODER_AGENT_*` env convention) so the inherited global and the agent
flag coexist without collision.
- **Flush trigger:** the recorded history is emitted before any `Error`+
log via slog's auto-flush. The agent additionally flushes manually on a
lost coderd connection, which it logs at `Warn` (reconnectable, and
`Error` would fail `slogtest`-based tests); the CLI's failures are
fatal, so it relies on the auto-flush-on-error.

<details>
<summary>Rollout context</summary>

Cross-repo sequence: coder/slog flight recorder (released as v3.2.0) ->
go.mod bump in coder/coder (#29600, merged) -> this PR (CLIENT-672,
agent) and CLI-4 (CLI), which are independent consumers and can merge in
any order.
</details>

---
<sub>Opened by Coder Agents on behalf of @aqandrew.</sub>
@aqandrew
aqandrew enabled auto-merge (squash) September 29, 2026 21:34
@aqandrew
aqandrew merged commit 4a67ac3 into main Sep 29, 2026
32 checks passed
@aqandrew
aqandrew deleted the cli-4-session-log-buffer branch September 29, 2026 22:27
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 29, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants