Skip to content

feat: buffer agent debug logs and flush on connection failure - #29671

Merged
aqandrew merged 11 commits into
mainfrom
client-672-agent-log-buffer
Sep 29, 2026
Merged

aqandrew merged 11 commits into
mainfrom
client-672-agent-log-buffer

Conversation

@aqandrew

@aqandrew aqandrew commented Sep 21, 2026 •

Copy link
Copy Markdown
Collaborator

What

Runs the workspace agent logger at Info with a slog flight recorder (v3.2.0) instead of unconditionally at Debug. Entries below the current log level are kept in a rolling in-memory ring and flushed to the sinks when the agent loses its connection to coderd (and before any Error+ log), so the detail leading up to a failure is available without writing debug logs during normal operation.

Implements CLIENT-672 (connection-log RFC req 13) for the workspace agent. Depends on the cdr.dev/slog/v3 v3.2.0 bump (already on main).

How

  • cli/agent.go:
    • The main agent logger is AppendSinks(...).Leveled(level).FlightRecorder(flightRecorderSize), where level is Info by default and Debug under the global -v / CODER_VERBOSE. The recorder is always attached; at Debug it is inert, which keeps the setup correct if the configured level changes in the future. The reaper logger is unchanged.
    • New --agent-flight-recorder-size / CODER_AGENT_FLIGHT_RECORDER_SIZE (default 1000, 0 disables the flight recorder), so the ring size is configurable per RFC req 13. The agent flag is distinct from CLI-4's global --flight-recorder-size (feat: buffer CLI session logs and flush on connection failure #29752) so the two don't collide when coder agent inherits the root global.
    • workspaceAgent is a RootCmd method so it reuses the root command's global verbose flag instead of defining an agent-specific one.
  • agent/agent.go (runLoop): on a lost coderd connection (the io.EOF disconnect and the generic run error) the agent calls logger.Flush(ctx) to emit the recorded history and then logs at Warn. These are Warn rather than Error because reconnectable disconnects are expected and slogtest fails tests on any Error log; the manual flush is what surfaces the recorded detail. Graceful shutdown paths (ctx canceled, agent closed) are not flushed.
  • Tests: agent.TestAgent_FlushFlightRecorderOnDisconnect covers the flush-on-disconnect path; cli.TestWorkspaceAgent/DisabledServers runs with verbose since its debug messages are otherwise recorded. Golden --help output regenerated.

Default log verbosity change

This changes the default agent-log verbosity. Today coder-agent.log receives Debug unconditionally; with this change, normal operation logs Info, and recorded below-level entries are emitted only when the agent loses its coderd connection (via a manual flush) or before any Error+ log, or always under -v / CODER_VERBOSE. This is the intended behavior of the feature: a quiet steady state with full detail retained for diagnosing failures. Operators who want the previous always-debug output can run with -v / CODER_VERBOSE.

Alignment with #29752 (CLI)

This mirrors the CLI-4 (CLI log buffer) design so the two implementations agree:

  • Level model: Info by default, debug available on demand via the global -v / CODER_VERBOSE.
  • Flight recorder size: default 1000, 0 disables. CLI-4 exposes a global --flight-recorder-size; the agent uses a distinct --agent-flight-recorder-size / CODER_AGENT_FLIGHT_RECORDER_SIZE (the CODER_AGENT_* env convention) so the inherited global and the agent flag coexist without collision.
  • Flush trigger: the recorded history is emitted before any Error+ log via slog's auto-flush. The agent additionally flushes manually on a lost coderd connection, which it logs at Warn (reconnectable, and Error would fail slogtest-based tests); the CLI's failures are fatal, so it relies on the auto-flush-on-error.
Rollout context

Cross-repo sequence: coder/slog flight recorder (released as v3.2.0) -> go.mod bump in coder/coder (#29600, merged) -> this PR (CLIENT-672, agent) and CLI-4 (CLI), which are independent consumers and can merge in any order.


Opened by Coder Agents on behalf of @aqandrew.

Run the workspace agent logger at Info with a slog flight recorder instead
of at Debug. Debug entries are kept in a rolling in-memory ring and flushed
to the sinks when the agent loses its connection to coderd (io.EOF
disconnect or other run error), so the detail leading up to a failure is
available without logging debug during normal operation.

Implements CLIENT-672 (RFC req 13) for the workspace agent.
@linear-code

linear-code Bot commented Sep 21, 2026

Copy link
Copy Markdown

CLIENT-672

…cape hatch

Add --log-buffer-size (CODER_AGENT_LOG_BUFFER_SIZE, default 1000, 0 to
disable) so the flight recorder size is configurable per RFC req 13, and
--verbose (CODER_AGENT_VERBOSE) which runs the agent logger at Debug and
bypasses the recorder, restoring the previous always-debug file behavior
for anyone who wants it.

This mirrors the CLI-4 model (Info by default, debug available on demand)
so the agent and CLI log-buffer implementations agree. Update the
DisabledServers test, whose debug messages now require --verbose to reach
the log file.
Add TestAgent_FlushLogBufferOnDisconnect, which runs the agent at Info with
a flight recorder and a fake client that fails ConnectRPC with io.EOF, then
asserts a buffered debug entry is emitted once runLoop takes the disconnect
path.
Comment thread agent/agent.go Resolved
Comment thread agent/agent.go Resolved
Comment thread cli/testdata/coder_agent_--help.golden Resolved
Comment thread cli/testdata/coder_agent_--help.golden Resolved
Comment thread cli/agent.go Resolved
@spikecurtis
spikecurtis removed their request for review September 28, 2026 11:28
Log the io.EOF disconnect and the generic run error at Error instead of
Info/Warn. The flight recorder auto-flushes on Error, so the manual
logger.Flush calls are removed as redundant.

Addresses review feedback on #29671.
Make workspaceAgent a RootCmd method so it reads r.verbose from the root
command instead of defining a duplicate agent-only --verbose flag. The
global -v / CODER_VERBOSE now controls agent debug logging.

Addresses review feedback on #29671.
Compute the level (Info, or Debug with -v) and always attach the flight
recorder, letting it buffer whatever is below the level rather than
conditionally enabling it. At Debug it has nothing to buffer, and this keeps
working unchanged if a log-level flag is added later.

Addresses review feedback on #29671.
Logging the disconnect and run errors at Error made slogtest-based tests
fatal, since slogtest fails on any Error log and these fire on unexpected,
reconnectable disconnects. Log both at Warn and flush the flight recorder
manually so the buffered debug history is still emitted.
@aqandrew
aqandrew requested a review from code-asher September 29, 2026 17:28
Comment thread cli/agent.go Resolved
Comment thread cli/agent.go Resolved
@aqandrew
aqandrew merged commit 6487425 into main Sep 29, 2026
33 checks passed
@aqandrew
aqandrew deleted the client-672-agent-log-buffer branch September 29, 2026 21:24
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 29, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants