Repository navigation
feat: buffer agent debug logs and flush on connection failure - #29671
Merged
Merged
Conversation
Run the workspace agent logger at Info with a slog flight recorder instead of at Debug. Debug entries are kept in a rolling in-memory ring and flushed to the sinks when the agent loses its connection to coderd (io.EOF disconnect or other run error), so the detail leading up to a failure is available without logging debug during normal operation. Implements CLIENT-672 (RFC req 13) for the workspace agent.
…cape hatch Add --log-buffer-size (CODER_AGENT_LOG_BUFFER_SIZE, default 1000, 0 to disable) so the flight recorder size is configurable per RFC req 13, and --verbose (CODER_AGENT_VERBOSE) which runs the agent logger at Debug and bypasses the recorder, restoring the previous always-debug file behavior for anyone who wants it. This mirrors the CLI-4 model (Info by default, debug available on demand) so the agent and CLI log-buffer implementations agree. Update the DisabledServers test, whose debug messages now require --verbose to reach the log file.
Add TestAgent_FlushLogBufferOnDisconnect, which runs the agent at Info with a flight recorder and a fake client that fails ConnectRPC with io.EOF, then asserts a buffered debug entry is emitted once runLoop takes the disconnect path.
aqandrew
marked this pull request as ready for review
September 24, 2026 00:01
code-asher
reviewed
Sep 28, 2026
spikecurtis
removed their request for review
September 28, 2026 11:28
Log the io.EOF disconnect and the generic run error at Error instead of Info/Warn. The flight recorder auto-flushes on Error, so the manual logger.Flush calls are removed as redundant. Addresses review feedback on #29671.
Make workspaceAgent a RootCmd method so it reads r.verbose from the root command instead of defining a duplicate agent-only --verbose flag. The global -v / CODER_VERBOSE now controls agent debug logging. Addresses review feedback on #29671.
Compute the level (Info, or Debug with -v) and always attach the flight recorder, letting it buffer whatever is below the level rather than conditionally enabling it. At Debug it has nothing to buffer, and this keeps working unchanged if a log-level flag is added later. Addresses review feedback on #29671.
Logging the disconnect and run errors at Error made slogtest-based tests fatal, since slogtest fails on any Error log and these fire on unexpected, reconnectable disconnects. Log both at Warn and flush the flight recorder manually so the buffered debug history is still emitted.
code-asher
approved these changes
Sep 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Runs the workspace agent logger at
Infowith a slog flight recorder (v3.2.0) instead of unconditionally atDebug. Entries below the current log level are kept in a rolling in-memory ring and flushed to the sinks when the agent loses its connection to coderd (and before anyError+ log), so the detail leading up to a failure is available without writing debug logs during normal operation.Implements CLIENT-672 (connection-log RFC req 13) for the workspace agent. Depends on the
cdr.dev/slog/v3v3.2.0bump (already onmain).How
cli/agent.go:AppendSinks(...).Leveled(level).FlightRecorder(flightRecorderSize), wherelevelisInfoby default andDebugunder the global-v/CODER_VERBOSE. The recorder is always attached; atDebugit is inert, which keeps the setup correct if the configured level changes in the future. The reaper logger is unchanged.--agent-flight-recorder-size/CODER_AGENT_FLIGHT_RECORDER_SIZE(default1000,0disables the flight recorder), so the ring size is configurable per RFC req 13. The agent flag is distinct from CLI-4's global--flight-recorder-size(feat: buffer CLI session logs and flush on connection failure #29752) so the two don't collide whencoder agentinherits the root global.workspaceAgentis aRootCmdmethod so it reuses the root command's global verbose flag instead of defining an agent-specific one.agent/agent.go(runLoop): on a lost coderd connection (theio.EOFdisconnect and the generic run error) the agent callslogger.Flush(ctx)to emit the recorded history and then logs atWarn. These areWarnrather thanErrorbecause reconnectable disconnects are expected andslogtestfails tests on anyErrorlog; the manual flush is what surfaces the recorded detail. Graceful shutdown paths (ctxcanceled, agent closed) are not flushed.agent.TestAgent_FlushFlightRecorderOnDisconnectcovers the flush-on-disconnect path;cli.TestWorkspaceAgent/DisabledServersruns with verbose since its debug messages are otherwise recorded. Golden--helpoutput regenerated.Default log verbosity change
This changes the default agent-log verbosity. Today
coder-agent.logreceivesDebugunconditionally; with this change, normal operation logsInfo, and recorded below-level entries are emitted only when the agent loses its coderd connection (via a manual flush) or before anyError+ log, or always under-v/CODER_VERBOSE. This is the intended behavior of the feature: a quiet steady state with full detail retained for diagnosing failures. Operators who want the previous always-debug output can run with-v/CODER_VERBOSE.Alignment with #29752 (CLI)
This mirrors the CLI-4 (CLI log buffer) design so the two implementations agree:
-v/CODER_VERBOSE.1000,0disables. CLI-4 exposes a global--flight-recorder-size; the agent uses a distinct--agent-flight-recorder-size/CODER_AGENT_FLIGHT_RECORDER_SIZE(theCODER_AGENT_*env convention) so the inherited global and the agent flag coexist without collision.Error+ log via slog's auto-flush. The agent additionally flushes manually on a lost coderd connection, which it logs atWarn(reconnectable, andErrorwould failslogtest-based tests); the CLI's failures are fatal, so it relies on the auto-flush-on-error.Rollout context
Cross-repo sequence: coder/slog flight recorder (released as v3.2.0) -> go.mod bump in coder/coder (#29600, merged) -> this PR (CLIENT-672, agent) and CLI-4 (CLI), which are independent consumers and can merge in any order.
Opened by Coder Agents on behalf of @aqandrew.