Repository navigation
docs: document Agent Relay reaper startup grace and reap delay - #31835
coder[bot] wants to merge 4 commits into
Conversation
2014a44 to
649a0c3
Compare
matifali
left a comment
There was a problem hiding this comment.
This is good, but I am struggling to understand the concept of Reaping in general and how these mnaynoptiosn help me as a platform admin.
It's not clear what to use, what not to use, and how they differ.
| ### `--reap-startup-grace` | ||
|
|
||
| | Property | Value | | ||
| |-------------|----------------------------------------------------| | ||
| | Type | <code>duration</code> | | ||
| | Environment | <code>$CODER_AGENT_RELAY_REAP_STARTUP_GRACE</code> | | ||
| | YAML | <code>reapStartupGrace</code> | | ||
| | Helm | <code>reapStartupGrace</code> | | ||
| | Default | <code>15m0s</code> | | ||
|
|
||
| How long the reaper waits for a workspace to build and start its worker, measured from when the latest build was created. | ||
| Must be positive. | ||
|
|
||
| - If a build is still pending or starting after the longer of this grace and the pool's `dispatch_deadline`, the reaper deletes the workspace. | ||
| - A workspace where the build finished but the worker still reports no status, or `pending`, fails after this grace. | ||
| The grace includes the time spent building. | ||
| - A short grace can fail workspaces where the agent connects slowly or the worker starts slowly. | ||
| Unless you run an isolated test deployment, keep the default. | ||
|
|
||
| The reaper sweeps about every 30 seconds, so cleanup isn't exact to the second. | ||
| Requires Agent Relay v0.15.0 or later. | ||
| Earlier versions use a fixed 15 minutes. | ||
|
|
||
| ### `--reap-delay` |
There was a problem hiding this comment.
It's a bit hard for me as a new reader to understand the difference between these two options.
There was a problem hiding this comment.
Assuming that this comment is re: --reap-delay and --reap-startup-grace, is it helpful to foreground that the first only matters for failed or canceled builds? Here, the word "completes" (and throughout Agent Relay) is complicated. "Completes" does not indicate success, but the word could confer that.
If this isn't the distinction you're referring to @matifali, the comment is void.
There was a problem hiding this comment.
@matifali Addressed in 3e869c0.
Workspace reaper now opens by contrasting the two options: --reap-startup-grace covers a workspace while it starts, and --reap-delay covers the time after the reaper decides to delete it.
Each option also leads with what it's for, and Choose cleanup settings in setup.md now says which reaper options you don't need for those goals.
@dluman Agreed on "completes".
The delay now says it counts from when the latest build finished, whether that build succeeded, failed, or was canceled, and not from when the worker exits.
It isn't limited to failed or canceled builds, though: it also holds workspaces after clean exits and failed workers, and builds that fail during dispatch skip it (reapDeadline in workspacereaper/reaper.go).
Generated by Coder Agents.
649a0c3 to
3b789ff
Compare
| The provider module publishes the worker's state through the `agent_relay_status` metadata item. | ||
| Agent Relay's reaper reads it on each sweep: | ||
| Agent Relay cleans up the workspaces it creates with a background process called the reaper. | ||
| About every 30 seconds, the reaper reads each relay workspace's `agent_relay_status` metadata item, which the provider module publishes, and acts on it: |
There was a problem hiding this comment.
The phrasing "[a]bout every 30 seconds" might be more definite if we somehow work in that:
- the reaper waits 30 seconds between sweeps
- a sweep takes longer with more pools and workspaces
- so, the cleanup can trail a threshold by 30 seconds or more
Maybe not all of those facts, but enough to establish some more precision around the window and why longer sweep times might occur.
There was a problem hiding this comment.
Reworded in 3e869c0 along those lines.
The reaper sleeps 30 seconds after each sweep rather than running on a fixed ticker, and a sweep checks each pool in turn (sweepLoop in servicereaper/reaper.go).
The docs now say cleanup can happen 30 seconds or more after a threshold, and the other pages link here instead of restating the timing.
Generated by Coder Agents.
| | A pool isn't serving | `/readyz` or `agent_relay_pool_healthy`, then the startup logs for the pool. A ready pool logs `template verified, polling` for Claude Code or `pool verified, serving` for Cursor. A pool that fails a check, such as a template without the module's parameters, logs `pool failed verification, retrying`. | | ||
| | A session never creates a workspace | Logs for the `session_id`, or the `session_uuid` for Claude Code. Look for identity resolution failures, a degraded pool, a dispatch error, or `job_error` on a failed build. | | ||
| | The workspace exists but the agent never starts | The workspace's build log, then the module logs in the workspace, then the `agent_relay_status` metadata on the workspace page in Coder. | | ||
| | A session ended but its workspace remains | Reaper decisions in the logs for the `workspace_id`. Agent Relay deletes on a sweep interval, and doesn't delete stopped workspaces. With a reap delay, Agent Relay keeps the workspace until the delay passes after its latest build completed. Each sweep that holds the workspace counts a `delayed` verdict. | |
There was a problem hiding this comment.
Should we describe what delayed is here in more detail? It looks like a 5th verdict, but I understand it to just be a kind of limbo between delete and failed. I realize this comment might also apply above.
There was a problem hiding this comment.
Done in 3e869c0.
delayed isn't a fifth decision.
The reaper counts it on each sweep where it has already decided delete or failed but the reap delay hasn't passed, so it comes before one of those two rather than between them.
This row now says that and names the debug log line, workspace awaiting reap with reap_at, and the --reap-delay bullet in configuration.md says the same.
Generated by Coder Agents.
| | `agent_relay_dispatcher_dispatches_total{result}` | Dispatch outcomes: `created`, `rebuilt`, `skipped_serving`, `skipped_deleting`, `expired`, or `failed`. | | ||
| | `agent_relay_dispatcher_claim_to_workspace_built_seconds` | Time from claim to a successful build. Compare it with the pool's build budget or deadline. | | ||
| | `agent_relay_reaper_workspaces{status}` | Workspaces by worker status, updated each sweep. Active sessions are `working` plus `serving`. | | ||
| | `agent_relay_reaper_verdicts_total{verdict}` | Reaper decisions: `keep`, `delete`, `failed`, `stuck`, and `delayed`. `stuck` means three delete builds failed in a row and the workspace needs manual cleanup. `delayed` counts sweeps that held a workspace for its reap delay, not distinct workspaces. | |
There was a problem hiding this comment.
delayed is a valid value here, but isn't as persistent as the others. It's a mystery state between delete and failed, if I remember right; should that be the take-away here?
There was a problem hiding this comment.
Yes, done in 3e869c0.
The row now says delayed means the reaper decided on delete or failed and is waiting for the reap delay, that it counts once per sweep, and that the workspace counts as delete or failed when the delay passes.
Generated by Coder Agents.
dluman
left a comment
There was a problem hiding this comment.
While I suggested a couple of things around delayed status, I'm not sure I have any really good actionable feedback beyond making sure that delayed is appropriately defined. A workspace doesn't stay in that lane, if I understand the docs correctly, and will eventually turn over into failed or delete.
This could be inconsequential, but it might make the wording of the sections involving delay clearer?
Agent Relay v0.15.0 replaced the fixed 300-second Claude Code spawn budget with a per-pool `dispatch_deadline` ([agent-relay#147](coder/agent-relay#147)). - `claude-code.md`: rewrite **Build budget** (default `5m`, `10s` to `15m`, whole seconds, can't be turned off, sent to Anthropic as the hint TTL, needs v0.15.0) and add the field to both pool examples. - `configuration.md`: add `dispatch_deadline` to the Claude Code pool fields. - `troubleshooting.md`, `monitoring/siem.md`: drop the fixed 300-second wording. Verification notes: - Checked against `internal/services/serviceclaudecoderunners` (`parseDispatchDeadline`, `poller.go` constants) at v0.15.0 (`9b96578`). - The old text said Agent Relay returns a missed-budget session as **retryable**. In v0.15.0, Claude requests use `TerminalOnDeadline` ([agent-relay#138](coder/agent-relay#138)), so a failure at or after the deadline is nacked non-retryable. The page now says Agent Relay "reports the failure" and doesn't claim either way. Stack 1/5. Review and merge in order: - #31834 - #31835 - #31836 - #31837 - #31838 Refs DOCS-1171 > Generated by Coder Agents. --------- Co-authored-by: coder[bot] <137810501+coder[bot]@users.noreply.github.com> Co-authored-by: Nick Vigilante <nick.vigilante@coder.com>
Agent Relay v0.15.0 makes the reaper's 15-minute startup window configurable (coder/agent-relay#152) and adds a reap delay (coder/agent-relay#134).
…ttings Atif found the reaper options hard to tell apart. Describe what the reaper does in Workspace lifecycle, add a table of which cleanup setting to change for which goal, and say when to use each option. Drop the per-option version notes in favor of the latest release.
3b789ff to
c4d5642
Compare
Review feedback on #31835 found the reaper docs hard to tell apart and the reap delay wording ambiguous. The delay counts from the latest build's completion, not from the worker's exit, so the earlier wording promised retention for long sessions that the reaper doesn't give. Contrast the startup grace and the reap delay at the top of the reaper reference, state when the delay starts and which outcomes it holds, define the delayed verdict as a wait before delete or failed, and describe the 30-second gap between sweeps exactly. Refs DOCS-1171
|
Closing and reopening under a human author, as required for this repository. |
Agent Relay v0.15.0 made the reaper's 15-minute startup window configurable (agent-relay#152) and added a reap delay (agent-relay#134).
configuration.md: add--reap-startup-grace(HelmreapStartupGrace) and--reap-delay, plus the per-poolreap_delayfield, under an intro that says how the two options differ. The Cursordispatch_deadline: 0row now points at the startup grace.setup.md: fix the script-ordering note and Workspace lifecycle, which hardcoded 15 minutes and said failed builds are deleted "on the next sweep". Workspace lifecycle now states the 30-second gap between sweeps, and Choose cleanup settings maps each goal to its setting.cursor.md: fix the Build deadline bullet and thependinglifecycle row.troubleshooting.md: the "session ended but its workspace remains" row mentions the reap delay.monitoring/index.md: add thedelayedverdict toagent_relay_reaper_verdicts_totaland define it as a wait beforedeleteorfailed.Verification notes:
reapDelayvalue at v0.15.0 (onlyreapStartupGrace; checkedvalues.yamlandtemplates/_helpers.tpl), so the docs say to setCODER_AGENT_RELAY_REAP_DELAYthroughextraEnv.max(grace, dispatch_deadline). A finished build whose runner reportspendingor nothing fails after the grace alone (workspacereaper/lifecycle.go).Job.CompletedAt, not from the worker's exit, so a session that ran longer than the delay gets no extra time (reapDeadlineinworkspacereaper/reaper.go).delayedis only a metric label. A sweep records it when the verdict isdeleteorfailedbut the delay hasn't passed. TheVerdicttype has onlykeep,delete,failed, andstuck.sweepLoopinservicereaper/reaper.go).Stack 2/5. Review and merge in order:
Refs DOCS-1171