Skip to content

docs: document Agent Relay reaper startup grace and reap delay - #31835

Closed
coder[bot] wants to merge 4 commits into
mainfrom
docs/agent-relay-reaper-timing
Closed

coder[bot] wants to merge 4 commits into
mainfrom
docs/agent-relay-reaper-timing

Conversation

@coder

@coder coder Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Agent Relay v0.15.0 made the reaper's 15-minute startup window configurable (agent-relay#152) and added a reap delay (agent-relay#134).

  • configuration.md: add --reap-startup-grace (Helm reapStartupGrace) and --reap-delay, plus the per-pool reap_delay field, under an intro that says how the two options differ. The Cursor dispatch_deadline: 0 row now points at the startup grace.
  • setup.md: fix the script-ordering note and Workspace lifecycle, which hardcoded 15 minutes and said failed builds are deleted "on the next sweep". Workspace lifecycle now states the 30-second gap between sweeps, and Choose cleanup settings maps each goal to its setting.
  • cursor.md: fix the Build deadline bullet and the pending lifecycle row.
  • troubleshooting.md: the "session ended but its workspace remains" row mentions the reap delay.
  • monitoring/index.md: add the delayed verdict to agent_relay_reaper_verdicts_total and define it as a wait before delete or failed.

Verification notes:

  • The chart has no reapDelay value at v0.15.0 (only reapStartupGrace; checked values.yaml and templates/_helpers.tpl), so the docs say to set CODER_AGENT_RELAY_REAP_DELAY through extraEnv.
  • Pending or starting builds wait max(grace, dispatch_deadline). A finished build whose runner reports pending or nothing fails after the grace alone (workspacereaper/lifecycle.go).
  • The reap delay counts from the latest build's Job.CompletedAt, not from the worker's exit, so a session that ran longer than the delay gets no extra time (reapDeadline in workspacereaper/reaper.go).
  • delayed is only a metric label. A sweep records it when the verdict is delete or failed but the delay hasn't passed. The Verdict type has only keep, delete, failed, and stuck.
  • The reaper sleeps 30 seconds after it sweeps every pool, rather than running on a fixed ticker (sweepLoop in servicereaper/reaper.go).

Stack 2/5. Review and merge in order:

Refs DOCS-1171

Generated by Coder Agents.

@linear-code

linear-code Bot commented Nov 6, 2026

Copy link
Copy Markdown

DOCS-1171

Comment thread docs/agent-relay/configuration.md Resolved
@nickvigilante
nickvigilante marked this pull request as ready for review November 6, 2026 18:22
@coder
coder Bot force-pushed the docs/agent-relay-reaper-timing branch from 2014a44 to 649a0c3 Compare November 6, 2026 18:29

@matifali matifali left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is good, but I am struggling to understand the concept of Reaping in general and how these mnaynoptiosn help me as a platform admin.

It's not clear what to use, what not to use, and how they differ.

Comment on lines +282 to +305
### `--reap-startup-grace`

| Property | Value |
|-------------|----------------------------------------------------|
| Type | <code>duration</code> |
| Environment | <code>$CODER_AGENT_RELAY_REAP_STARTUP_GRACE</code> |
| YAML | <code>reapStartupGrace</code> |
| Helm | <code>reapStartupGrace</code> |
| Default | <code>15m0s</code> |

How long the reaper waits for a workspace to build and start its worker, measured from when the latest build was created.
Must be positive.

- If a build is still pending or starting after the longer of this grace and the pool's `dispatch_deadline`, the reaper deletes the workspace.
- A workspace where the build finished but the worker still reports no status, or `pending`, fails after this grace.
The grace includes the time spent building.
- A short grace can fail workspaces where the agent connects slowly or the worker starts slowly.
Unless you run an isolated test deployment, keep the default.

The reaper sweeps about every 30&nbsp;seconds, so cleanup isn't exact to the second.
Requires Agent Relay v0.15.0 or later.
Earlier versions use a fixed 15&nbsp;minutes.

### `--reap-delay`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a bit hard for me as a new reader to understand the difference between these two options.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming that this comment is re: --reap-delay and --reap-startup-grace, is it helpful to foreground that the first only matters for failed or canceled builds? Here, the word "completes" (and throughout Agent Relay) is complicated. "Completes" does not indicate success, but the word could confer that.

If this isn't the distinction you're referring to @matifali, the comment is void.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@matifali Addressed in 3e869c0.
Workspace reaper now opens by contrasting the two options: --reap-startup-grace covers a workspace while it starts, and --reap-delay covers the time after the reaper decides to delete it.
Each option also leads with what it's for, and Choose cleanup settings in setup.md now says which reaper options you don't need for those goals.

@dluman Agreed on "completes".
The delay now says it counts from when the latest build finished, whether that build succeeded, failed, or was canceled, and not from when the worker exits.
It isn't limited to failed or canceled builds, though: it also holds workspaces after clean exits and failed workers, and builds that fail during dispatch skip it (reapDeadline in workspacereaper/reaper.go).

Generated by Coder Agents.

@nickvigilante
nickvigilante force-pushed the docs/agent-relay-reaper-timing branch from 649a0c3 to 3b789ff Compare November 6, 2026 19:59
Comment thread docs/agent-relay/setup.md Outdated
The provider module publishes the worker's state through the `agent_relay_status` metadata item.
Agent Relay's reaper reads it on each sweep:
Agent Relay cleans up the workspaces it creates with a background process called the reaper.
About every 30&nbsp;seconds, the reaper reads each relay workspace's `agent_relay_status` metadata item, which the provider module publishes, and acts on it:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The phrasing "[a]bout every 30 seconds" might be more definite if we somehow work in that:

  • the reaper waits 30 seconds between sweeps
  • a sweep takes longer with more pools and workspaces
  • so, the cleanup can trail a threshold by 30 seconds or more

Maybe not all of those facts, but enough to establish some more precision around the window and why longer sweep times might occur.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reworded in 3e869c0 along those lines.
The reaper sleeps 30 seconds after each sweep rather than running on a fixed ticker, and a sweep checks each pool in turn (sweepLoop in servicereaper/reaper.go).
The docs now say cleanup can happen 30 seconds or more after a threshold, and the other pages link here instead of restating the timing.

Generated by Coder Agents.

Comment thread docs/agent-relay/troubleshooting.md Outdated
| A pool isn't serving | `/readyz` or `agent_relay_pool_healthy`, then the startup logs for the pool. A ready pool logs `template verified, polling` for Claude Code or `pool verified, serving` for Cursor. A pool that fails a check, such as a template without the module's parameters, logs `pool failed verification, retrying`. |
| A session never creates a workspace | Logs for the `session_id`, or the `session_uuid` for Claude Code. Look for identity resolution failures, a degraded pool, a dispatch error, or `job_error` on a failed build. |
| The workspace exists but the agent never starts | The workspace's build log, then the module logs in the workspace, then the `agent_relay_status` metadata on the workspace page in Coder. |
| A session ended but its workspace remains | Reaper decisions in the logs for the `workspace_id`. Agent Relay deletes on a sweep interval, and doesn't delete stopped workspaces. With a reap delay, Agent Relay keeps the workspace until the delay passes after its latest build completed. Each sweep that holds the workspace counts a `delayed` verdict. |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we describe what delayed is here in more detail? It looks like a 5th verdict, but I understand it to just be a kind of limbo between delete and failed. I realize this comment might also apply above.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 3e869c0.
delayed isn't a fifth decision.
The reaper counts it on each sweep where it has already decided delete or failed but the reap delay hasn't passed, so it comes before one of those two rather than between them.
This row now says that and names the debug log line, workspace awaiting reap with reap_at, and the --reap-delay bullet in configuration.md says the same.

Generated by Coder Agents.

Comment thread docs/agent-relay/monitoring/index.md Outdated
| `agent_relay_dispatcher_dispatches_total{result}` | Dispatch outcomes: `created`, `rebuilt`, `skipped_serving`, `skipped_deleting`, `expired`, or `failed`. |
| `agent_relay_dispatcher_claim_to_workspace_built_seconds` | Time from claim to a successful build. Compare it with the pool's build budget or deadline. |
| `agent_relay_reaper_workspaces{status}` | Workspaces by worker status, updated each sweep. Active sessions are `working` plus `serving`. |
| `agent_relay_reaper_verdicts_total{verdict}` | Reaper decisions: `keep`, `delete`, `failed`, `stuck`, and `delayed`. `stuck` means three delete builds failed in a row and the workspace needs manual cleanup. `delayed` counts sweeps that held a workspace for its reap delay, not distinct workspaces. |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

delayed is a valid value here, but isn't as persistent as the others. It's a mystery state between delete and failed, if I remember right; should that be the take-away here?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, done in 3e869c0.
The row now says delayed means the reaper decided on delete or failed and is waiting for the reap delay, that it counts once per sweep, and that the workspace counts as delete or failed when the delay passes.

Generated by Coder Agents.

@dluman dluman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I suggested a couple of things around delayed status, I'm not sure I have any really good actionable feedback beyond making sure that delayed is appropriately defined. A workspace doesn't stay in that lane, if I understand the docs correctly, and will eventually turn over into failed or delete.

This could be inconsequential, but it might make the wording of the sections involving delay clearer?

Base automatically changed from docs/agent-relay-claude-dispatch-deadline to main November 9, 2026 19:09
nickvigilante added a commit that referenced this pull request Nov 9, 2026
Agent Relay v0.15.0 replaced the fixed 300-second Claude Code spawn
budget with a per-pool `dispatch_deadline`
([agent-relay#147](coder/agent-relay#147)).

- `claude-code.md`: rewrite **Build budget** (default `5m`, `10s` to
`15m`, whole seconds, can't be turned off, sent to Anthropic as the hint
TTL, needs v0.15.0) and add the field to both pool examples.
- `configuration.md`: add `dispatch_deadline` to the Claude Code pool
fields.
- `troubleshooting.md`, `monitoring/siem.md`: drop the fixed 300-second
wording.

Verification notes:
- Checked against `internal/services/serviceclaudecoderunners`
(`parseDispatchDeadline`, `poller.go` constants) at v0.15.0 (`9b96578`).
- The old text said Agent Relay returns a missed-budget session as
**retryable**. In v0.15.0, Claude requests use `TerminalOnDeadline`
([agent-relay#138](coder/agent-relay#138)), so a
failure at or after the deadline is nacked non-retryable. The page now
says Agent Relay "reports the failure" and doesn't claim either way.

Stack 1/5. Review and merge in order:
- #31834
- #31835
- #31836
- #31837
- #31838

Refs DOCS-1171

> Generated by Coder Agents.

---------

Co-authored-by: coder[bot] <137810501+coder[bot]@users.noreply.github.com>
Co-authored-by: Nick Vigilante <nick.vigilante@coder.com>
coder Bot and others added 3 commits November 9, 2026 14:09
Agent Relay v0.15.0 makes the reaper's 15-minute startup window configurable (coder/agent-relay#152) and adds a reap delay (coder/agent-relay#134).
…ttings

Atif found the reaper options hard to tell apart. Describe what the
reaper does in Workspace lifecycle, add a table of which cleanup setting
to change for which goal, and say when to use each option. Drop the
per-option version notes in favor of the latest release.
@nickvigilante
nickvigilante force-pushed the docs/agent-relay-reaper-timing branch from 3b789ff to c4d5642 Compare November 9, 2026 19:09
@nickvigilante
nickvigilante temporarily deployed to main November 9, 2026 19:09 — with GitHub Actions Inactive
Review feedback on #31835 found the reaper docs hard to tell apart and
the reap delay wording ambiguous. The delay counts from the latest
build's completion, not from the worker's exit, so the earlier wording
promised retention for long sessions that the reaper doesn't give.

Contrast the startup grace and the reap delay at the top of the
reaper reference, state when the delay starts and which outcomes it
holds, define the delayed verdict as a wait before delete or failed,
and describe the 30-second gap between sweeps exactly.

Refs DOCS-1171
@nickvigilante
nickvigilante temporarily deployed to main November 9, 2026 19:15 — with GitHub Actions Inactive
@nickvigilante
nickvigilante temporarily deployed to main November 9, 2026 19:16 — with GitHub Actions Inactive
@nickvigilante
nickvigilante requested a review from dluman November 9, 2026 19:41
@nickvigilante

Copy link
Copy Markdown
Collaborator

Closing and reopening under a human author, as required for this repository.

@nickvigilante
nickvigilante temporarily deployed to main November 10, 2026 15:53 — with GitHub Actions Inactive
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 10, 2026
@nickvigilante
nickvigilante temporarily deployed to main November 10, 2026 15:53 — with GitHub Actions Inactive
@nickvigilante
nickvigilante temporarily deployed to main November 10, 2026 15:53 — with GitHub Actions Inactive

This branch was previously deployed

1 inactive deployment
main — 3e869c02 Deployed Nov 10, 2026 by nickvigilante via cla #253454
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

community Pull Requests and issues created by the community.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants