Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 19 additions & 18 deletions docs/agent-relay/monitoring/siem.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,21 +75,22 @@ Claude Code adds `claude_tag_channel_user`, and Cursor adds `cursor_pool`, `reli

## Failure and rebuild fields

| Field | Meaning |
|------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------|
| `error` | Error detail on most `WARN` and `ERROR` records, such as a failed claim, identity lookup, build, or delete |
| `retryable` | `true` when the provider retries or the user can try again. `false` when an administrator must fix configuration or identity. |
| `message` | Cursor: the exact text shown to the user when Agent Relay fails the claim |
| `detail` | Cursor: the local reason for a failed claim. It's never sent to Cursor and is truncated at 512 characters. |
| `stage` | Cursor: the step where the claim failed, such as `owner` or `building` |
| `reason` | Reaper: why it deleted the workspace, such as `the runner exited with status 1`. Cursor refusals: `no_email`, `no_match`, `inactive`, or `dormant`. |
| `job_status` | The final provisioner job status of a failed build, such as `failed` or `canceled` |
| `job_error` | Coder's error summary for a failed build |
| `build_status` | The latest build's status when Agent Relay rebuilds a workspace or the reaper evaluates it |
| `deadline` | Claude Code: when Anthropic stops waiting for a runner. Logged when the deadline passes before dispatch. |
| `status` | Reaper: the runner status the workspace reported, such as `working`, `done 1`, `failed <reason>`, or `orphaned` |
| `age` | Reaper: time elapsed after the workspace's latest build was created |
| `failed_deletes` | Reaper: how many delete builds have failed in a row for the workspace |
| Field | Meaning |
|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `error` | Error detail on most `WARN` and `ERROR` records, such as a failed claim, identity lookup, build, or delete |
| `retryable` | `true` when the provider retries or the user can try again. `false` when an administrator must fix configuration or identity. |
| `message` | Cursor: the exact text shown to the user when Agent Relay fails the claim |
| `detail` | Cursor: the local reason for a failed claim. It's never sent to Cursor and is truncated at 512 characters. |
| `stage` | Cursor: the step where the claim failed, such as `owner` or `building` |
| `reason` | Reaper: why it deleted the workspace, such as `the runner exited with status 1`. Cursor refusals: `no_email`, `no_match`, `inactive`, or `dormant`. |
| `job_status` | The final provisioner job status of a failed build, such as `failed` or `canceled` |
| `job_error` | Coder's error summary for a failed build |
| `build_status` | The latest build's status when Agent Relay rebuilds a workspace or the reaper evaluates it |
| `deadline` | Claude Code: when Anthropic stops waiting for a runner. Logged when the deadline passes before dispatch. |
| `status` | Reaper: the runner status the workspace reported, such as `working`, `done 1`, `failed <reason>`, or `orphaned` |
| `agent_lifecycle` | The workspace agent's lifecycle state, such as `starting` or `ready`, on `workspace built, runner handoff` and reaper records. Empty when no agent is connected. `starting` means startup scripts are still running, and `start_error` or `start_timeout` means they failed. |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: This locates agent_lifecycle by quoting workspace built, runner handoff as the lookup key, but the page's own "Match rules" section a few headings down says msg text can change between Agent Relay releases and warns readers not to rely on it that way. Tying the field's provenance to an unstable msg string is evidence (which code path emits it), not the reader-facing fact of when the field is present (docs/.style/content-guidelines.md#evidence-justifies-a-claim-and-does-not-belong-in-the-claim). (Reader Advocate)

Separately, in the same cell, "start_error or start_timeout means they failed" has no clean antecedent for "they": the clause-2 subject (start_error/start_timeout) is singular and isn't itself the thing that "failed," so the reader has to reach back into the prior clause's object ("startup scripts") to resolve it (docs/.style/style-guide/voice-and-tone.md#give-each-pronoun-one-clear-referent). (Copy Editor)

Combining both fixes:

Suggested change
| `agent_lifecycle` | The workspace agent's lifecycle state, such as `starting` or `ready`, on `workspace built, runner handoff` and reaper records. Empty when no agent is connected. `starting` means startup scripts are still running, and `start_error` or `start_timeout` means they failed. |
| `agent_lifecycle` | The workspace agent's lifecycle state, such as `starting` or `ready`, present once a workspace finishes building and on every reaper record. Empty when no agent is connected. `starting` means startup scripts are still running, and `start_error` or `start_timeout` means the scripts failed. |

| `age` | Reaper: time elapsed after the workspace's latest build was created |
| `failed_deletes` | Reaper: how many delete builds have failed in a row for the workspace |

Claude Code nack records don't repeat the failure reason.
Find it in the `error` or `reason` field of the `WARN` record written just before the nack.
Expand Down Expand Up @@ -223,14 +224,14 @@ Some conditions, such as a stranded Cursor claim, appear only in logs.
| Page when sustained | `pool failed verification, retrying` | Both | `pool`, `template` | A bad API key or environment key, a template without the Agent Relay module, or an organization problem. Alert after several records in a row. |
| Rate threshold | `nacked spawn hint` | Claude Code | `pool`, `retryable` | Sessions handed back to Anthropic. `retryable: false` ends the session and needs an administrator. |
| Rate threshold | `failed the claim upstream, the user sees the message` | Cursor | `pool`, `retryable`, `stage` | Users saw a failure. `retryable: false` needs an administrator. `retryable: true` usually means a transient or capacity problem. |
| Rate threshold | `workspace build failed, reporting retryable and deleting` | Both | `job_error` | A template or infrastructure failure, such as exhausted quota. Agent Relay deletes the workspace right away, so `job_error` might be the only record of why. |
| Rate threshold | `workspace build passed the session deadline, reporting retryable and deleting` | Both | `pool` | Builds take longer than the pool's `dispatch_deadline`. |
| Rate threshold | `workspace build failed` | Both | `job_error` | A template or infrastructure failure, such as exhausted quota. Agent Relay deletes the workspace right away, so `job_error` might be the only record of why. |
| Rate threshold | `workspace build passed the session deadline, reporting failure` or `workspace build succeeded after the session deadline` | Both | `pool` | Builds take longer than the pool's `dispatch_deadline`. |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: This row now lists two different outcomes under one "Why": workspace build passed the session deadline, reporting failure (a failure) and the new workspace build succeeded after the session deadline (the build actually finished, just late). Two problems with that:

  1. docs/agent-relay/claude-code.md's "Build budget" section still says unconditionally that "When a build misses the budget, Agent Relay reports the failure to Anthropic, cancels the build if it's still running, and deletes the workspace," and docs/agent-relay/cursor.md's "Build deadline" section still says unconditionally that "When the deadline passes, Agent Relay deletes the workspace and the developer's chat shows agent-relay: workspace build exceeded the spawn budget." Neither page mentions the late-success case this new message implies exists; the old siem.md wording ("...reporting retryable and deleting") matched those pages, the new wording no longer does. (Continuity Editor)
  2. Even within this table, the shared "Why" column ("Builds take longer than the pool's dispatch_deadline.") doesn't say whether an alert firing on the second value needs the same urgent response as the first, or is only an informational, late-success signal. (Newcomer)

Fix either by adding a sentence to claude-code.md and cursor.md describing the late-success case, or by splitting this row so each msg value has its own "Why" text that says whether it means failure or late success.

| Rate threshold | `session did not finish cleanly, reporting and deleting` (`reaper`) | Both | `reason` | Runners that exited non-zero, couldn't start, stopped unexpectedly, or never started. |
| Ticket | `session owner is unresolvable, nacking non-retryable` | Claude Code | `org_service_name` | A Claude user has no active Coder account in the pool's organization, or a Claude Tag channel session arrived without `claude_tag_channel_user`. |
| Ticket | `request owner cannot be served, refusing the request` | Cursor | `reason` | A Cursor request has no email, no matching Coder account in the organization, or a suspended or dormant account. |
| Ticket | `work order failed verification, nacking non-retryable` | Claude Code | `pool` | Agent Relay rejected the Anthropic-signed work order. `error` gives the reason. |
| Ticket | `workspace delete keeps failing, leaving it for an operator` (`reaper`) | Both | `workspace_id` | The reaper stopped retrying after repeated delete failures. Clean up the workspace by hand. |
| Ticket | `delete workspace failed, leaving for reconciliation` or `delete workspace failed, retrying next sweep` | Both | `workspace_id` | A workspace might still be running and using resources. |
| Ticket | `delete workspace failed, leaving it for the reaper` or `delete workspace failed, retrying next sweep` | Both | `workspace_id` | A workspace might still be running and using resources. |
| Dashboard | `resolved session owner`, `claimed pending request`, `workspace build created`, `workspace built, runner handoff`, or `deleting workspace` | Both | `pool`, `client_platform` | The normal session lifecycle, for volume and funnel views. |

## Retention and privacy
Expand Down
Loading