Repository navigation
feat: add idempotent start tokens to the workspace agent process API - #27102
ibetitsmike wants to merge 30 commits into
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8cf446d6b8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
8cf446d to
26369ad
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 26369ad600
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
Codex Review: Didn't find any major issues. Already looking forward to the next diff. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
437fcd2 to
8e72ddd
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8e72ddd320
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2916ac17d4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
2916ac1 to
0dcbd7e
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0dcbd7eb97
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
0dcbd7e to
62ca73b
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 62ca73b8a8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
Codex Review: Didn't find any major issues. What shall we delve into next? Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
d6c273e to
11deac3
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 11deac3123
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…robeable interrupts unknown
…kground processes found by token
…d double-unescaping
…tead of committing results A transport failure on StartProcess or a token probe, a token conflict, and a found-but-unrecorded adoption all leave the dispatch recoverable through the durable execution token. Committing a tool result in those cases stamps result_committed_at and permanently ends same-token recovery, so a retried call would use a fresh token and could duplicate a process the failed call could not observe. These paths now abort the attempt uncommitted; structured agent start failures (other than the token conflict) released the reservation, so they resolve no_effect and commit normally.
…inst concurrent process records The unverifiable-dispatch paths in stale-claim recovery (wrong-agent dispatch target, probe route unavailable, untrustworthy absent token) now resolve unknown through the guarded starting-only transition: a claim owner's RecordStart landing concurrently wins, and the attempt resumes the recorded process instead of downgrading it to a permanent unknown.
…nterrupt commit A background row only stays cancel_requested when its handle was missing at commit time, so its committed synthetic result carries no background_process_id. Sparing such a process as detached (via the token probe or a late handle write) leaves it running with no addressable ID. Kill it like foreground work.
…mmitted A transient Recorder.Claim failure during stale-claim recovery must not commit a tool result: result_committed_at would stop later retries from recovering the token even though the original dispatch may still publish a process. Abort the batch like the initial claim path.
…through the token trust window With token recovery available, resolving a stale starting row to unknown after only claimStaleAfter forfeits adopting a live process once the agent is reachable again. Abort dial failures while the claim is within TokenTrustWindow; past the window absence answers prove nothing anyway, so the guarded unknown write unbricks the chat. Also align the sweep doc with killing background processes found by token.
…ng absent tokens An interrupt can land after the ledger claim but before the already-sent StartProcess reaches the agent: the probe then sees no reservation, and trusting that absence resolved the row canceled while the arriving dispatch could still launch the process with no ledger handle. Re-probe absent tokens until the record grace window since the claim closes, like pending reservations. Also route the no-probe fallback's unknown write through the guarded no-process outcome so a handle landing after the last identity poll wins and gets the kill flow.
…al claim time A same-token redispatch that attaches to the original dispatch's process recorded startedAt as the attach time, granting the process a fresh timeout on every retry and underreporting wall duration. The takeover also stamps a fresh claimed_at, so the original claim time is preserved on the dispatched record and attached starts resume through the re-attach flow: past-deadline processes detach with the graceful timed-out result instead of waiting a full timeout again.
A start waiting behind a pending reservation for its client token could time out and surface as a structured 500, which chatd commits as no_effect even though the reservation owner may still publish a process under the token. Route waiter aborts (context expiry, or manager close with the token owned) through a sentinel mapped to 409, which chatd already aborts uncommitted, keeping same-token recovery available.
… exit and chat-scope pending tokens The agent token index keeps exited processes until reaping, and signaling one answers 409, which killAndConfirm treated as a transient failure: a fast command that exited before the probe stayed cancel_requested with every sweep re-signaling it. A 409 is a definitive exit, so the row resolves canceled. Pending token reservations now carry the reserving chat's ID and byToken hides them from other chats, closing the window where a probe could learn about another chat's in-flight start before the published process became hidden.
…om test seeds The base branch removed the test-only Command field from ExecutionRecord; these seeds still set it.
81f3737 to
e817be7
Compare
|
Superseded by the recomposed CODAGT-757 stack: #27317, #27318, #27319, #27320, #27321, #27322, #27323, #27324, #27325. The combined final tree of the new stack is byte-identical to the reviewed heads of this stack (minus an accidental workflow orchestration file); the recomposition splits the work into reviewable pieces.
|
Problem
PR 1 of this stack (#27100) makes retried execute tool calls re-attach through a durable Postgres ledger. One hole remains: a
StartProcessrequest whose response is lost in transit (or whose attempt dies between dispatch and recording the process ID) leaves astartingclaim with no recorded process. Without agent cooperation, coderd cannot tell whether that process exists, so the safest resolution wasunknown.What this PR changes
This PR closes that hole with an agent-side idempotency layer keyed by the ledger execution UUID.
Agent (
agent/agentproc)StartProcessRequestgains aclient_token; coderd sets it to the ledger execution UUID on every execute start. The token is opaque to the agent.attached: true.GET /api/v0/processes/tokens?token=...returnsfound,pending,process_id, andtoken_index_age_ms.pendingdistinguishes a token reservation whose process ID is not yet published from a token the agent has never seen; a route-level 404 identifies an older agent without the endpoint. The index age exists because the token index is in-memory and does not survive agent restarts: an absent token is only trustworthy when the index predates the claim.Coderd stale-claim recovery (
coderd/x/chatd/chattool)A stale
startingclaim (previously resolved asunknown) is now recovered through the token probe. The claim CAS records its dispatch target (workspace_agent_id) before dispatching, and recovery only trusts the token index of that exact agent: when the chat has rebound to a different agent (or the row predates target recording), the row resolves tounknownwithout probing, since another agent's index cannot observe what the original target did with the dispatch. On the matching agent:found: adopt the process onto the existing claim epoch viaRecordStart, then re-attach. If the adoption write fails, the attempt returns a retryable error and leaves the rowstartinginstead of re-attaching, so a handle-less row is never terminalized; the durable token lets the next attempt probe and adopt again.pending: take over the stale claim and re-dispatch with the same token; the agent dedups against the in-flight reservation.unknown.Start transport failures likewise leave the row
startinginstead ofunknown, so a later attempt can resolve the dispatch through the token index or safely re-dispatch.Recovery never mints a new token for a tool call.
Interrupt reconciliation upgrade (
coderd/x/chatd/tasks.go)An interrupted
cancel_requestedrow without recorded process identity (previously resolved asunknown) now probes the token index of the dispatch target recorded at claim time (never the chat's current agent): a found process is adopted onto the row and goes through the normal kill-and-confirm flow (background rows included: their committed synthetic result carries no handle, so sparing them would strand a running process);pendingis re-probed until the dispatch window closes, so a process appearing moments after the interrupt is still killed; a trustworthy absent token resolves tocanceled; transport failures preservecancel_requested. When the token index cannot help (untrustworthy absence, old agents whose probe route 404s, or no recorded dispatch target), the reconciler waits out the record grace window for a lateRecordStarthandle write before resolvingunknown, killing the process if the handle lands. Both consumers share theTrustAbsentTokenrule.The periodic sweep from PR 1 now also claims
cancel_requestedrows without recorded process identity and routes them through the same token probe, so a post-interrupt dial failure is retried on later sweeps instead of stranding the row (and a possibly-running process) forever.Recovered processes are adopted with
started_atanchored at the stale claim's time (a lower bound of the true start), so a long-running recovered command cannot win a fresh full timeout.codersdk/workspacesdkgains the probe as an optionalProcessTokenProbercapability (kept off theAgentConncontract so out-of-tree implementations keep compiling) with aProbeProcessTokenhelper that degrades a missing implementation toErrProcessTokenProbeUnsupported, treated by callers like a pre-probe agent.What this PR deliberately does not change
Test plan
go test ./agent/agentproc/ ./codersdk/workspacesdk/ ./coderd/x/chatd/... -count=1: all pass locally.unknown, old-agent 404 tounknown, transport-error retryability, agent-rebind mismatch tounknown.TestInterruptProbesUnrecordedExecutecovers the interrupt probe outcomes.make fmtandmake lintclean; commits ran through the pre-commit hook.Stack
PR 2 of 3 for CODAGT-757: builds on #27100 (execution ledger); #27103 (attempt watchdog) builds on this branch.