Skip to content

feat: add idempotent start tokens to the workspace agent process API - #27102

Closed
ibetitsmike wants to merge 30 commits into
mike/codagt-757-execution-recordsfrom
mike/codagt-757-agent-idempotency-token
Closed

ibetitsmike wants to merge 30 commits into
mike/codagt-757-execution-recordsfrom
mike/codagt-757-agent-idempotency-token

Conversation

@ibetitsmike

@ibetitsmike ibetitsmike commented Jul 8, 2026 •

Copy link
Copy Markdown
Collaborator

This PR was authored by Mux, an AI coding agent, working on behalf of Mike.

Problem

PR 1 of this stack (#27100) makes retried execute tool calls re-attach through a durable Postgres ledger. One hole remains: a StartProcess request whose response is lost in transit (or whose attempt dies between dispatch and recording the process ID) leaves a starting claim with no recorded process. Without agent cooperation, coderd cannot tell whether that process exists, so the safest resolution was unknown.

What this PR changes

This PR closes that hole with an agent-side idempotency layer keyed by the ledger execution UUID.

Agent (agent/agentproc)

  • StartProcessRequest gains a client_token; coderd sets it to the ledger execution UUID on every execute start. The token is opaque to the agent.
  • The process manager indexes processes by bare token. A reservation is inserted under the manager lock before spawning, so concurrent starts with the same token wait for the owner and attach instead of racing to spawn duplicates. A repeated token returns the existing process with attached: true.
  • A repeated token with a different command, workdir, background flag, environment, or chat ID is rejected with HTTP 409 (the IdempotentParameterMismatch analog). Keying by bare token means reusing a token from another chat now conflicts loudly instead of starting an isolated duplicate.
  • New probe endpoint GET /api/v0/processes/tokens?token=... returns found, pending, process_id, and token_index_age_ms. pending distinguishes a token reservation whose process ID is not yet published from a token the agent has never seen; a route-level 404 identifies an older agent without the endpoint. The index age exists because the token index is in-memory and does not survive agent restarts: an absent token is only trustworthy when the index predates the claim.
  • The exited-process reap age is raised from 5 to 60 minutes so a token keeps deduplicating across the realistic retry window; reaping frees token index entries and also runs on start so the index cannot grow unbounded.

Coderd stale-claim recovery (coderd/x/chatd/chattool)

A stale starting claim (previously resolved as unknown) is now recovered through the token probe. The claim CAS records its dispatch target (workspace_agent_id) before dispatching, and recovery only trusts the token index of that exact agent: when the chat has rebound to a different agent (or the row predates target recording), the row resolves to unknown without probing, since another agent's index cannot observe what the original target did with the dispatch. On the matching agent:

  • found: adopt the process onto the existing claim epoch via RecordStart, then re-attach. If the adoption write fails, the attempt returns a retryable error and leaves the row starting instead of re-attaching, so a handle-less row is never terminalized; the durable token lets the next attempt probe and adopt again.
  • pending: take over the stale claim and re-dispatch with the same token; the agent dedups against the in-flight reservation.
  • absent and trustworthy (claim within the 30 minute trust window AND the agent's token index predates the claim): take over and re-dispatch with the same token. The window stays well inside the 60 minute reap age, so a trustworthy "absent" reliably means "never started".
  • absent but untrustworthy (past the trust window, or the agent restarted with a younger token index), or an old agent without the probe route: mark unknown.
  • transport error: leave the row untouched and return a retryable error.

Start transport failures likewise leave the row starting instead of unknown, so a later attempt can resolve the dispatch through the token index or safely re-dispatch.

Recovery never mints a new token for a tool call.

Interrupt reconciliation upgrade (coderd/x/chatd/tasks.go)

An interrupted cancel_requested row without recorded process identity (previously resolved as unknown) now probes the token index of the dispatch target recorded at claim time (never the chat's current agent): a found process is adopted onto the row and goes through the normal kill-and-confirm flow (background rows included: their committed synthetic result carries no handle, so sparing them would strand a running process); pending is re-probed until the dispatch window closes, so a process appearing moments after the interrupt is still killed; a trustworthy absent token resolves to canceled; transport failures preserve cancel_requested. When the token index cannot help (untrustworthy absence, old agents whose probe route 404s, or no recorded dispatch target), the reconciler waits out the record grace window for a late RecordStart handle write before resolving unknown, killing the process if the handle lands. Both consumers share the TrustAbsentToken rule.

The periodic sweep from PR 1 now also claims cancel_requested rows without recorded process identity and routes them through the same token probe, so a post-interrupt dial failure is retried on later sweeps instead of stranding the row (and a possibly-running process) forever.

Recovered processes are adopted with started_at anchored at the stale claim's time (a lower bound of the true start), so a long-running recovered command cannot win a fresh full timeout.

  • codersdk/workspacesdk gains the probe as an optional ProcessTokenProber capability (kept off the AgentConn contract so out-of-tree implementations keep compiling) with a ProbeProcessToken helper that degrades a missing implementation to ErrProcessTokenProbeUnsupported, treated by callers like a pre-probe agent.

What this PR deliberately does not change

  • No database schema changes; the ledger from PR 1 is untouched and remains the source of truth when the agent cannot help.
  • Requests without a token behave exactly as before.
  • No dedup by command string; only the explicit token is used.

Test plan

  • go test ./agent/agentproc/ ./codersdk/workspacesdk/ ./coderd/x/chatd/... -count=1: all pass locally.
  • Agent tests: same-token attach, parameter and cross-chat 409s, concurrent same-token starts spawn exactly one process (race detector), token probe found/pending/absent, canceled waiters return promptly, reaping frees the token index.
  • Chattool tests: stale-claim probe adoption, pending takeover and re-dispatch, trust-window expiry to unknown, old-agent 404 to unknown, transport-error retryability, agent-rebind mismatch to unknown.
  • Chatd tests: TestInterruptProbesUnrecordedExecute covers the interrupt probe outcomes.
  • make fmt and make lint clean; commits ran through the pre-commit hook.

Stack

PR 2 of 3 for CODAGT-757: builds on #27100 (execution ledger); #27103 (attempt watchdog) builds on this branch.

@linear-code

linear-code Bot commented Jul 8, 2026

Copy link
Copy Markdown

CODAGT-757

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8cf446d6b8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread agent/agentproc/process.go Resolved
Comment thread agent/agentproc/process.go Resolved
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 26369ad600

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread agent/agentproc/process.go Resolved
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: ceb245fddd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-agent-idempotency-token branch 5 times, most recently from 437fcd2 to 8e72ddd Compare July 15, 2026 21:25
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8e72ddd320

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/chattool/execute.go Resolved
Comment thread coderd/x/chatd/chattool/execute.go Resolved
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2916ac17d4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread codersdk/workspacesdk/agentconn.go Resolved
@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-agent-idempotency-token branch from 2916ac1 to 0dcbd7e Compare July 15, 2026 22:11
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0dcbd7eb97

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/chattool/execute.go Resolved
@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-agent-idempotency-token branch from 0dcbd7e to 62ca73b Compare July 15, 2026 22:32
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 62ca73b8a8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/tasks.go Resolved
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. What shall we delve into next?

Reviewed commit: d6c273ecd7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike
ibetitsmike force-pushed the mike/codagt-757-agent-idempotency-token branch from d6c273e to 11deac3 Compare July 16, 2026 00:24
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 11deac3123

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/chattool/execute.go Resolved
…tead of committing results

A transport failure on StartProcess or a token probe, a token
conflict, and a found-but-unrecorded adoption all leave the dispatch
recoverable through the durable execution token. Committing a tool
result in those cases stamps result_committed_at and permanently ends
same-token recovery, so a retried call would use a fresh token and
could duplicate a process the failed call could not observe. These
paths now abort the attempt uncommitted; structured agent start
failures (other than the token conflict) released the reservation, so
they resolve no_effect and commit normally.
…inst concurrent process records

The unverifiable-dispatch paths in stale-claim recovery (wrong-agent
dispatch target, probe route unavailable, untrustworthy absent
token) now resolve unknown through the guarded starting-only
transition: a claim owner's RecordStart landing concurrently wins,
and the attempt resumes the recorded process instead of downgrading
it to a permanent unknown.
…nterrupt commit

A background row only stays cancel_requested when its handle was
missing at commit time, so its committed synthetic result carries no
background_process_id. Sparing such a process as detached (via the
token probe or a late handle write) leaves it running with no
addressable ID. Kill it like foreground work.
…mmitted

A transient Recorder.Claim failure during stale-claim recovery must
not commit a tool result: result_committed_at would stop later
retries from recovering the token even though the original dispatch
may still publish a process. Abort the batch like the initial claim
path.
…through the token trust window

With token recovery available, resolving a stale starting row to
unknown after only claimStaleAfter forfeits adopting a live process
once the agent is reachable again. Abort dial failures while the
claim is within TokenTrustWindow; past the window absence answers
prove nothing anyway, so the guarded unknown write unbricks the chat.
Also align the sweep doc with killing background processes found by
token.
…ng absent tokens

An interrupt can land after the ledger claim but before the
already-sent StartProcess reaches the agent: the probe then sees no
reservation, and trusting that absence resolved the row canceled
while the arriving dispatch could still launch the process with no
ledger handle. Re-probe absent tokens until the record grace window
since the claim closes, like pending reservations. Also route the
no-probe fallback's unknown write through the guarded no-process
outcome so a handle landing after the last identity poll wins and
gets the kill flow.
…al claim time

A same-token redispatch that attaches to the original dispatch's
process recorded startedAt as the attach time, granting the process
a fresh timeout on every retry and underreporting wall duration. The
takeover also stamps a fresh claimed_at, so the original claim time
is preserved on the dispatched record and attached starts resume
through the re-attach flow: past-deadline processes detach with the
graceful timed-out result instead of waiting a full timeout again.
A start waiting behind a pending reservation for its client token
could time out and surface as a structured 500, which chatd commits
as no_effect even though the reservation owner may still publish a
process under the token. Route waiter aborts (context expiry, or
manager close with the token owned) through a sentinel mapped to
409, which chatd already aborts uncommitted, keeping same-token
recovery available.
… exit and chat-scope pending tokens

The agent token index keeps exited processes until reaping, and
signaling one answers 409, which killAndConfirm treated as a
transient failure: a fast command that exited before the probe
stayed cancel_requested with every sweep re-signaling it. A 409 is a
definitive exit, so the row resolves canceled.

Pending token reservations now carry the reserving chat's ID and
byToken hides them from other chats, closing the window where a
probe could learn about another chat's in-flight start before the
published process became hidden.
…om test seeds

The base branch removed the test-only Command field from
ExecutionRecord; these seeds still set it.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

Superseded by the recomposed CODAGT-757 stack: #27317, #27318, #27319, #27320, #27321, #27322, #27323, #27324, #27325. The combined final tree of the new stack is byte-identical to the reviewed heads of this stack (minus an accidental workflow orchestration file); the recomposition splits the work into reviewable pieces.

This comment was posted by Mux, an AI coding agent, on Mike's behalf.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant