Repository navigation
feat(coderd): track chat execute tool calls in a durable execution ledger - #27100
ibetitsmike wants to merge 34 commits into
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 195dde9c71
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7f08dd6941
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
Codex Review: Didn't find any major issues. Keep them coming! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
/coder-agents-review |
|
Chat: Spend limit reached | View chat Review history
deep-review v0.9.0 | Round 10 | Last posted: Round 10, 73 findings (3 P1, 9 P2, 18 P3, 17 Nit, 26 Note), COMMENT. Review Finding inventoryFinding inventory - PR #27100Findings
Law analysis
Contested and acknowledgedCRF-2 (Note, execution_recorder.go) - Command written-never-read
CRF-3 / CRF-18 (Note, chattool/execute.go) - failed/lost start reports "may have run"
CRF-20 (Note, chattool/execute.go) - cross-host clock skew on grace deadline
CRF-37 (Note, chattool/execute.go:891) - large-output advisory lost on re-attach
CRF-41 (Note, execution_recorder.go:203) - no_effect distinct 9th enum value
CRF-65 (Note, chattoolcallexecutions.sql) - spare/kill split rides a caller-supplied bool
Round logRound 10 panelNetero + 14-reviewer panel + Law on head 260e700 (one commit: "restart the kill budget and widen coverage for edit-mapped executions"). Churn guard: PROCEED (CRF-67/68/69 all addressed). All three Round-9 findings verified genuinely fixed and MUTATION-TESTED by multiple reviewers (Netero, Hisoka, Takumi, Kite, Meruem, Melody, Pariston, Mafuuu, Mafu-san, Razor, Ryosuke, Bisky, Knov all ran the two new tests and reverted the fix to confirm they bite): CRF-67 re-stamps result_committed_at = @updated_at so resolveExpiredCancel restarts the 24h give-up clock at the edit (retention-safe: both purges exclude cancel_requested and age on created_at; sole value-consumer is the give-up bound; dual meaning contained); CRF-68 widens to status IN ('running','detached') with NO over-kill (every matched row is in the deleted suffix whose carrier is soft-deleted; a running row always carries process_id via UpdateChatToolCallExecutionProcess + the CHECK, so none is stranded; the two arms are mutually exclusive, not double-writing, because arm 1 moves outstanding rows out of the (running,detached) set first); CRF-69 rename cascade complete across all six layers. Law: Don't split / Advisory (reran, +668 since R6). 11 of 15 reviewers no findings. New: 3 Nits + 1 Note, all doc/test hardening, no bug: CRF-70 (migration column comment for result_committed_at still asserts the narrow "set when result commits" meaning, now contradicted by the re-stamp; operator-facing via dump.sql; Knuckle), CRF-71 (ARCHITECTURE.md:933 still says result_committed_at "nothing bumps afterwards", contradicted by the same commit's :912 re-stamp; Meruem), CRF-72 (the new running-row test lacks a pre-edit precondition assertion, so a future setup drift could silently pass it through the detached arm and stop guarding CRF-68; Bisky), CRF-73 Note (give-up-clock freshness rides two different mechanisms across the two arms; implicit coupling; Razor/Ryosuke/Meruem). One factual disagreement, immaterial to findings: Ryosuke claimed EditMessage cannot run mid-generation (only from R0), while 5 reviewers (Hisoka, Takumi, Melody, Bisky, Knov) cited transition.go allowing edit from R1/running; the over-kill conclusion is identical either way (killing deleted-suffix rows is correct), so no finding turns on it. Event COMMENT (max severity Nit). Reviewed against e489092..260e700. Round 9 panelNetero + 14-reviewer panel on head 5c74d97 (Law skipped, +484 since R6 < 500). CRF-64 (P3) and CRF-66 (Note) verified genuinely fixed by the whole panel: the new MarkDetachedChatToolCallExecutionsCancelRequested query routes every detached row anchored >= the edited user message to cancel_requested before the soft-deletes; boundary is off-by-one-free (target is always a user message, assistant ids strictly greater; >= == >), chat-scoped, dbauthz via parent-chat ActionUpdate; TestEditMessage_KillsPriorDetachedExecutions covers both arms (kill-on-deleted-carrier, spare-on-surviving-carrier) and fails under a neutered query (Bisky/knov mutation-tested). 11 of 15 reviewers no findings. New: 1 P2 (CRF-67: the edit-detached path never refreshes result_committed_at, so resolveExpiredCancel's 24h give-up bound resolves a >24h-old backgrounded process to unknown with zero kill attempts, orphaning it; Kite, orchestrator-verified against tasks.go:911 + generation.go:985 + synthetics.go:262), 1 P3 (CRF-68: the status='detached'-only edit query misses a running-with-committed-handle row left by a failed best-effort markTerminal(Detached); the interrupt map covers running, the edit map does not; Takumi, orchestrator-verified against execute.go:794-795,844 + query text), 1 Nit (CRF-69: min_assistant_message_id param name vs passed user-message id; Kite). Note the disagreement on CRF-68: 6 reviewers (Meruem, Melody, Pariston, Knuckle, knov, Razor) asserted "running never carries a committed result"; that holds only on the happy path. Takumi traced the best-effort-write-failure exception, which the orchestrator confirmed reachable with process_id recorded. Event COMMENT (max severity P2). Reviewed against e489092..5c74d97. Round 9 updateChurn guard: PROCEED. New head 5c74d97 (one commit: "kill prior-detached background executes when an edit deletes their handle carrier"). CRF-64 addressed (Mike's decision: option A/kill), CRF-66 addressed (doc), CRF-65 acknowledged (caller-supplied bool accepted for this PR + safe default; structural fix queued for planned stack recomposition, no ticket but a real acceptance rationale, not a bare deferral). Effective additions 6652 -> 6863 (+211; Law does not re-run, +484 since R6 < 500). CI green. Key verification: the new MarkDetachedChatToolCallExecutionsCancelRequested query + mapExecutionsForHistoryDelete(fromMessageID) boundary - detached rows anchored at/after the edited message must go to cancel_requested (killed), rows anchored BEFORE must keep their carriers (untouched); check the boundary (>= assistant_message_id) for off-by-one, chat-scoping, and whether ANY detached shape is not anchored by assistant_message_id (the next sibling). Reviewed against e489092..5c74d97. Round 8 panelNetero + 14-reviewer panel on head 57579ad (Law skipped, +30). All 5 round-7 findings (CRF-59..CRF-63) verified genuinely fixed: CRF-59 via the SpareBackground split (all 3 callers pass the correct value, CASE correct, edited-away bg -> cancel_requested -> sweep kills, no 4th caller; safe default is false/kill), CRF-60 doc rewrite accurate, CRF-61/62/63. 12 of 15 reviewers no findings; Hisoka explicitly hunted the sibling and cleared it on the outstanding path. New: 1 P3 (CRF-64: the recurring sibling for the ALREADY-detached case - a background execute spared by a prior interrupt then edited-away has its handle carrier deleted but is not swept, orphaning it; violates the PR's own kill-when-no-carrier invariant; Melody, verified), 2 Note (CRF-65 spare_background caller-dependency, CRF-66 doc-vs-structure). Event COMMENT (max P3). Reviewed against e489092..57579ad. Round 8 updateChurn guard: PROCEED. New head 57579ad (one commit: "kill edited-away background executes instead of sparing them"). All 5 round-7 findings (CRF-59..CRF-63) have code changes; panel verifies. Effective additions 6622 -> 6652 (+30; Law does not re-run). CI green. Key verification: the new SpareBackground param on MarkChatToolCallExecutionsInterrupted (a shared mapping query) - every caller must pass the correct value (synthetic-result transitions + interrupt = true/spare; history-delete = false/kill), the CASE must branch correctly, and edited-away background rows must become cancel_requested and actually be killed by the sweep. Also verify the ARCHITECTURE.md rewrite is accurate and no new sibling arm bypasses the mapping. Reviewed against e489092..57579ad. Round 7 panelNetero + 18-reviewer panel on head 43be9cf (Law skipped, +243 < 500). All round-6 findings verified genuinely fixed: CRF-43 foreground arm (mapOutstandingExecutionsForHistoryDelete before soft-delete, only EditMessage deletes history, no sibling bypass, orchestrator-verified), CRF-51 (partial index idx_..._uncommitted, migration renumbered 000546 no collision), CRF-52/53/54/55/56/57/58. New findings: 1 P2 (CRF-59: the CRF-43 fix spares in-flight BACKGROUND executes to detached but drops the handle since the edited turn commits no result -> orphaned unaddressable process; 5-reviewer convergence, orchestrator-verified; the foreground sibling the round-5/6/7 pattern predicts), 1 P3 (CRF-60 ARCHITECTURE.md:911 falsely claims edit preserves the spared handle + undocumented ordering invariant), 1 Nit (CRF-61), 2 Note (CRF-62 double-fetch, CRF-63 stale migration number in description). Event COMMENT (max severity P2). Reviewed against e489092..43be9cf. Round 7 updateChurn guard: PROCEED. New head 43be9cf (one commit: "map edited-away executions and index the abandoned purge"). All round-6 findings (CRF-43 P1 re-raise + CRF-51..CRF-58) have code changes; panel verifies. Effective additions 6379 -> 6622 (+243, Law does not re-run). CI green. Key verification targets: the CRF-43 fix (mapOutstandingExecutionsForHistoryDelete before EditMessage's soft-delete; third attempt at this P1) and whether any sibling history-deleting transition still bypasses it; new index migration 000546 (no collision). Reviewed against e489092..43be9cf. Round 6 panelNetero + Law + 20-reviewer panel on head 3e73ff1 (first panel on the R5 fixes). Verified genuine: CRF-27 (worker-ctx abort now fires before the drain, R5 re-raise resolved), CRF-44 (map-first, TOCTOU closed), CRF-45 (anchor on result_committed_at + agent-gone-first), CRF-48/49/50 and mostly CRF-47. New findings: 1 P1 re-raise (CRF-43: mapCanceledExecutions is inert on EditMessage, which soft-deletes the in-flight assistant message before synthesize; 5-reviewer convergence + orchestrator-verified), 1 P2 (CRF-51: CRF-46's abandoned-purge query is unindexed -> seq-scan of the highest-volume table every 10m, 2 EXPLAINs), 2 P3 (CRF-52 batch-cap untested, CRF-53 CRF-45 reorder untested), 4 Nit/Note (CRF-54..CRF-58). Event REQUEST_CHANGES. Reviewed against e489092..3e73ff1. Round 6 updateChurn guard: PROCEED. New head 3e73ff1 (one commit: "map the execution ledger in every cancellation transition and fix interrupt races"). All 8 round-5 findings (CRF-43..CRF-50) plus the CRF-27 re-raise have code changes; panel verifies. Effective additions 5845 -> 6379. CI green. This is the first panel pass on the round-5 fixes, notably the CRF-43 P1 fix (mapCanceledExecutions wired into 8 cancellation transitions) and the CRF-27/CRF-44/CRF-45 concurrency reorders. Reviewed against e489092..3e73ff1. Round 5 panelFirst panel pass on head aa4a05f (R3/R4 were BLOCKED). Netero + 21-reviewer panel (Law skipped: +352 LOC since R2, under 500). All round-3 fixes (CRF-25/26/28/29/30/33/34/36/38/39/40/42) verified genuine by multiple reviewers. New findings: 1 P1 (CRF-43 ledger not wired into edit/promotion/archive cancellation transitions), 1 P2 re-raise (CRF-27 shutdown-abort inert: AfterFunc on s.server.ctx fires only after chatWorker.Close(), which blocks on the reconcile; verified via chatd.go:4697/4707 + worker/runner drain), 1 new P2 (CRF-44 background-interrupt payload TOCTOU under READ COMMITTED), 3 P3 (CRF-45 give-up anchored on claim time; CRF-46 uncommitted rows escape purge; CRF-47 timed-out result builder 4th copy), 1 Nit (CRF-48 errors.AsType missed 3 sites), 2 Note. Event REQUEST_CHANGES. Reviewed against e489092..aa4a05f. Round 5 updateChurn guard: PROCEED. CRF-24 now deferred with linked ticket CODAGT-811 (quartz.Clock threading + mock-clock/sweep-ticker tests); do not re-evaluate. Head aa4a05f has never been panel-reviewed (R3/R4 were BLOCKED), so the round-3 fix commits (CRF-25 renumber, CRF-26 terminal give-up, CRF-27 AfterFunc shutdown, CRF-28 batch 10000, CRF-29 bounded sweep, CRF-33/34/36/38/39/40/42) get their first panel pass this round. Reviewed against e489092..aa4a05f. Round 4 updateChurn guard: BLOCKED. Head unchanged since round 3 (aa4a05f); no new commit. Author re-triggered /coder-agents-review without addressing CRF-24 and marked the CRF-24 thread Resolved with no code change, no linked ticket, and no substantive acceptance, so it is still silent. Re-posted the COMMENT block: marking the thread Resolved does not satisfy the gate; the finding needs a fix, a tracked ticket, or an explicit human accept. No panel spawned. Reviewed against e489092..aa4a05f. Round 2 updateChurn guard: PROCEED (18 addressed, 5 acknowledged, 0 contested/deferred/silent). PR substantially reworked into a durable execution ledger (effective additions +1948 -> +5493). CI is currently red (gen, lint, sqlc-vet, test-go-pg, test-go-race-pg, test-e2e). Panel re-runs against the reshaped code. Base 21d0824..head 80aeae9. Round 3 updateChurn guard: BLOCKED. New head aa4a05f (base rebased onto main, e489092). Of 19 round-2 findings: 16 addressed, 1 acknowledged (CRF-37), 1 contested (CRF-41), 1 silent (CRF-24). CI now green. No panel spawned this round: CRF-24 (Note) was deferred as "follow-up testability work" with no linked ticket and no code change, which blocks under the no-follow-up-without-ticket rule. Posted COMMENT (not REQUEST_CHANGES) asking the author to fix CRF-24, file a tracked ticket, or explicitly accept the gap, then re-request review. All P0-P3 findings from R1/R2 are resolved. Reviewed against e489092..aa4a05f. Round 2 panelNetero + Law + 23-reviewer panel. Root cause of the red CI found: CRF-25 (P1) migration 000544 collides with 000544_mcp_token_refresh_failure now on main (verified via git ls-tree origin/main). New findings: 1 P1, 3 P2 (CRF-26 seven-reviewer convergence on non-converging cancel_requested rows; CRF-27 shutdown-drain block; CRF-28 purge batch under-sized), 2 P3, 5 Nit, 8 Note. Event REQUEST_CHANGES. Reviewed against 21d0824..80aeae9. Round 1Netero + Law infrastructure pass, then 24-reviewer panel. Netero: 1 P3 + 2 Notes (mechanical floor clean). Law: Don't split (advisory). Panel: 1 P1 (CRF-4, 7-reviewer convergence), 2 P2 (CRF-5, CRF-6), 10 P3, 4 Nit, 6 Note. Event REQUEST_CHANGES on the P1. Reviewed against 344a977..eaf092d. About deep-reviewCRF = Coder Review Finding (P0-P4, Nit, Note)
|
There was a problem hiding this comment.
This is a well-scoped, well-tested fix for a real incident class (CODAGT-757). The reserve-then-start compare-and-set on (chat_id, tool_call_id) is the right primitive for electing exactly one creator across retries and coderd instances, the timeout round-up correctly stops a sub-second timeout from reconstructing as an instant 0s on re-attach, the 404-only unknown-state synthesis has exactly the right blast radius, the dbauthz policies follow the parent-chat pattern, and 62.5% test density with real negative assertions (background-survival, timed-out-spare) is strong. Law's verdict: don't split, this is one reviewable idea.
The panel found one blocker. CRF-4 (P1, seven reviewers converging) is the incident class this PR exists to eliminate, reintroduced through the interrupt path. An interrupt landing in the StartProcess->RecordStart window reads a NULL-handle reservation, skips the kill, then deletes the row, so the just-started foreground process is never signaled and the handle write fails with sql.ErrNoRows. The process runs orphaned for the life of the workspace with no record left to reap it. The interrupt tests only fire after the handle is recorded, so this window is uncovered. This is distinct from the PR-2 transport-loss gap: here the process started and coderd had the handle, and the interrupt path itself destroyed the reservation.
Two P2s: the interrupt kill runs synchronously (serial per-agent 5s dials) before the status-change publish, so a slow or unreachable agent delays the interrupt's client-visible ack, contradicting the stated "a slow or unreachable agent never delays the interrupt" goal (CRF-5); and the idempotency safety net is invisible to operators (no log, no metric, row deleted after commit) for a change whose entire justification is operational (CRF-6).
The rest are P3 and below: two independent agent-identity mismatches that can defeat the interrupt kill and re-attach in the multi-agent case (CRF-9, CRF-10), an unbounded purge DELETE that diverges from every batched sibling (CRF-8), a silent and undocumented 4h timeout clamp the model cannot see (CRF-7, the widest convergence in the panel), a missing test for that clamp (CRF-11), plus assorted correctness edges, duplication, and schema-hardening notes.
Process note: two intermediate commit subjects exceed 72 chars (record process starts on an uncanceled context... at 99, scope timeout validation to foreground... at 87). Both bodies are excellent; harmless if this squash-merges under the 70-char PR title.
Hisoka summed up the panel's read: "You built something that holds up under most of what I threw at it. A good fight."
No finding here has a follow-up owner assumed. Please fix CRF-4 in this PR or state explicitly why the orphan window is acceptable.
coderd/x/chatd/chattool/execute.go:707
Nit [CRF-14] process_output accepts a negative wait_timeout and silently treats it as an immediate snapshot, while execute rejects a non-positive timeout. (Melody Nit, Pen-Botter Note)
execute rejects parsed <= 0 with "timeout must be positive"; process_output applies min(parsed, maxExecuteTimeout) then guards if timeout > 0, so -5s falls through to a non-blocking snapshot with no error, and the description documents only 0s as the snapshot sentinel. Reject parsed < 0 (or document that any non-positive value means snapshot) so the two tools agree on what a bad duration is.
🤖
🤖 This review was automatically generated with Coder Agents.
|
@codex review |
1 similar comment
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 621accc3d7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a5d02fd7d7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 82e924ca69
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8a6621e801
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0769f2f6a5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6eaf2dc23d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 552a7e0d0b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3b3eb34a04
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Round 9. The CRF-64 prior-detached-carrier orphan (P3) and the CRF-66 doc mismatch (Note) are both genuinely fixed, and CRF-64 is fixed with the right shape: instead of squeezing the kill decision back through the last-assistant outstanding scan that let the sibling slip for five rounds, the new MarkDetachedChatToolCallExecutionsCancelRequested anchors on durable ledger state and routes every detached row anchored at or after the edited message to cancel_requested before the soft-deletes. Eleven of fifteen reviewers found nothing. The panel verified the boundary is off-by-one-free (the edit target is always a user message, so assistant_message_id >= target.ID equals > target.ID and equals "the carrier is in the deleted suffix"), chat-scoped, and authorized via the parent-chat ActionUpdate pattern; TestEditMessage_KillsPriorDetachedExecutions exercises both directions and was mutation-tested (it fails when the new query or its >= predicate is neutered).
The recurring "fix the named arm, leave a sibling" class has narrowed but not fully closed. Two new gaps surfaced on the paths this fix touches:
- CRF-67 (P2): the edit-detached mapping never refreshes
result_committed_at, so the sweep's 24h give-up bound gives up on a long-lived backgrounded process with zero kill attempts and resolves itunknown, orphaning exactly the process this PR exists to reap. - CRF-68 (P3): the edit query matches only
status = 'detached', while the interrupt map it mirrors also coversrunning; arunningrow that kept a committed handle after a failed best-effort terminal write is missed on edit.
One Nit on parameter naming (CRF-69). CRF-65 (caller-supplied @spare_background bool) remains acknowledged for this PR with a stated rationale and a safe default; not re-raised.
🤖 This review was automatically generated with Coder Agents.
Round 9 dispositionAll Round 9 findings are addressed in 260e700 ( CRF-67 (P2): edit-mapped rows kept a stale give-up anchor. Fixed. The history-delete mapping re-stamps CRF-68 (P3): edit rescue narrower than the interrupt map. Fixed. The query (renamed to CRF-69 (Nit): misleading parameter name. Fixed. Validation:
|
|
/coder-agents-review |
|
@ibetitsmike ⛔ This review has reached its per-chat spend limit ($426.41 / $400.00). Further review rounds are paused. To raise the limit and continue, comment: This is a per-chat budget, separate from any account-level usage limit.
|
|
/coder-agents-review set-spend-limit:450 |
|
OK fine, @ibetitsmike, $450.00 it is... but I'm telling finance about this. 💸
|
|
/coder-agents-review |
There was a problem hiding this comment.
Round 10. The three open Round-9 findings are all genuinely fixed by one surgical commit, and the panel verified each with mutation tests rather than admiration: reverting the result_committed_at re-stamp flips the sweep test to unknown with zero kill attempts, and narrowing the status set back to detached leaves the failed-detach-write row orphaned. CRF-67 (re-stamp restarts the 24h give-up clock at the edit) is retention-safe: both purge queries exclude cancel_requested and age on created_at, and the only value-consumer of the column is the give-up bound, which wants the cancellation time. CRF-68 (widen to status IN ('running','detached')) does not over-kill: every matched row is anchored in the deleted suffix whose carrier is soft-deleted, a running row always carries process_id (so none is stranded), and the outstanding arm moves last-message rows out of the set before the widened query runs, so the two arms are mutually exclusive rather than double-writing. CRF-69's rename cascade is complete across all six layers. Law reran (growth since Round 6 crossed the split threshold) and holds Don't split / Advisory: the growth is intra-feature bug-fix depth, not new scope.
Eleven of fifteen reviewers found nothing. What remains is documentation and test hardening, no behavioral defect: two stale doc assertions that this same commit's re-stamp now contradicts (CRF-70, CRF-71), a missing test precondition (CRF-72), and one note on an implicit cross-arm coupling (CRF-73).
🤖 This review was automatically generated with Coder Agents.
…unning-row test precondition
Round 10 dispositionAll Round 10 items are addressed in 07140fe ( CRF-70 (Nit): migration column comment for Validation: make gen clean, targeted chatstate test green, pre-commit hook (lint/go) passed. PR2/PR3 restacked and pushed.
|
|
/coder-agents-review |
|
@ibetitsmike ⛔ This review has reached its per-chat spend limit ($468.96 / $450.00). Further review rounds are paused. To raise the limit and continue, comment: This is a per-chat budget, separate from any account-level usage limit.
|
|
Superseded by the recomposed CODAGT-757 stack: #27317, #27318, #27319, #27320, #27321, #27322, #27323, #27324, #27325. The combined final tree of the new stack is byte-identical to the reviewed heads of this stack (minus an accidental workflow orchestration file); the recomposition splits the work into reviewable pieces.
|
Problem
When a chatd task attempt dies mid-
execute(attempt timer expiry, retryable error, pod replacement), the retry re-reads history, finds the unresolved tool call, and callsStartProcessagain. Each retry cycle spawns another OS process in the workspace and the old ones are never killed, producing the orphaned duplicate-process incident class tracked by CODAGT-757 (for example, duplicate long-running builds racing each other in the same worktree).What this PR changes
This PR introduces a durable execution ledger that owns the external-process lifecycle of local
executetool calls. ChatMachine keeps owning chat semantics and message commits; the ledger records what actually happened to the process, independently of what the chat history says.Invariant: one logical local tool call creates at most one external process. Later attempts re-attach, observe, reconcile cancellation, or return a stable unknown-outcome result. They never silently start another process.
Schema (migration
000546)chat_tool_call_executionswith a UUID primary key generated when the assistant message intent is persisted. The UUID is the stable external-process identity (and becomes the agent idempotency token in PR 2 of this stack).(chat_id, assistant_message_id, tool_call_id). Provider tool-call IDs can be reused after history regeneration, so a regenerated assistant message creates an independent row instead of mutating the old process identity.input_sha256over the raw persisted tool input; claims refuse to dispatch against a row created for different input.reserved,starting,running,exited,detached,cancel_requested,canceled,unknown,no_effect.result_committed_atis a timestamp, not a status: it is set in the same transaction that commits the tool result (real or synthetic), so diagnostic lifecycle states likecancel_requestedandunknownstay readable after chat history has moved on.claim_epoch+claimed_atimplement claim ownership;cancel_signal_sent_atrecords a delivered but unconfirmed kill.CHECK ((process_id IS NULL) = (started_at IS NULL))makes the process-identity invariant structural: a row either has a dispatched process with its start time or neither.timeout_secsrecords the clamped foreground deadline and is zero for background executions, which have none.created_at), lineage lookups (assistant_message_id), theworkspace_agentsforeign key (partial index onworkspace_agent_id), and the cancellation sweep (partial index onupdated_atforcancel_requestedrows).Atomic companions via the ChatMachine transaction
reservedintent per execute call in the same transaction (commitGenerationStep).result_committed_atin the same transaction. The old post-commit delete path is gone; rows are never deleted at resolution time.detached,reservedrows becomecanceled, and everything else in flight (foregroundstarting/running, plus background starts whose handle has not landed yet, plus foreground rows detached by a timed-out wait whose result never committed) becomescancel_requested, so no row terminalizes as detached without a recoverable handle. The mapping runs before the synthetic results are built and each result derives from its mapped RETURNING row, so the spare-as-detached decision and the handle in the committed result are one observation.result_committed_atstamp in its own committing transaction; the periodic sweep converges thecancel_requestedrows they leave. Message edit soft-deletes every message from the edited one onward, so before the delete it maps the outstanding executions with background sparing disabled (no surviving result can carry a handle, so every dispatched row becomescancel_requestedfor the sweep to kill) and also routes process-bearing rows (runningordetached) anchored in the deleted suffix tocancel_requested, because the committed results that carried their process handles are deleted with it; that mapping re-stampsresult_committed_atso the sweep's give-up clock restarts at the edit. Abandon and archive deliberately do not map; the turn's calls stay re-attachable. The post-interrupt reconciliation runs after the interruption is published so interrupts stay responsive. A ledger claim that fails on infrastructure aborts the whole local tool batch instead of committing an error result, so the task retry re-claims the intent.Execute tool flow
reservedintent or a stale (>60s)startingclaim, advancingclaim_epoch. A freshstartingclaim is polled until its owner records a process; a stale one resolves tounknownin this PR (PR 2 upgrades this to a token probe).running/exited/detachedrows re-attach via an output snapshot: an exited process commits its real result even past the deadline; a running process with time left is block-waited for the remainder; past the deadline the graceful timed-out result withbackground_process_idis returned and the row markeddetached. Background rows return the process handle directly.unknownand produces anis_errorresult saying the command may have run, and only when the probed connection targets the recorded owning agent. When the turn's connection targets a different agent, re-attach dials the recorded owner directly; if that dial fails, the attempt aborts instead of committing a result that would end re-attachment. Transport errors, cancellations, and 5xx keep the lifecycle state unchanged and the process retrievable.no_effect; ID-less tool calls are refused before claiming, since the ledger keys rows by tool call ID. Safety-net resolutions (unknown-state rows, stale-claim expiries, failed re-attach probes) are logged so operators can observe the idempotency net firing. Process identity is recorded on a bounded uncanceled context so an interrupt landing mid-start cannot lose the handle, and lifecycle writes are skipped once the interrupt has canceled the generation context. Foreground wait outcomes (exited/detached) are recorded only when the identity write landed, so a failed write leaves the rowstartingfor claim recovery instead of stranding a terminal row with no handle.unknown,canceled,cancel_requested,no_effect) return their stable results even when the workspace connection cannot be established, and background rows with a recorded handle likewise resolve without a connection (their durable result is the handle itself); only calls that actually need the agent surface connection errors.Interrupt reconciliation
After the interrupt commit, a reconciler kills recorded processes best-effort: confirmed termination resolves
cancel_requestedtocanceled; a delivered but unconfirmed kill keeps the rowcancel_requestedwith full process identity andcancel_signal_sent_at, so nothing about the process is forgotten just because the kill could not be confirmed. Rows without recorded process identity are polled through a grace window first, because an interrupted in-flightStartProcesscan still land its handle via the uncanceled record write: a handle that arrives gets the kill flow (background rows included: their committed synthetic result carries no handle, so sparing them would strand a running process), and only after the window closes does the row resolve tounknownin this PR.The immediate pass is best-effort, so a periodic per-replica sweep guarantees convergence: every minute it claims
cancel_requestedrows with process identity whoseupdated_atis older than a 3-minute retry age (FOR UPDATE SKIP LOCKED; the claim bumpsupdated_atas a cross-replica lease) and re-runs the kill flow until a terminal outcome is recorded. An interrupted process is eventually killed even when its agent was unreachable at interrupt time or the server died mid-reconciliation. Every pass is deadline-bounded so unreachable agents cannot collapse the retry cadence, and the immediate pass aborts promptly on server shutdown (the sweep backstops it). Rows whose kill can never be confirmed still terminalize: a deleted agent (including the FK nullingworkspace_agent_id) resolves tocanceledsince the process died with its workspace, and a row whose agent merely stays unreachable resolves tounknowna day after its cancellation (anchored on the cancellation commit, so a long-lived call interrupted a day after its dispatch still gets its full kill budget).Retention
dbpurgedeletes ledger rows older than 7 days in bounded batches of 10000 (this is the highest-volume chat table: one row per execute call), but only rows whose tool result was committed: an uncommitted row still guards dedup for a call a future retry may re-execute. Uncommitted rows idle past 30 days are reaped on a separate horizon (a row that old belongs to a turn that will never rerun; without this, deployments without chat retention would grow the table unboundedly).cancel_requestedrows are retained regardless of age, since they carry the only stored identity of a process the sweep must kill. Deletion only discards diagnostic history; a still-running detached process stays addressable through the process handle in its committed tool result.ActionUpdate/ActionRead;ResourceSystem+ActionDeletefor the purge query).coderd/x/chatd/ARCHITECTURE.mddocuments the full lifecycle.What this PR deliberately does not change
StartProcessresponse lost in transit can still leak one process; the ledger bounds it to at most one per tool call thereafter.enterprise/audit/table.gois untouched: this is an operational child table and the auditedChatmodel is unchanged.ExecuteOptions.Recorder == nil), keeping existing chattool unit tests valid.Test plan
Ran locally against Postgres:
go test ./coderd/x/chatd/...(full tree: recorder claim/reattach/terminal-status suites, intent and result-commit atomicity, interrupt mapping with failed-kill retention, late-handle interrupt races, periodic sweep convergence and lease retry, lineage collision after history regeneration, owning-agent-scoped 404 handling)go test ./coderd/database/dbauthz/ -run 'TestMethodTestSuite/TestChats'go test ./coderd/database/dbpurge/(batched ledger purge; uncommitted rows retained)go test ./coderd/database/migrations/(new migration + fixture)make fmt,make lint, andmake genwith a clean diff after.Stack
PR 1 of 3 for CODAGT-757 (base:
main). PR 2 (#27102, agent-side idempotency token) and PR 3 (#27103, attempt watchdog) build on this branch.