Skip to content

feat: send tool call IDs and cancel unresolved calls on interrupt - #29954

Merged
mafredri merged 23 commits into
mainfrom
mafredri/codagt-757/02-agent-tool-call-records
Sep 30, 2026
Merged

mafredri merged 23 commits into
mainfrom
mafredri/codagt-757/02-agent-tool-call-records

Conversation

@mafredri

@mafredri mafredri commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

chatd now sends a tool call ID, derived from committed rows, with execute starts, edit_files, and write_file, so a retry or replica takeover gets the first run's result instead of running the tool again. A retried foreground execute waits only for the rest of its timeout.

An interrupt cancels the unresolved calls of those three tools on the agent and commits what happened: tool call was canceled before it ran; no changes were made, the killed command's output with tool call was canceled while running, or the real result. If the cancel fails, including an old agent's 404, the result is today's generic text.

Part 2 of 2 of the CODAGT-757 stack.
Depends on #29946.

Fixes https://linear.app/codercom/issue/CODAGT-757

Design: run chat tool calls once, and cancel them truthfully

CODAGT-757: run chat tool calls once, and cancel them truthfully

Status: revision 4, being implemented as two draft PRs. Earlier revisions are kept outside the repo (/home/coder/wt/PLAN-revision-2.md, PLAN-revision-3.md) and as local tags codagt-757-rev2/* and codagt-757-rev3/*.

Goal: the fewest lines that stop duplicate side effects and give truthful interrupt results. Rare cases that cost more than they are worth keep today's behavior (section 4).

1. Terms

Term Meaning
tool call ID A UUIDv5 of the chat ID, the assistant message ID, and the provider tool call ID, computed by chatd. Every replica and every attempt computes the same value from committed rows. For execute it is also the process ID.
Coder-Tool-Call-Id The request header that carries the tool call ID on start, edit, and write requests.
unresolved tool call A tool call in the latest assistant message without a committed result (existing term).
saved response The status, Content-Type, and body the agent returned for a tool call ID the first time.

2. User stories

  • US1 One run per tool call. A task retry or an ownership change sends the same tool call ID; the agent returns the saved response (for execute, the running or exited process) instead of running again. Holds while the agent keeps running.
  • US2 Interrupt stops unresolved execute. Foreground and background processes of unresolved calls are killed. The result is "tool call was canceled while running" with the output so far, or the real result if the command had finished.
  • US3 Interrupted edits have a definite result. "tool call was canceled before it ran; no changes were made" if the agent never received the edit, otherwise the real edit result.
  • US4 No run after an interrupt. A start, edit, or write that reaches the agent after the interrupt's cancel is refused.
  • US5 Timeouts are not restarted. The agent keeps waitUntil from the first start, so a retried foreground execute waits only for the rest of its timeout.
  • US6 Processes the model knows about survive. Only unresolved calls are canceled.
  • US7 Old agents behave as today. They ignore the header; the cancel route responds with 404, which gives today's generic interrupt result. No detection code.
  • US8 Chats stay apart. The agent keys tool call records and processes by chat ID. A tool call ID is read only from a request that has a chat ID; a process started without a chat ID is keyed by the zero UUID.

3. Design

Agent (agent/agenttoolcall, agentproc)

  • Coder-Tool-Call-Id is read by agentchat.Middleware into agentchat.Context.ToolCallID, next to the chat ID (and logged as tool_call_id). Handlers and the middleware below read it from the context.

  • A table keyed by chat ID and tool call ID. A record is running (with a channel that closes when the run finishes), done (with the saved response), or canceled.

  • One middleware, registered once for the whole agent API right after agentchat.Middleware: a request whose context carries a tool call ID runs once per chat and ID. Without a tool call ID: unchanged. With it:

    Record Action
    none run the handler once, save its response, return it
    running or done wait, return the saved response
    canceled 409, do nothing

    A handler panic saves a 500 saying the outcome is unknown, so repeats and cancels do not wait forever.

  • POST /api/v0/tool-calls/{id}/cancel, mounted from Table.Routes(): no record stores a canceled record, kills a process with that ID (it outlives the record), and responds with received: false. A record is marked canceled, the cancel waits for the run to finish, kills its process if the ID has one, and responds with received: true and the saved response.

  • Processes are keyed by chat ID and process ID. A process started with a tool call ID uses it as its process ID; a start for a key that already has a process returns that process. Output, signal, list, and kill see only the requester's chat's processes. The start request's timeout_ms sets the process's waitUntil at the first start; a repeat does not move it, and the process keeps running after it. A start with both background and a non-zero timeout_ms is rejected with 400, since only regular output waits read background processes. A blocking output wait with timeout_from_start_process=true (the execute tool's own wait) returns at exit, at waitUntil, or at the existing 5-minute cap; other waits, such as process_output's, ignore waitUntil. The output response reports timed_out (running past waitUntil) and canceled (killed by a cancel).

  • Adding a record deletes records that no request or cancel has used for an hour (forgetAfter), to bound memory. chatd resends every unresolved call on each retry, and a long batch is retried about every 15 minutes, so a batch that chatd is still retrying keeps its records.

  • Exited processes stay listed for an hour after they exit (5 minutes on main). Redesigning this retention is CODAGT-1220.

SDK (codersdk/workspacesdk)

  • CoderToolCallIDHeader, WithToolCallID(ctx, id), ToolCallIDFromContext(ctx). Only StartProcess, EditFiles, and WriteFile send the header, because only they act.
  • CancelToolCall(ctx, id) (CancelToolCallResponse, error); CancelToolCallResponse{Received bool; Status int; ContentType string; Body []byte} with one decoder per saved response, each calling the decode code of the method it mirrors: StartProcessResult(), EditFilesResult(), WriteFileResult().
  • StartProcessRequest.TimeoutMs; ProcessOutputOptions.TimeoutFromStartProcess; ProcessOutputResponse.TimedOut, Canceled.

chatd (coderd/x/chatd)

  • The dispatcher gives each tool call's context its tool call ID (chattool.ToolCallIDs), attached with WithToolCallID. Only the three acting SDK methods send it. A call whose provider tool call ID is empty or repeats in the message gets none and behaves as today, since calls sharing an ID would get each other's saved responses.

  • execute foreground sends timeout_ms, waits with timeout_from_start_process=true, and waits until the output response reports exit or timed_out. chatd's own timeout, counted from each attempt, stays as a backstop, which gives an old agent (it ignores timeout_ms) today's behavior.

  • The interrupt task reads the unresolved execute, edit_files, and write_file calls that have a tool call ID in its locked read (the same read that checks the chat is still in the state the task started for). After releasing the lock, it dials the agent once and sends one cancel per call, all at once, under one 30-second timeout. A cancel waits for a running call to finish, so sending them at once keeps one slow call from using up the others' time. A panic in one cancel gives that call the generic text. Results:

    Response execute edit_files / write_file
    received: false read the output: not found gives the plain-text error tool call was canceled before it ran; no changes were made; found is handled like a saved success tool call was canceled before it ran; no changes were made
    saved error response the start error, as today the edit or write error, as today
    saved success read the output after the process exits (the kill does not wait for the exit): canceled gives tool call was canceled while running with the output so far, marked "canceled": true (the UI shows the killed marker); exited gives the real result the edit or write result
    404, error, timeout today's generic text, and a warning log today's generic text, and a warning log

    Results go into the existing commit path (committedPendingLocalToolCancellationMessages, FinishInterruption) as a map by provider tool call ID.

4. Not handled

Retries keep today's behavior (the tool runs again):

  • An agent restart mid-call. The restart already killed the process.
  • A retry more than an hour after chatd last sent the call (for example after a coderd outage), once a newer tool call deleted the record, unless the process is still listed.
  • An agent that stays connected but never responds.
  • Two agent processes with one token at the same time.
  • Windows: a cancel kills the sh process only, as process_signal does today.
  • A process that exits on its own just before a cancel's kill can be reported as canceled while running.

An execute that finished early in a batch that stays unresolved (for example next to a long sibling) keeps its record while chatd retries, but its process is removed an hour after it exits, the next time any chat lists processes. Later retries then get an output error ("get process output: ...") instead of its result. It does not run again, but its output is lost (CODAGT-1220).

After a retry, an execute result's wall_duration_ms counts only the last attempt (a 20-minute timeout showed "for 5m" in UAT): chatd measures it per attempt, and the agent's responses do not carry the process's start time. The timeout itself counts from the first start (CODAGT-1243).

Interrupts give a false "canceled before it ran" when the agent has no record of a call that did reach it. Today these cases give the generic text. Accepted to keep the change small; a complete fix needs a record that survives the agent process or a comparison of the call's age with the agent's uptime, both rejected (section 6):

  • an edit or write interrupted more than an hour after chatd last sent it (only after an outage, since every retry sends it again);
  • an agent restart, or the chat moving to another agent, after the call ran;
  • a rolling chatd upgrade in which an old replica ran the call without the header;
  • an execute whose process exited more than an hour earlier and was dropped from the process list after the record was deleted.

A workspace that cannot be reached during the interrupt records no cancel, so a request that arrives later still runs. The committed result is today's generic text.

5. PRs

  1. feat: run chat tool calls once per tool call ID on the agent #29946 mafredri/codagt-757/01-sdk-tool-call: agent and SDK.
  2. feat: send tool call IDs and cancel unresolved calls on interrupt #29954 mafredri/codagt-757/02-agent-tool-call-records: chatd.

#29958, #29965, #29976, and #29978 are closed as superseded. Branches are rewritten from main and force-pushed (operator approval, 2026-09-28).

6. Decisions (2026-09-28, operator)

  • LOC reduction first; do not pay for rare cases; relax user stories to allow simpler designs.
  • No agent protobuf, manifest, agent API version, or coderd database changes. The contract lives in the agent HTTP API.
  • No comparison between chatd's and the agent's clocks, and no old-agent detection. The one-hour expiry uses only the agent's clock.
  • Interrupts kill unresolved background processes and processes past waitUntil (accepted earlier).
  • Edits, archives, and other generic cancellations sending a cancel first: follow-up work (CODAGT-1222).
  • Implement fast; reviews and fixes afterwards. Hooks are off in this workspace.
  • After review: received: false for execute falls back to the process list, and the remaining false results are documented in section 4 rather than fixed. Attach and kill respect chat boundaries.
  • Third-party review: adopted the panic fix, chat checks, the empty or repeated provider ID guard, the section 4 additions, and tests for concurrent requests, panics, cancels during a run, and cross-chat access. Rejected: buffering request bodies (write_file is atomic, so a cut-off upload saves a true error); retention counted from run end (recorded runs are short; the process list covers execute); restricting IDs to three tools and per-route middleware (the SDK sends the header only from the three acting methods); a mixed clock (already one clock); reporting time left on retry (the timeout counts from the first start).
  • Only the execute tool's own wait stops at waitUntil (timeout_from_execute=true); process_output waits keep their own timeout. A background start with a timeout is rejected (operator, after the first review round).
  • 2026-09-29, after Codex and Copilot review: interrupt cancels are sent at once, and the timeout is 30 seconds instead of 10, because a chance of a real result is worth the wait more than unknown results. Processes are keyed by chat ID and process ID; non-chat requests seeing chat processes was a bug, not compatibility to keep, and the key makes Codex's cross-chat start race unreachable. Exited processes stay an hour instead of 5 minutes; the redesign is CODAGT-1220. entryMaxAge is renamed forgetAfter.
  • 2026-09-29, second review round: the interrupt reads its cancelable calls under the chat lock (ARCHITECTURE.md runner rule); the output read after a cancel waits for the exit; each cancel goroutine recovers panics. Not changed: a cancel that gives up (timeout, or chatd went away) does not kill a process whose start is still running, since the interrupt then records the generic text; the size cap keeps the default budget.
  • 2026-09-29, third-party finding: a record's age was counted from when it was added, so in a batch with a long execute, an edit's record could be deleted while chatd still retried the batch, and the edit ran again. Every request or cancel that finds a record now resets its age. This replaces the earlier rejection of retention counted from run end.
  • 2026-09-29, UAT round 1 (FAIL on one UI regression: canceled and not-run execute cards read "Ran ... for 0ms"): the canceled result passes the agent's canceled flag through as "canceled": true, and the UI shows it with the marker it already uses for commands killed by process_signal; "not run" is a plain-text error like the edits' "not applied" ("Failed to run"); a zero wall_duration_ms is left out. Rejected: marking the canceled JSON result is_error, which says "Failed to run" for a command that ran, makes interrupts the only execute failures flagged as errors, and would drop the output from the model on replay (only the error field of an is_error JSON part reaches it). The collapsed card not marking any failed command (exit code, timeout) is a separate UI question. The per-attempt duration after a retry is documented in section 4 (operator: a chatd and agent API issue; a WebSocket between them in the future would remove the blindness, and also the start-then-wait of execute).
  • 2026-09-30, operator: one result text per outcome for all three tools, without naming the transport: tool call was canceled before it ran; no changes were made and tool call was canceled while running. CancelToolCall has a case per tool name and rejects other tools before sending. The agent's cancel hook is named cancel, not kill, because it also serves edits. ARCHITECTURE.md states the at-most-once exception and the interrupt's cancel step instead of a TODO. Comments keep only what the code does not show. TimeoutFromExecute (timeout_from_execute) is renamed TimeoutFromStartProcess (timeout_from_start_process), because it is part of the agent process API and should not be named after a chatd tool.
  • Tests: each behavior is asserted once, at the highest layer that reaches it (real SDK and agent; one chatd end-to-end interrupt test with a real agent); unit tests only for what those cannot reach. Tests are deterministic: fake clock for time, no real-time deadlines or timed polls, assertions that hold in every interleaving.

🤖 This PR was created with the help of Coder Agents, and will be reviewed by a human. 🏂🏻

@linear-code

linear-code Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

CODAGT-757

CODAGT-1222

@mafredri
mafredri added this pull request to stack #29985 September 26, 2026 12:50
@mafredri
mafredri force-pushed the mafredri/codagt-757/02-agent-tool-call-records branch 2 times, most recently from 0e8c849 to d420b04 Compare September 28, 2026 11:17
@mafredri mafredri changed the title feat(agent/agenttoolcall): add tool call records feat(coderd/x/chatd): send tool call IDs and cancel unresolved calls on interrupt Sep 28, 2026
@mafredri
mafredri force-pushed the mafredri/codagt-757/02-agent-tool-call-records branch from d420b04 to e71fffb Compare September 28, 2026 13:02
@mafredri
mafredri removed this pull request from stack #29985 September 28, 2026 13:06
@mafredri
mafredri added this pull request to stack #30043 September 28, 2026 13:07
@mafredri
mafredri force-pushed the mafredri/codagt-757/02-agent-tool-call-records branch from e71fffb to 64557a1 Compare September 28, 2026 13:41
@mafredri
mafredri force-pushed the mafredri/codagt-757/02-agent-tool-call-records branch from 64557a1 to 85ab12e Compare September 28, 2026 14:04
@mafredri
mafredri force-pushed the mafredri/codagt-757/02-agent-tool-call-records branch from 85ab12e to 56dde11 Compare September 28, 2026 14:39
@mafredri
mafredri force-pushed the mafredri/codagt-757/02-agent-tool-call-records branch 2 times, most recently from f212a37 to f3602fd Compare September 28, 2026 17:27
@mafredri
mafredri requested a balanced review from Copilot September 29, 2026 08:42
@mafredri

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-30T08:48:37.443310Z 7505856 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f3602fd83d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/ARCHITECTURE.md Resolved
A task retry or ownership change runs unresolved tool calls again,
and an interrupt records a generic result even when the command or
edit already ran. Each local tool call now carries a tool call ID
computed from committed rows, so the agent can return the first run's
result, and the interrupt task asks the agent to cancel unresolved
execute, edit_files, and write_file calls and records what it did.
Agents without the cancel route keep the generic result.

Foreground execute sends its timeout to the agent, which keeps the
deadline of the first start, so a retry returns when the agent
reports timed_out instead of waiting a full timeout again. chatd's
own timeout stays as the backstop for agents that ignore timeout_ms.
An interrupted execute reported "not run" whenever the agent had no
record of the call, but the agent forgets calls after an hour while
their processes keep running. chatd now reads the process output in
that case and reports "not run" only when the agent has no process
with the tool call ID.

Calls with an empty or repeated provider tool call ID in one message
would share a tool call ID, and the second would get the first one's
saved response. Those calls now get no tool call ID and keep today's
behavior.

The interrupt fell back to the generic result without saying why.
Each failure to load, dial, or cancel now logs a warning.
The agent now ends a blocking output wait at the start timeout only
when the request asks for it, so process_output waits keep their own
timeout. The execute tool's foreground wait asks, so a retried execute
still waits only for the rest of its timeout.

The test's fake agent now honors that contract instead of answering
timed_out unconditionally.
The SDK renamed the option that asks the agent to end a blocking
output wait at the execute timeout to TimeoutFromExecute. The execute
tool's foreground wait uses the new name so the branch builds against
the renamed SDK.
The interrupt only commits the agent's real results when the
dispatcher and the interrupt compute the same tool call ID. The unit
tests faked the agent and checked each side separately, so they could
not catch the two sides disagreeing.

The new test runs write_file and a long execute on a real agent,
interrupts, and checks that the committed results are the agent's
write result and a killed command. It replaces the dispatcher unit
test and the WriteApplied and ExecuteCanceled rows, which asserted
the same behavior with fakes.
The interrupt dialed the agent whenever any call was unresolved, so a
chat whose open calls were only other tools could wait up to the
cancel timeout on an unreachable workspace. It now keeps only execute,
edit_files, and write_file calls that have a tool call ID and returns
before dialing when none remain. CancelToolCall no longer checks the
tool name, since it only sees those three tools.

The interrupt test gains a row for another tool, which must get no
cancel and the generic result. It no longer pins the chat header,
which the end-to-end test checks, and the row named for a missing
cancel route is now CancelError, since it passes a plain error.
Results committed by an interrupt skipped the size cap that chatloop
applies to normal tool results, so a large edit diff or command output
from the agent went into the prompt whole. The interrupt now caps them
with chatloop's truncation at the default budget, since it does not
know the model's context window.

A truncated JSON result is no longer valid JSON, and committing it
fails the whole interrupt. The result is now built with
chatprompt.PartFromContent, which wraps such text as {"output": ...}
as the normal path does.
The agent answers a cancel only after the call's running handler
finishes. Sent one at a time under one shared timeout, a slow first
cancel left later cancels starting after the deadline, so those calls
got the generic interrupted result and their processes kept running.

Raise the timeout from 10s to 30s so running tool calls get a chance
to finish and the interrupt records their real result.
The interrupt task read the chat's messages for the cancels without
the chat lock or the task fence. A stalled task could then read tool
calls a newer turn started and cancel them on the agent before its
own commit failed the fence.

Read them in the task's locked read instead. A failed message read now
fails that read and the task retries, as the transaction cannot commit
after a failed query.
The agent answers a cancel once it sends SIGKILL, and collects the
process's last output when the process exits. A snapshot read right
after the cancel could miss that output. Wait for the exit instead;
the interrupt's cancel timeout bounds the wait.
Each cancel runs in its own goroutine, outside the recover of
executeTaskSafely, so a panic in one crashed coderd. Recover it as that
call's error: the call keeps the generic interrupted result and the
interrupt commits, as chatloop does for parallel tool calls.
An interrupted execute result read "Ran X for 0ms" in the chat UI: the
canceled result looked like a normal run, and the zero duration of a
result that was never timed was shown.

The canceled result now carries canceled: true from the agent's
process output, so the UI can mark the command as killed. The error
text stays for the model. A call the agent never received is now a
plain-text error, like the edits' "not applied", since nothing ran.
wall_duration_ms is omitted when zero, so untimed results show no
duration.
An execute result with canceled: true is a command the user's
interrupt killed with SIGKILL, so it gets the same killed indicator as
a command killed by a process_signal call.
CancelToolCall picked its branch from negated name checks, so any tool
other than execute got the edit/write "not applied" result, and the
interrupt task's filter repeated the tool names as string literals.
Each cancelable tool now has its own case, any other tool is an error,
and CanCancelToolCall keeps the filter's list next to the switch.
…cancels

The at least once paragraph did not mention that the workspace agent
now runs execute, edit_files, and write_file calls once, and the
interrupt goroutine steps did not include cancelling those calls on
the workspace agent. The TODO that asked for this is removed.
Keep only comments that state a contract, invariant, or reason the code
does not show, and write them as Go doc comments.
edit_files and write_file said "not applied" and execute said "not run"
for the same outcome, and a killed execute said "canceled by the user".
Each outcome now has one message for all tools. The test field
agentless becomes notCancelable, since read_file does use the agent.
@mafredri
mafredri force-pushed the mafredri/codagt-757/02-agent-tool-call-records branch from adc0f83 to 18790dc Compare September 30, 2026 11:08
@mafredri
mafredri merged commit fa5bac6 into main Sep 30, 2026
49 of 51 checks passed
@mafredri
mafredri deleted the mafredri/codagt-757/02-agent-tool-call-records branch September 30, 2026 11:31
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 30, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants