Skip to content

feat: add dynamic tools support for chat API - #24036

Merged
kylecarbs merged 28 commits into
mainfrom
feat/chat-dynamic-tools
Apr 8, 2026
Merged

kylecarbs merged 28 commits into
mainfrom
feat/chat-dynamic-tools

Conversation

@kylecarbs

@kylecarbs kylecarbs commented Apr 5, 2026 •

Copy link
Copy Markdown
Member

Adds client-executed dynamic tools to the chat API. Dynamic tools are declared by the client at chat creation time, presented to the LLM alongside built-in tools, but executed by the client rather than chatd. This enables external systems (Slack bots, IDE extensions, Discord bots, CI/CD integrations) to plug custom tools into the LLM chat loop without modifying chatd's built-in tool set.

Modeled after OpenAI's Assistants API: the chat pauses with requires_action status when the LLM calls a dynamic tool, the client POSTs results back via POST /chats/{id}/tool-results, and the chat resumes.

See this example as a reference for how this is used. It's highly-configurable, which would enable creating chats from webhooks, periodically polling, or running as a Slackbot.

Design context

Architecture

The chatloop exits when it encounters dynamic tools and re-enters when results arrive. No blocking channels, no pubsub for tool results, no in-memory registry. The DB is the only coordination mechanism.

Phase 1 (chatloop):
  LLM response → execute built-in tools only →
  Persist(assistant + built-in results) →
  status = requires_action → chatloop exits

Phase 2 (POST /tool-results):
  Persist(dynamic tool results) →
  status = pending → wakeCh → chatloop re-enters

Validation (POST /tool-results)

  1. Chat status must be requires_action (409 if not)
  2. Read chat's dynamic_tools → set of dynamic tool names
  3. Read last assistant message → extract tool-call parts matching dynamic tool names
  4. Submitted tool_call_ids must match exactly (400 for missing/extra)
  5. Persist tool-result message parts, set status to pending, signal wake

Idempotency

Tool call IDs scoped per LLM step. State machine (requires_action → pending) is the guard. First POST wins, subsequent get 409.

Mixed tool calls

When the LLM calls both built-in and dynamic tools in one step, built-in tools execute immediately. Their results are persisted in phase 1. Dynamic tool results arrive via POST in phase 2. The LLM sees all results when the chatloop resumes.

🤖 Generated by Coder Agents

@kylecarbs
kylecarbs requested a review from hugodutka April 6, 2026 13:04
@kylecarbs
kylecarbs force-pushed the feat/chat-dynamic-tools branch 4 times, most recently from afe2170 to ee8b052 Compare April 7, 2026 14:20
@kylecarbs
kylecarbs marked this pull request as ready for review April 7, 2026 14:21
@coderagents

coderagents Bot commented Apr 7, 2026 •

Copy link
Copy Markdown
Contributor

Documentation Check

Updates Needed

  • docs/ai-coder/agents/chats-api.md — Add unsafe_dynamic_tools field (marked highly experimental) to the Create a chat request table; document the new POST /api/experimental/chats/{chat}/tool-results endpoint with its SubmitToolResultsRequest body (results[] with tool_call_id: string, tool_name: string, output: json, is_error: bool); add requires_action to the Chat statuses table; add action_required (with its tool_calls payload) to the Stream updates event types table; add action_required to the Watch all chats event kinds table; document the WatchChats SDK method and ChatWatchEvent/ChatWatchEventKind types for global chat event streaming.

    ⚠️ Still unaddressed — no docs changes found in this PR. Note: since previous review, ToolResult.Output is now json.RawMessage (not a plain string), and tool call fields are ToolCallID/ToolName (previously ID/Name).


Automated review via Coder Tasks

@kylecarbs kylecarbs left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid architecture — DB as sole coordination mechanism means replicas and restarts work by design, and the sentinel error exit from the chatloop is clean. The validation of pending tool-call IDs (exact-match, both directions) is well done.

This review contains findings that may need attention before merge.

2 P1, 3 P2, 4 nits across 9 inline comments.

🤖 Generated with Coder Agents

Comment thread coderd/exp_chats.go Resolved
Comment thread coderd/exp_chats.go Resolved
Comment thread coderd/exp_chats.go Resolved
Comment thread coderd/x/chatd/chatd.go Resolved
Comment thread coderd/x/chatd/chatd.go Resolved
Comment thread codersdk/chats.go Resolved
Comment thread codersdk/chats.go Resolved
Comment thread coderd/x/chatd/chatd.go Resolved
Comment thread coderd/x/chatd/dynamictool.go Resolved
Comment thread coderd/database/db2sdk/db2sdk.go Resolved
Comment thread coderd/exp_chats.go Resolved
Comment thread coderd/exp_chats.go Resolved
}
return xerrors.Errorf("persist step: %w", err)
}
return ErrDynamicToolCall

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If an agent exclusively uses dynamic tools, every step ends with ErrDynamicToolCall. The chatloop never reaches a step where shouldContinue is false (model naturally stops) or where the step completes normally. Compaction never triggers. The context window grows unbounded until the LLM provider rejects it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good observation. If a chat exclusively uses dynamic tools, the chatloop exits via ErrDynamicToolCall every step and never reaches the compaction check. The context window grows until the provider rejects it.

This is a real gap but it's not unique to dynamic tools — any chat with long tool-call chains has similar risk. Compaction should probably happen in PersistStep regardless of how the step ends, but that's a broader chatloop change. Deferring to a follow-up.

🤖 Generated with Coder Agents

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^I'm fixing this as part of this change. This is undeniably an issue.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b97fef8. Extracted a tryCompactOnExit helper and call it just before return ErrDynamicToolCall. Added TestRun_Compaction/TriggersOnDynamicToolExit to cover the scenario.

🤖 Generated with Coder Agents

@hugodutka hugodutka left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean architecture — using the DB as the sole coordination mechanism avoids in-memory state and cross-replica complexity. The TOCTOU protection in postChatToolResults and the ErrDynamicToolCall sentinel error are well done.

A couple things to look at: 1 P1, 4 P2, 3 P3, 5 nits across 13 inline comments.

This review contains a P1 finding that may need attention before merge.

🤖 Generated with Coder Agents

Comment thread coderd/x/chatd/chatd.go Resolved
Comment thread coderd/exp_chats.go Resolved
Comment thread coderd/exp_chats.go Resolved
Comment thread coderd/x/chatd/chatd.go Resolved
Comment thread coderd/exp_chats.go Resolved
Comment thread coderd/x/chatd/dynamictool.go Resolved
Comment thread coderd/x/chatd/chatloop/chatloop.go Resolved
Comment thread coderd/x/chatd/chatloop/chatloop.go Resolved
Comment thread coderd/x/chatd/chatd.go Resolved
Comment thread codersdk/chats.go Resolved

johnstcn commented Apr 7, 2026

Copy link
Copy Markdown
Member

🤖 Deep review (round 3) — cross-referenced against all prior comments and verified against commits on the branch

All P1s and P2s from rounds 1–2 (kylecarbs-agent, hugodutka) are resolved across 11a2865, 4c0690e, 68126c3, and b97fef8. Nice work.

What remains are test coverage gaps for the most complex code paths in this PR, plus a few minor typing issues.

2× P2, 4× P3.


P2 — No test for concurrent tool result submissions

The SELECT ... FOR UPDATE TOCTOU protection inside SubmitToolResults is the core correctness guarantee of this PR ("first POST wins, subsequent get 409"). There is no test that exercises it under contention.

TestSubmitToolResults/WrongStatus only tests sequential wrong-status. A test that spawns N goroutines all submitting valid results to the same requires_action chat would verify that exactly 1 succeeds (204) and N-1 get 409. This is the kind of bug that only manifests in production under webhook storms — exactly the use case dynamic tools are designed for.


P2 — No test for mixed built-in + dynamic tool calls in one LLM step

The chatd integration test (TestDynamicToolCallPausesAndResumes) explicitly notes:

"the built-in tools are never invoked because the only tool call targets our dynamic tool."

The mixed-call code path — where the LLM calls both a built-in tool and a dynamic tool in the same step — is the most complex logic in the chatloop changes:

  1. Partition toolCalls into builtinCalls and dynamicCalls
  2. Execute built-in calls inline
  3. Persist assistant blocks + built-in results
  4. Exit with ErrDynamicToolCall
  5. Client submits dynamic results
  6. Chatloop re-enters, LLM sees all results

None of this sequence is tested end-to-end. A mock LLM that returns both a built-in tool call (e.g. coder_workspace_exec) and a dynamic tool call in one streaming response would cover it.


P3 — No test for stale recovery of requires_action chats

TestRecoverStaleChatsPeriodically only creates running chats. The new GetStaleChats query OR clause for requires_action and the error-status recovery path ("Dynamic tool execution timed out") added in 4c0690e have zero test coverage. (Also flagged by hugodutka, still open.)

P3 — No unit tests for dynamicToolsFromSDK

No dynamictool_test.go exists on the branch. dynamicToolsFromSDK contains non-trivial logic: JSON Schema unmarshalling with silent fallback to empty parameters on parse failure (now logged at WARN per 11a2865). A malformed InputSchema producing a tool with nil parameter constraints is not tested. (Also flagged by hugodutka, still open.)

P3 — Generated TypeScript type for input_schema is Record<string, string>

JSON Schema property values are objects/arrays/booleans, not strings. TypeScript clients constructing DynamicTool objects with real schemas will get compile-time type errors. Likely a generator limitation, but this is a new public-facing type. (Also flagged by hugodutka, still open.)

P3 — ToolResult.Output typed as string but must be valid JSON

Doc comment was added in 4c0690e (good), but the type itself doesn't communicate the constraint. Changing to json.RawMessage would make it self-documenting and eliminate the json.Valid check. Acceptable as-is for an experimental API.


What's correct (verified against HEAD b97fef8)

  • TOCTOU protection: GetChatByIDForUpdate inside transaction with status re-check. First POST wins.
  • Tool name collision: dynamicToolDefs filtered before dynamicToolsFromSDK. Built-in tools always win.
  • State machine: requires_action → pending (tool results) or error (stale recovery). No illegal transitions.
  • Stale recovery: Both running (heartbeat) and requires_action (updated_at) freshness checks under FOR UPDATE lock.
  • Migration: Up is minimal. Down handles enum removal correctly (rename-create-cast-drop, drops dependent partial index first).
  • Compaction on dynamic tool exit: tryCompactOnExit added in b97fef8 prevents unbounded context growth for dynamic-tool-heavy chats.

@johnstcn johnstcn left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Human review comments below, I'd like to see the P2 bot findings above resolved before merge if possible.

Comment thread coderd/database/db2sdk/db2sdk.go Outdated
@@ -1608,6 +1608,7 @@ func Chat(c database.Chat, diffStatus *database.ChatDiffStatus, files []database
chat.LastInjectedContext = parts
}
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

@@ -407,7 +408,8 @@ INSERT INTO chats (
sqlc.narg('mode')::chat_mode,
@status::chat_status,
COALESCE(@mcp_server_ids::uuid[], '{}'::uuid[]),
COALESCE(sqlc.narg('labels')::jsonb, '{}'::jsonb)
COALESCE(sqlc.narg('labels')::jsonb, '{}'::jsonb),
sqlc.narg('dynamic_tools')::jsonb

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can see us ending up COALESCE-ing these like the above, so should we just make it DEFAULT '{}'::jsonb?

Comment thread codersdk/chats.go
// SubmitToolResults submits the results of dynamic tool calls for a chat
// that is in requires_action status.
func (c *ExperimentalClient) SubmitToolResults(ctx context.Context, chatID uuid.UUID, req SubmitToolResultsRequest) error {
res, err := c.Request(ctx, http.MethodPost, fmt.Sprintf("/api/experimental/chats/%s/tool-results", chatID), req)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought, non-blocking: submit with tool call ID in URL e.g. /api/experimental/chats/%s/tool-results/%s for easier greppability in logs?

Comment thread codersdk/chats.go
streamCtx, streamCancel := context.WithCancel(ctx)
events := make(chan ChatWatchEvent, 128)

go func() {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not a sync.WaitGroup here?

Comment thread codersdk/chats.go
Error *ChatStreamError `json:"error,omitempty"`
Retry *ChatStreamRetry `json:"retry,omitempty"`
QueuedMessages []ChatQueuedMessage `json:"queued_messages,omitempty"`
ActionRequired *ChatStreamActionRequired `json:"action_required,omitempty"`

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need this? Won't the tool call requests be in a ChatMessage or ChatMessagePart envelope?

Copy link
Copy Markdown
Member Author

All P2/P3 test coverage gaps from round 3 addressed in 7435c1f:

Finding Test Status
P2 Concurrent tool result submissions TestSubmitToolResultsConcurrency — 10 goroutines race, exactly 1 wins, 9 get ToolResultStatusConflictError ✅
P2 Mixed built-in + dynamic tool calls TestDynamicToolCallMixedWithBuiltIn — LLM returns read_file + my_dynamic_tool in one step, verifies built-in executed before parking, client submits dynamic result, chatloop resumes ✅
P3 Stale recovery of requires_action TestRecoverStaleRequiresActionChat — backdates updated_at via raw SQL, verifies recovery sets error status with "Dynamic tool execution timed out" ✅
P3 dynamicToolsFromSDK unit tests TestDynamicToolsFromSDK in dynamictool_internal_test.go — 6 subtests covering valid schema, empty/nil, malformed JSON fallback, multiple tools, schema without properties ✅
P3 TS Record<string, string> Generator limitation — acknowledged, deferred
P3 ToolResult.Output as string Acceptable as-is per review — doc comment already present

🤖 Generated with Coder Agents

@johnstcn johnstcn left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No further blockers from me, so approving.
I would still like to see responses to my human review.

@hugodutka hugodutka left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean approach to the dynamic tools feature. The DB-as-coordination design (exit chatloop → persist → re-enter) is well-considered, and the FOR UPDATE row lock in SubmitToolResults ensures clean exactly-once semantics. The mixed tool call handling, stale recovery to error (not pending), and the ErrDynamicToolCall sentinel are all solid design choices. Test coverage is strong overall.

This review contains findings that may need attention before merge.

2 P1, 5 P2, 2 P3, 3 Obs across 12 inline comments.

🤖 Generated with Coder Agents

Comment thread coderd/x/chatd/chatd.go Resolved
Comment thread coderd/x/chatd/chatd.go Resolved
Comment thread coderd/x/chatd/chatd.go Resolved
Comment thread coderd/x/chatd/chatd.go Resolved
Comment thread coderd/x/chatd/chatd.go Resolved
Comment thread coderd/exp_chats.go Resolved
Comment thread coderd/x/chatd/chatd.go Resolved
Comment thread codersdk/chats.go Resolved
Comment thread coderd/exp_chats_test.go Resolved
Comment thread coderd/x/chatd/chatd_test.go Resolved

@kylecarbs kylecarbs left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did a manual review. It seems good to me.

kylecarbs added 11 commits April 8, 2026 13:35
Adds the infrastructure for client-executed dynamic tools in the chat
API. Dynamic tools are declared by the client at chat creation time,
presented to the LLM alongside built-in tools, but executed by the
client rather than chatd.

- Add `dynamic_tools` JSONB column to chats table
- Add `requires_action` chat status for when dynamic tools need results
- Add `POST /api/experimental/chats/{id}/tool-results` endpoint
- Add codersdk types: DynamicTool, SubmitToolResultsRequest, ToolResult
- Add `action_required` stream event type
- Wire dynamic tools through chat creation flow
- Validate tool results match pending dynamic tool calls exactly
Complete the dynamic tools implementation with chatloop integration:

- Add dynamicTool wrapper implementing fantasy.AgentTool
- Include dynamic tools in LLM tool list via dynamicToolsFromSDK()
- Partition tool calls into built-in vs dynamic in executeTools()
- Two-phase persistence: persist assistant + built-in results, then
  exit with ErrDynamicToolCall when dynamic tools are called
- Handle ErrDynamicToolCall in processChat: set requires_action status
  and publish action_required stream event
- Add GetRequiresActionChats query for timeout recovery
- Extend recoverStaleChats to handle timed-out dynamic tool chats
…ation

Replace full message history fetch with a single-row lookup of the
last assistant message when validating dynamic tool results.
Tests the POST /chats/{id}/tool-results endpoint:
- Success: submits results for 2 dynamic tool calls, verifies status
  transitions and one tool-result message per result in the DB
- WrongStatus: verifies 409 when chat is not in requires_action
- MissingResult: verifies 400 when not all tool call IDs are covered
- UnexpectedResult: verifies 400 for unknown tool call IDs

Also fixes tool result persistence to insert one message per result,
matching the existing chatd pattern where each built-in tool result
is its own tool-role message row.
TestDynamicToolCallPausesAndResumes exercises the full dynamic tools
lifecycle through the chatloop with a mock LLM:

1. Mock LLM returns a tool call targeting a dynamic tool
2. Chatloop pauses with requires_action status
3. Tool results are submitted (mimicking POST /tool-results)
4. Chatloop resumes and completes
5. Verifies the dynamic tool appeared in the LLM tool list
6. Verifies the tool result was included in the follow-up LLM call
- Fix migration down: reorder statements so UPDATE runs before enum
  rename, add IF EXISTS to DROP COLUMN
- Fix TestChat_AllFieldsPopulated: populate DynamicTools in test input
- Fix TestMethodTestSuite: add dbauthz test for GetRequiresActionChats
…igration

The idx_chats_pending partial index WHERE clause references the
chat_status enum type. When the enum is renamed to chat_status_old
during the down migration, PostgreSQL cannot compare the renamed
type with the new type, causing 'operator does not exist:
chat_status = chat_status_old'.
Replace the custom codersdk.DynamicTool struct with mcp.Tool from
github.com/mark3labs/mcp-go/mcp. This aligns the dynamic tools API
with the MCP standard tool definition format and eliminates a
redundant type. The unused TimeoutSeconds field is dropped.
Merge GetRequiresActionChats into GetStaleChats with an OR clause
covering both running (heartbeat expired) and requires_action
(updated_at expired) chats. Consolidate the two recovery functions
into a single recoverStaleChats that branches on status.
…eam filter

The stream merge goroutine filters local channel events when pubsub
is active, only forwarding message_part and status types. The
action_required event introduced for dynamic tools is published
exclusively on the local stream and was silently dropped by this
filter, so clients never received it.

Add ChatStreamEventTypeActionRequired to the filter alongside the
other transient event types.
…tream

Extends the /chats/watch pubsub channel to deliver action_required
events with tool call payloads, enabling global watchers (e.g. a
Slackbot) to react to dynamic tool calls without streaming each
chat individually.

- Add ChatEventKindActionRequired and ToolCalls field to ChatEvent
- Add publishChatActionRequired in chatd to broadcast on global pubsub
- Add ChatWatchEvent types and WatchChats SDK client method
kylecarbs added 14 commits April 8, 2026 13:36
…neration

Adds ToolCall, ToolResponse, DynamicTool, and NewDynamicTool[T] to
codersdk/chats.go. DynamicTool pairs a tool definition (Name,
Description, InputSchema) with a client-side Handler. JSON schema is
derived from T using invopop/jsonschema at init time.

CreateChatRequest.DynamicTools changed from []mcp.Tool to
[]DynamicTool — wire format is compatible since the server
unmarshals into mcp.Tool which shares the same JSON field names.

Removes the ToolRegistry/ToolHandler API from dynamictools.go.
Required as a direct import by the DynamicTool schema generation.
…cTools

Mark the field as highly experimental and subject to change.
Too early to expose stored dynamic tools on the response. The
field can be added back once the API stabilizes.
…aming)

- P1: Move status check inside transaction in postChatToolResults
  using GetChatByIDForUpdate + re-check. Return 409 on conflict.
- P2: Use codersdk.DynamicTool consistently (remove mcp.Tool).
- P2: Validate json.Valid on ToolResult.Output before storing.
- P2: Check dynamic tool name collisions with built-in tools in
  chatloop, log warning and prefer built-in.
- P2: Validate dynamic tool names are non-empty and unique at
  creation time.
- Nit: Re-verify updated_at in stale recovery for requires_action.
- Nit: Extract pendingToStreamToolCalls helper.
- Nit: Rename ToolCall/ToolResponse to DynamicToolCall/DynamicToolResponse
  with proper json tags.
- Nit: Change inputSchema to input_schema for snake_case consistency.
- Nit: Log warning on InputSchema parse failure in dynamicToolsFromSDK.
- P1: Filter dynamicToolDefs to exclude colliding names before
  passing to dynamicToolsFromSDK (prevents built-in tool override).
- P2: Return 500 on ParseContent failure instead of silently
  falling through with empty pendingCallIDs.
- P2: Reject duplicate ToolCallIDs in submitted results (400).
- P2: Add MaxBytesReader to postChatToolResults endpoint.
- P2: Set requires_action timeout recovery to error status
  (not pending) to avoid dangling tool calls.
- P3: Add doc comment on ToolResult.Output explaining JSON
  constraint.
- Nit: Separate PendingToolCall doc comment from PersistedStep.
- Nit: Pre-allocate pending slice with known length.
- Nit: Add clarifying comment on dynamicToolNames post-collision.
- Nit: Reword DynamicTool doc to describe dual client/server role.
Move all DB logic from postChatToolResults handler into a new
Server.SubmitToolResults method, following the existing pattern
of CreateChat/SendMessage/EditMessage.

- Move GetLastChatMessageByRole inside the transaction.
- Make SignalWake private again (signalWake).
- Handler is now thin: parse request, delegate, map errors.
- Define ToolResultValidationError and ToolResultStatusConflictError
  for clean HTTP status mapping.
When the chatloop exits via ErrDynamicToolCall it bypasses both the
mid-loop and post-run compaction checks. Chats that exclusively use
dynamic tools could grow unbounded until the provider rejects them.

Extract tryCompactOnExit helper and invoke it just before returning
ErrDynamicToolCall so that the context window is compacted when the
usage threshold is exceeded.
Address review feedback (johnstcn round 3):

- P2: TestSubmitToolResultsConcurrency — verify exactly 1 of N
  concurrent submissions succeeds under the FOR UPDATE lock
- P2: TestDynamicToolCallMixedWithBuiltIn — verify the mixed
  built-in + dynamic tool call path (partition, execute built-in,
  exit with ErrDynamicToolCall, resume after results)
- P3: TestRecoverStaleRequiresActionChat — verify stale
  requires_action chats recover to error status with the timeout
  message
- P3: TestDynamicToolsFromSDK — unit tests for schema parsing,
  malformed input fallback, empty/nil inputs
Address review feedback (hugodutka review 4074938555):

P1: Add requires_action to shouldQueueUserMessage so SendMessage
    during dynamic tool execution queues instead of corrupting
    conversation state with dangling tool calls.

P1: Guard InterruptChat for requires_action chats — insert
    synthetic error tool-result messages before transitioning to
    waiting, preventing orphaned tool calls in LLM history.

P2: Move action_required stream event from runChat to
    processChat's defer block, after UpdateChatStatus commits.
    Prevents race where fast client reacts before status is
    visible in DB.

P2: Insert synthetic error tool-result messages during stale
    recovery of requires_action chats. Extract shared
    insertSyntheticToolResultsTx helper used by both InterruptChat
    and recoverStaleChats.

P2: Add MaxBytesReader to postChats handler to cap request body
    size when dynamic tools with large schemas are present.

P2: Add TestPostChats_DynamicToolValidation with sub-tests for
    max tool count, empty name, and duplicate name validation.

P2: Add TestNewDynamicTool SDK tests covering schema generation,
    argument deserialization, and invalid JSON error path.

P3: Add InvalidJSONOutput and DuplicateToolCallID sub-tests to
    TestPostChatToolResults (was TestSubmitToolResults).

P3: Fix concurrency test barrier to use close(ready) pattern
    for proper N-way goroutine synchronization.
@kylecarbs
kylecarbs force-pushed the feat/chat-dynamic-tools branch from 6abee54 to 3b67ccd Compare April 8, 2026 13:38

@hugodutka hugodutka left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean architecture — the DB-only coordination with exit/re-enter chatloop, FOR UPDATE row locks in SubmitToolResults, and synthetic tool-result insertion for LLM history consistency are all well-designed. The stale recovery extension and collision detection are solid too.

This review contains a P1 finding that may need attention before merge. Plus 4 P2, 7 P3, and 5 nits across 13 inline comments.

🤖 Generated with Coder Agents

Comment thread coderd/x/chatd/chatd.go Resolved
Comment thread coderd/x/chatd/chatd.go Resolved
Comment thread codersdk/chats.go Resolved
Comment thread coderd/x/chatd/chatd.go Resolved
Comment thread coderd/x/chatd/chatd.go Resolved
Comment thread coderd/exp_chats.go Resolved
Comment thread coderd/x/chatd/chatloop/chatloop.go Resolved
Comment thread coderd/x/chatd/chatd_test.go Resolved
Comment thread codersdk/chats.go Resolved

@hugodutka hugodutka left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good to merge after the last round of agent review is addressed

P1: InterruptChat now wraps requires_action path in InTx with
GetChatByIDForUpdate, preventing race with SubmitToolResults.

P2: SubmitToolResults excludes already-handled tool-call IDs
(built-in results) from the pending set, fixing collision
filtering mismatch with runChat. Client-submitted tool results
now include ToolName for consistency with synthetic results.

P3: Extract parseDynamicToolNames helper to centralize the
repeated unmarshal+map pattern. Down migration sets status to
error instead of pending. Internal error details no longer
leak through the default error handler.

Nits: Fix orphaned PersistedStep doc comment, modernize
concurrency test to for-range+wg.Go, clarify input_schema
JSON key comment.
@kylecarbs
kylecarbs force-pushed the feat/chat-dynamic-tools branch from 3b67ccd to 3248dfb Compare April 8, 2026 14:39
- ToolResult.Output changed from string to json.RawMessage,
  making the JSON contract self-documenting and eliminating
  the runtime json.Valid cast.
- DynamicToolCall fields renamed from ID/Name to
  ToolCallID/ToolName for consistency with ChatStreamToolCall.
- Added EmptyResults and NotFoundForDifferentUser subtests
  to TestPostChatToolResults.
…shal

After changing ToolResult.Output to json.RawMessage, the SDK client
rejects invalid JSON during json.Marshal before the request reaches
the server. This test must bypass the SDK to verify server-side
validation of malformed JSON from non-SDK clients.
@kylecarbs
kylecarbs merged commit b969d66 into main Apr 8, 2026
48 of 50 checks passed
@kylecarbs
kylecarbs deleted the feat/chat-dynamic-tools branch April 8, 2026 15:54
@github-actions github-actions Bot locked and limited conversation to collaborators Apr 8, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants