Repository navigation
feat: add dynamic tools support for chat API - #24036
Conversation
afe2170 to
ee8b052
Compare
Documentation CheckUpdates Needed
Automated review via Coder Tasks |
kylecarbs
left a comment
There was a problem hiding this comment.
Solid architecture — DB as sole coordination mechanism means replicas and restarts work by design, and the sentinel error exit from the chatloop is clean. The validation of pending tool-call IDs (exact-match, both directions) is well done.
This review contains findings that may need attention before merge.
2 P1, 3 P2, 4 nits across 9 inline comments.
🤖 Generated with Coder Agents
| } | ||
| return xerrors.Errorf("persist step: %w", err) | ||
| } | ||
| return ErrDynamicToolCall |
There was a problem hiding this comment.
If an agent exclusively uses dynamic tools, every step ends with ErrDynamicToolCall. The chatloop never reaches a step where shouldContinue is false (model naturally stops) or where the step completes normally. Compaction never triggers. The context window grows unbounded until the LLM provider rejects it.
There was a problem hiding this comment.
Good observation. If a chat exclusively uses dynamic tools, the chatloop exits via ErrDynamicToolCall every step and never reaches the compaction check. The context window grows until the provider rejects it.
This is a real gap but it's not unique to dynamic tools — any chat with long tool-call chains has similar risk. Compaction should probably happen in PersistStep regardless of how the step ends, but that's a broader chatloop change. Deferring to a follow-up.
🤖 Generated with Coder Agents
There was a problem hiding this comment.
^I'm fixing this as part of this change. This is undeniably an issue.
There was a problem hiding this comment.
Fixed in b97fef8. Extracted a tryCompactOnExit helper and call it just before return ErrDynamicToolCall. Added TestRun_Compaction/TriggersOnDynamicToolExit to cover the scenario.
🤖 Generated with Coder Agents
hugodutka
left a comment
There was a problem hiding this comment.
Clean architecture — using the DB as the sole coordination mechanism avoids in-memory state and cross-replica complexity. The TOCTOU protection in postChatToolResults and the ErrDynamicToolCall sentinel error are well done.
A couple things to look at: 1 P1, 4 P2, 3 P3, 5 nits across 13 inline comments.
This review contains a P1 finding that may need attention before merge.
🤖 Generated with Coder Agents
|
🤖 Deep review (round 3) — cross-referenced against all prior comments and verified against commits on the branch All P1s and P2s from rounds 1–2 (kylecarbs-agent, hugodutka) are resolved across What remains are test coverage gaps for the most complex code paths in this PR, plus a few minor typing issues. 2× P2, 4× P3. P2 — No test for concurrent tool result submissionsThe
P2 — No test for mixed built-in + dynamic tool calls in one LLM stepThe chatd integration test (
The mixed-call code path — where the LLM calls both a built-in tool and a dynamic tool in the same step — is the most complex logic in the chatloop changes:
None of this sequence is tested end-to-end. A mock LLM that returns both a built-in tool call (e.g. P3 — No test for stale recovery of
|
| @@ -1608,6 +1608,7 @@ func Chat(c database.Chat, diffStatus *database.ChatDiffStatus, files []database | |||
| chat.LastInjectedContext = parts | |||
| } | |||
| } | |||
|
|
|||
| @@ -407,7 +408,8 @@ INSERT INTO chats ( | |||
| sqlc.narg('mode')::chat_mode, | |||
| @status::chat_status, | |||
| COALESCE(@mcp_server_ids::uuid[], '{}'::uuid[]), | |||
| COALESCE(sqlc.narg('labels')::jsonb, '{}'::jsonb) | |||
| COALESCE(sqlc.narg('labels')::jsonb, '{}'::jsonb), | |||
| sqlc.narg('dynamic_tools')::jsonb | |||
There was a problem hiding this comment.
I can see us ending up COALESCE-ing these like the above, so should we just make it DEFAULT '{}'::jsonb?
| // SubmitToolResults submits the results of dynamic tool calls for a chat | ||
| // that is in requires_action status. | ||
| func (c *ExperimentalClient) SubmitToolResults(ctx context.Context, chatID uuid.UUID, req SubmitToolResultsRequest) error { | ||
| res, err := c.Request(ctx, http.MethodPost, fmt.Sprintf("/api/experimental/chats/%s/tool-results", chatID), req) |
There was a problem hiding this comment.
thought, non-blocking: submit with tool call ID in URL e.g. /api/experimental/chats/%s/tool-results/%s for easier greppability in logs?
| streamCtx, streamCancel := context.WithCancel(ctx) | ||
| events := make(chan ChatWatchEvent, 128) | ||
|
|
||
| go func() { |
| Error *ChatStreamError `json:"error,omitempty"` | ||
| Retry *ChatStreamRetry `json:"retry,omitempty"` | ||
| QueuedMessages []ChatQueuedMessage `json:"queued_messages,omitempty"` | ||
| ActionRequired *ChatStreamActionRequired `json:"action_required,omitempty"` |
There was a problem hiding this comment.
Do we really need this? Won't the tool call requests be in a ChatMessage or ChatMessagePart envelope?
|
All P2/P3 test coverage gaps from round 3 addressed in 7435c1f:
|
johnstcn
left a comment
There was a problem hiding this comment.
No further blockers from me, so approving.
I would still like to see responses to my human review.
hugodutka
left a comment
There was a problem hiding this comment.
Clean approach to the dynamic tools feature. The DB-as-coordination design (exit chatloop → persist → re-enter) is well-considered, and the FOR UPDATE row lock in SubmitToolResults ensures clean exactly-once semantics. The mixed tool call handling, stale recovery to error (not pending), and the ErrDynamicToolCall sentinel are all solid design choices. Test coverage is strong overall.
This review contains findings that may need attention before merge.
2 P1, 5 P2, 2 P3, 3 Obs across 12 inline comments.
🤖 Generated with Coder Agents
kylecarbs
left a comment
There was a problem hiding this comment.
Did a manual review. It seems good to me.
Adds the infrastructure for client-executed dynamic tools in the chat
API. Dynamic tools are declared by the client at chat creation time,
presented to the LLM alongside built-in tools, but executed by the
client rather than chatd.
- Add `dynamic_tools` JSONB column to chats table
- Add `requires_action` chat status for when dynamic tools need results
- Add `POST /api/experimental/chats/{id}/tool-results` endpoint
- Add codersdk types: DynamicTool, SubmitToolResultsRequest, ToolResult
- Add `action_required` stream event type
- Wire dynamic tools through chat creation flow
- Validate tool results match pending dynamic tool calls exactly
Complete the dynamic tools implementation with chatloop integration: - Add dynamicTool wrapper implementing fantasy.AgentTool - Include dynamic tools in LLM tool list via dynamicToolsFromSDK() - Partition tool calls into built-in vs dynamic in executeTools() - Two-phase persistence: persist assistant + built-in results, then exit with ErrDynamicToolCall when dynamic tools are called - Handle ErrDynamicToolCall in processChat: set requires_action status and publish action_required stream event - Add GetRequiresActionChats query for timeout recovery - Extend recoverStaleChats to handle timed-out dynamic tool chats
…ation Replace full message history fetch with a single-row lookup of the last assistant message when validating dynamic tool results.
Tests the POST /chats/{id}/tool-results endpoint:
- Success: submits results for 2 dynamic tool calls, verifies status
transitions and one tool-result message per result in the DB
- WrongStatus: verifies 409 when chat is not in requires_action
- MissingResult: verifies 400 when not all tool call IDs are covered
- UnexpectedResult: verifies 400 for unknown tool call IDs
Also fixes tool result persistence to insert one message per result,
matching the existing chatd pattern where each built-in tool result
is its own tool-role message row.
TestDynamicToolCallPausesAndResumes exercises the full dynamic tools lifecycle through the chatloop with a mock LLM: 1. Mock LLM returns a tool call targeting a dynamic tool 2. Chatloop pauses with requires_action status 3. Tool results are submitted (mimicking POST /tool-results) 4. Chatloop resumes and completes 5. Verifies the dynamic tool appeared in the LLM tool list 6. Verifies the tool result was included in the follow-up LLM call
- Fix migration down: reorder statements so UPDATE runs before enum rename, add IF EXISTS to DROP COLUMN - Fix TestChat_AllFieldsPopulated: populate DynamicTools in test input - Fix TestMethodTestSuite: add dbauthz test for GetRequiresActionChats
…igration The idx_chats_pending partial index WHERE clause references the chat_status enum type. When the enum is renamed to chat_status_old during the down migration, PostgreSQL cannot compare the renamed type with the new type, causing 'operator does not exist: chat_status = chat_status_old'.
Replace the custom codersdk.DynamicTool struct with mcp.Tool from github.com/mark3labs/mcp-go/mcp. This aligns the dynamic tools API with the MCP standard tool definition format and eliminates a redundant type. The unused TimeoutSeconds field is dropped.
Merge GetRequiresActionChats into GetStaleChats with an OR clause covering both running (heartbeat expired) and requires_action (updated_at expired) chats. Consolidate the two recovery functions into a single recoverStaleChats that branches on status.
…eam filter The stream merge goroutine filters local channel events when pubsub is active, only forwarding message_part and status types. The action_required event introduced for dynamic tools is published exclusively on the local stream and was silently dropped by this filter, so clients never received it. Add ChatStreamEventTypeActionRequired to the filter alongside the other transient event types.
…tream Extends the /chats/watch pubsub channel to deliver action_required events with tool call payloads, enabling global watchers (e.g. a Slackbot) to react to dynamic tool calls without streaming each chat individually. - Add ChatEventKindActionRequired and ToolCalls field to ChatEvent - Add publishChatActionRequired in chatd to broadcast on global pubsub - Add ChatWatchEvent types and WatchChats SDK client method
…neration Adds ToolCall, ToolResponse, DynamicTool, and NewDynamicTool[T] to codersdk/chats.go. DynamicTool pairs a tool definition (Name, Description, InputSchema) with a client-side Handler. JSON schema is derived from T using invopop/jsonschema at init time. CreateChatRequest.DynamicTools changed from []mcp.Tool to []DynamicTool — wire format is compatible since the server unmarshals into mcp.Tool which shares the same JSON field names. Removes the ToolRegistry/ToolHandler API from dynamictools.go.
Required as a direct import by the DynamicTool schema generation.
…namic tool handling
…cTools Mark the field as highly experimental and subject to change.
Too early to expose stored dynamic tools on the response. The field can be added back once the API stabilizes.
…aming) - P1: Move status check inside transaction in postChatToolResults using GetChatByIDForUpdate + re-check. Return 409 on conflict. - P2: Use codersdk.DynamicTool consistently (remove mcp.Tool). - P2: Validate json.Valid on ToolResult.Output before storing. - P2: Check dynamic tool name collisions with built-in tools in chatloop, log warning and prefer built-in. - P2: Validate dynamic tool names are non-empty and unique at creation time. - Nit: Re-verify updated_at in stale recovery for requires_action. - Nit: Extract pendingToStreamToolCalls helper. - Nit: Rename ToolCall/ToolResponse to DynamicToolCall/DynamicToolResponse with proper json tags. - Nit: Change inputSchema to input_schema for snake_case consistency. - Nit: Log warning on InputSchema parse failure in dynamicToolsFromSDK.
- P1: Filter dynamicToolDefs to exclude colliding names before passing to dynamicToolsFromSDK (prevents built-in tool override). - P2: Return 500 on ParseContent failure instead of silently falling through with empty pendingCallIDs. - P2: Reject duplicate ToolCallIDs in submitted results (400). - P2: Add MaxBytesReader to postChatToolResults endpoint. - P2: Set requires_action timeout recovery to error status (not pending) to avoid dangling tool calls. - P3: Add doc comment on ToolResult.Output explaining JSON constraint. - Nit: Separate PendingToolCall doc comment from PersistedStep. - Nit: Pre-allocate pending slice with known length. - Nit: Add clarifying comment on dynamicToolNames post-collision. - Nit: Reword DynamicTool doc to describe dual client/server role.
Move all DB logic from postChatToolResults handler into a new Server.SubmitToolResults method, following the existing pattern of CreateChat/SendMessage/EditMessage. - Move GetLastChatMessageByRole inside the transaction. - Make SignalWake private again (signalWake). - Handler is now thin: parse request, delegate, map errors. - Define ToolResultValidationError and ToolResultStatusConflictError for clean HTTP status mapping.
When the chatloop exits via ErrDynamicToolCall it bypasses both the mid-loop and post-run compaction checks. Chats that exclusively use dynamic tools could grow unbounded until the provider rejects them. Extract tryCompactOnExit helper and invoke it just before returning ErrDynamicToolCall so that the context window is compacted when the usage threshold is exceeded.
Address review feedback (johnstcn round 3): - P2: TestSubmitToolResultsConcurrency — verify exactly 1 of N concurrent submissions succeeds under the FOR UPDATE lock - P2: TestDynamicToolCallMixedWithBuiltIn — verify the mixed built-in + dynamic tool call path (partition, execute built-in, exit with ErrDynamicToolCall, resume after results) - P3: TestRecoverStaleRequiresActionChat — verify stale requires_action chats recover to error status with the timeout message - P3: TestDynamicToolsFromSDK — unit tests for schema parsing, malformed input fallback, empty/nil inputs
Address review feedback (hugodutka review 4074938555):
P1: Add requires_action to shouldQueueUserMessage so SendMessage
during dynamic tool execution queues instead of corrupting
conversation state with dangling tool calls.
P1: Guard InterruptChat for requires_action chats — insert
synthetic error tool-result messages before transitioning to
waiting, preventing orphaned tool calls in LLM history.
P2: Move action_required stream event from runChat to
processChat's defer block, after UpdateChatStatus commits.
Prevents race where fast client reacts before status is
visible in DB.
P2: Insert synthetic error tool-result messages during stale
recovery of requires_action chats. Extract shared
insertSyntheticToolResultsTx helper used by both InterruptChat
and recoverStaleChats.
P2: Add MaxBytesReader to postChats handler to cap request body
size when dynamic tools with large schemas are present.
P2: Add TestPostChats_DynamicToolValidation with sub-tests for
max tool count, empty name, and duplicate name validation.
P2: Add TestNewDynamicTool SDK tests covering schema generation,
argument deserialization, and invalid JSON error path.
P3: Add InvalidJSONOutput and DuplicateToolCallID sub-tests to
TestPostChatToolResults (was TestSubmitToolResults).
P3: Fix concurrency test barrier to use close(ready) pattern
for proper N-way goroutine synchronization.
6abee54 to
3b67ccd
Compare
hugodutka
left a comment
There was a problem hiding this comment.
Clean architecture — the DB-only coordination with exit/re-enter chatloop, FOR UPDATE row locks in SubmitToolResults, and synthetic tool-result insertion for LLM history consistency are all well-designed. The stale recovery extension and collision detection are solid too.
This review contains a P1 finding that may need attention before merge. Plus 4 P2, 7 P3, and 5 nits across 13 inline comments.
🤖 Generated with Coder Agents
hugodutka
left a comment
There was a problem hiding this comment.
good to merge after the last round of agent review is addressed
P1: InterruptChat now wraps requires_action path in InTx with GetChatByIDForUpdate, preventing race with SubmitToolResults. P2: SubmitToolResults excludes already-handled tool-call IDs (built-in results) from the pending set, fixing collision filtering mismatch with runChat. Client-submitted tool results now include ToolName for consistency with synthetic results. P3: Extract parseDynamicToolNames helper to centralize the repeated unmarshal+map pattern. Down migration sets status to error instead of pending. Internal error details no longer leak through the default error handler. Nits: Fix orphaned PersistedStep doc comment, modernize concurrency test to for-range+wg.Go, clarify input_schema JSON key comment.
3b67ccd to
3248dfb
Compare
- ToolResult.Output changed from string to json.RawMessage, making the JSON contract self-documenting and eliminating the runtime json.Valid cast. - DynamicToolCall fields renamed from ID/Name to ToolCallID/ToolName for consistency with ChatStreamToolCall. - Added EmptyResults and NotFoundForDifferentUser subtests to TestPostChatToolResults.
…shal After changing ToolResult.Output to json.RawMessage, the SDK client rejects invalid JSON during json.Marshal before the request reaches the server. This test must bypass the SDK to verify server-side validation of malformed JSON from non-SDK clients.
Adds client-executed dynamic tools to the chat API. Dynamic tools are declared by the client at chat creation time, presented to the LLM alongside built-in tools, but executed by the client rather than chatd. This enables external systems (Slack bots, IDE extensions, Discord bots, CI/CD integrations) to plug custom tools into the LLM chat loop without modifying chatd's built-in tool set.
Modeled after OpenAI's Assistants API: the chat pauses with
requires_actionstatus when the LLM calls a dynamic tool, the client POSTs results back viaPOST /chats/{id}/tool-results, and the chat resumes.See this example as a reference for how this is used. It's highly-configurable, which would enable creating chats from webhooks, periodically polling, or running as a Slackbot.
Design context
Architecture
The chatloop exits when it encounters dynamic tools and re-enters when results arrive. No blocking channels, no pubsub for tool results, no in-memory registry. The DB is the only coordination mechanism.
Validation (POST /tool-results)
requires_action(409 if not)dynamic_tools→ set of dynamic tool namespending, signal wakeIdempotency
Tool call IDs scoped per LLM step. State machine (
requires_action→pending) is the guard. First POST wins, subsequent get 409.Mixed tool calls
When the LLM calls both built-in and dynamic tools in one step, built-in tools execute immediately. Their results are persisted in phase 1. Dynamic tool results arrive via POST in phase 2. The LLM sees all results when the chatloop resumes.