Repository navigation
fix: use one route-aware media rule for files and tool results - #31384
Conversation
Docs previewCheck off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here. |
8845f94 to
b8b2c52
Compare
b8b2c52 to
8356501
Compare
|
@codex review |
|
@codex security review |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8356501f8b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
8356501 to
0cb6bf3
Compare
|
@codex review |
|
Codex Review: Didn't find any major issues. Can't wait for the next one! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
0cb6bf3 to
8d99e82
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8d99e82fa9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review re: Preserve the Copilot provider in media routing The code path is real, but this is not worth fixing here.
|
|
Codex Review: Didn't find any major issues. Can't wait for the next one! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
8d99e82 to
a0e0337
Compare
a0e0337 to
3d5c67b
Compare
|
@codex review |
|
Codex Review: Didn't find any major issues. Keep it up! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
…l results
chatprovider.Model now carries the configured provider type, set in
newModel. AcceptsFilePartMediaType reads a table keyed on the route
(configured provider and transport), and InlineImageCapBytes is a Model
method that applies to images only. ToolResultMediaOmission is removed.
replaceUnsupportedMedia replaces both replaceUnsupportedToolMedia and
replaceUnsupportedFileParts. It handles user file parts and tool result
media with the same rule, and the turn prompt now filters the pending
tail too. A PDF at the image cap no longer gets the image cap note.
Tool result media that the route cannot take now gets the note that
fantasy's providers write ("The tool returned <type> content, which
cannot be displayed."). Each note comes before the accompanying text,
because it takes the place of the media.
…s in MCP tool results
…utes
Azure OpenAI Chat Completions rejects file content parts with HTTP 400
("This model does not support file content types"), and Gemini's
OpenAI-compatible endpoint does not process them. chatd reaches both
providers through the OpenAI-compatible client, so the route rule now
rejects PDFs on them. User attachments and tool results get the usual
text note instead of a failed request or a silently dropped file.
…tachments The tool result media rules apply to org servers too, not only inline servers, so they move to a Tool results section. The section names the one image or PDF limit per result and the size notes. The File attachments text said that a rejected part is downgraded to text. Only text files are. Other rejected files, such as PDFs on Azure OpenAI and Google, reach the model as a note that names the file.
The OpenAI transport bullet said that the file part rule gates only attachments and that a mismatch drops text attachments. The rule now also gates tool-result media, and neither OpenAI transport takes text file parts, so audio is the only type that the transports treat differently. The compaction bullet now also names tool-result media, which the compaction prompt already replaces.
3d5c67b to
409b224
Compare
Part of [CODAGT-1430](https://linear.app/codercom/issue/CODAGT-1430). Stacked on #31384. Workspace MCP tool results go through the agent's `convertResult`, which turns an embedded resource into a `[embedded resource: ...]` string and drops structured content, so a PDF from a workspace MCP server never reaches the model. Remote MCP tools already go through `mcpclient`'s converter, which keeps both. When a call-tool request sets `include_result`, the agent now returns the MCP `CallToolResult` as-is in `Result` and leaves `Content` empty. chattool always sets it and converts the result with `mcpclient.ConvertCallResult`, so workspace and remote MCP tools share one converter. Older agents ignore the flag and send the legacy `Content`, which we adapt to MCP content and pass to the same converter. The workspace MCP docs get a short Tool results section too.
Part of CODAGT-1430 Relates to CODAGT-1541 Stacked on #31743. This bumps fantasy to pick up coder/fantasy#84 and coder/fantasy#85. A Chat Completions tool message can only hold text, so fantasy sends tool result media (images, audio, and now PDFs from #31384) in a `role: "user"` message right after it. That gives whatever the tool returned the authority of the user's own words, which Codex flagged on #31383 as a prompt injection risk for MCP tools that return untrusted files. coder/fantasy#84 starts that user message with a text part naming the tool call and saying the content is tool output, not instructions from the user. It only changes the Chat Completions paths (`openai` without the Responses API, `openaicompat`, OpenRouter and Vercel); Anthropic and Responses keep the media inside the tool result. AI Bridge records the last user message's text as the prompt, so on the step after a tool returns media, it'll now record the label. I think that's fine for now. coder/fantasy#85 fixes the output token limit field. The OpenAI provider sent `max_tokens` unless the model ID looked like a reasoning model, and reasoning models reject it: ``` Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead. ``` That breaks any reasoning model whose ID fantasy doesn't recognise, e.g. an Azure deployment name, which doesn't say what model it serves (Azure providers start using this client in #31952). The OpenAI provider now sends `max_completion_tokens` on every Chat Completions request, which is what OpenAI deprecated `max_tokens` in favour of. `openaicompat`, OpenRouter and Vercel keep `max_tokens`, so self-hosted servers behind the OpenAI Compatible provider are unaffected. I've added a test that checks the field on each route, and the models docs no longer say Max Completion Tokens is only for reasoning models.
Part of CODAGT-1430. Stacked on #31383.
We had two rules for which media a model can take.
AcceptsFilePartMediaTypefiltered user attachments, whilstToolResultMediaOmissionfiltered tool results with its own lists, and only let images through on Anthropic. Now that fantasy converts tool result media like user files, the two should agree, so we'll keep one rule.chatprovider.Modelnow carries the configured provider type, andAcceptsFilePartMediaTypereads a table keyed on the route, i.e. the configured provider plus the transport.replaceUnsupportedMediaapplies it to user file parts and tool result media alike, including the pending tail of the turn prompt. As a result, tool result PDFs now reach Anthropic, and the 5 MiB inline image cap only applies to images.Azure OpenAI Chat Completions rejects file content parts with a 400 (
This model does not support file content types, see Azure/azure-sdk-for-net#51873), and Gemini's OpenAI-compatible endpoint ignores them. chatd reaches both through the openai-compat client, so the rule rejects PDFs there, and the model gets a text note instead of a failed request or a silently dropped file.Unsupported tool result media now gets the same note fantasy writes ("The tool returned
<type>content, which cannot be displayed."), so the model sees one wording whichever layer drops it. I've also added a Tool results section to the MCP servers docs, and fixed the File attachments docs, which said every rejected attachment becomes text. Only text files do; a rejected PDF becomes a note that names it.I've also tested this end to end on Bedrock. I ran chatd from this branch with an Anthropic provider pointed at our AI gateway's Bedrock route (InvokeModel, us-east-2), plus an MCP tool that returns a PDF with a code word inside. Claude Haiku 5.5 and Claude Opus 5.5 (adaptive thinking) both read the code word from the tool result, and read it again on the next turn, when the PDF is replayed from history. The request carries the PDF as a
documentblock insidetool_result, the same body chatd sends to Anthropic.