Skip to content

fix: use one route-aware media rule for files and tool results - #31384

Merged
ethanndickson merged 6 commits into
codagt-1430-tool-result-filenamefrom
codagt-1430-media-rule
Nov 10, 2026
Merged

ethanndickson merged 6 commits into
codagt-1430-tool-result-filenamefrom
codagt-1430-media-rule

Conversation

@ethanndickson

@ethanndickson ethanndickson commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Part of CODAGT-1430. Stacked on #31383.

We had two rules for which media a model can take. AcceptsFilePartMediaType filtered user attachments, whilst ToolResultMediaOmission filtered tool results with its own lists, and only let images through on Anthropic. Now that fantasy converts tool result media like user files, the two should agree, so we'll keep one rule.

chatprovider.Model now carries the configured provider type, and AcceptsFilePartMediaType reads a table keyed on the route, i.e. the configured provider plus the transport. replaceUnsupportedMedia applies it to user file parts and tool result media alike, including the pending tail of the turn prompt. As a result, tool result PDFs now reach Anthropic, and the 5 MiB inline image cap only applies to images.

Azure OpenAI Chat Completions rejects file content parts with a 400 (This model does not support file content types, see Azure/azure-sdk-for-net#51873), and Gemini's OpenAI-compatible endpoint ignores them. chatd reaches both through the openai-compat client, so the rule rejects PDFs there, and the model gets a text note instead of a failed request or a silently dropped file.

Unsupported tool result media now gets the same note fantasy writes ("The tool returned <type> content, which cannot be displayed."), so the model sees one wording whichever layer drops it. I've also added a Tool results section to the MCP servers docs, and fixed the File attachments docs, which said every rejected attachment becomes text. Only text files do; a rejected PDF becomes a note that names it.

I've also tested this end to end on Bedrock. I ran chatd from this branch with an Anthropic provider pointed at our AI gateway's Bedrock route (InvokeModel, us-east-2), plus an MCP tool that returns a PDF with a code word inside. Claude Haiku 5.5 and Claude Opus 5.5 (adaptive thinking) both read the code word from the tool result, and read it again on the next turn, when the PDF is replayed from history. The request carries the PDF as a document block inside tool_result, the same body chatd sends to Anthropic.

@linear-code

linear-code Bot commented Oct 28, 2026

Copy link
Copy Markdown

CODAGT-1430

@github-actions

Copy link
Copy Markdown
Contributor

Docs preview

Check off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here.

@ethanndickson

Copy link
Copy Markdown
Member Author

@codex review

@ethanndickson

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-09T06:08:54.180265Z 3d5c67b Manual request
🔒 Security Review ✅ Completed 2026-11-03T06:26:56.934894Z 8356501 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 8356501f8b

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8356501f8b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ethanndickson
ethanndickson force-pushed the codagt-1430-media-rule branch from 8356501 to 0cb6bf3 Compare November 4, 2026 07:02
@ethanndickson

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: 0cb6bf35b7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ethanndickson
ethanndickson force-pushed the codagt-1430-media-rule branch from 0cb6bf3 to 8d99e82 Compare November 4, 2026 07:16
@ethanndickson
ethanndickson marked this pull request as ready for review November 4, 2026 07:47

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8d99e82fa9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/chatprovider/model.go Resolved
@ethanndickson

Copy link
Copy Markdown
Member Author

@codex review

re: Preserve the Copilot provider in media routing

The code path is real, but this is not worth fixing here. RouteProvider() does return "" for copilot, so Copilot media becomes a text note. That is fine for now:

  1. Coder Agents does not support Copilot (fix: stop Agents dead-ending on unsupported providers #26841). The availability check skips it (coderd/exp_chats.go:1063-1066 and 1116-1118), so creating a chat or sending a message with a Copilot model config is rejected.
  2. Copilot needs a per-request token that only an official Copilot client can mint. chatd sends a placeholder bearer token, and the aibridge Copilot provider uses the Authorization header as the credential (aibridge/provider/copilot.go:209-217). A Copilot turn fails at authentication before media matters.
  3. A media row alone would not make images work. Copilot clients such as Zed and LiteLLM read vision support per model from the Copilot models API and send a Copilot-Vision-Request header. chatd does neither.
  4. Unknown routes accept nothing by design (see the AcceptsFilePartMediaType doc comment). A fix in NormalizeProvider would also let Copilot through the availability gate.
    If Agents ever supports Copilot, that work should add a dedicated client with verified media types. A cleaner follow-up is to reject Agents-unsupported provider types when chatd resolves the route, so chatd never builds a Copilot client.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: 8d99e82fa9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ethanndickson

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: 3d5c67b694

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…l results

chatprovider.Model now carries the configured provider type, set in
newModel. AcceptsFilePartMediaType reads a table keyed on the route
(configured provider and transport), and InlineImageCapBytes is a Model
method that applies to images only. ToolResultMediaOmission is removed.

replaceUnsupportedMedia replaces both replaceUnsupportedToolMedia and
replaceUnsupportedFileParts. It handles user file parts and tool result
media with the same rule, and the turn prompt now filters the pending
tail too. A PDF at the image cap no longer gets the image cap note.

Tool result media that the route cannot take now gets the note that
fantasy's providers write ("The tool returned <type> content, which
cannot be displayed."). Each note comes before the accompanying text,
because it takes the place of the media.
…utes

Azure OpenAI Chat Completions rejects file content parts with HTTP 400
("This model does not support file content types"), and Gemini's
OpenAI-compatible endpoint does not process them. chatd reaches both
providers through the OpenAI-compatible client, so the route rule now
rejects PDFs on them. User attachments and tool results get the usual
text note instead of a failed request or a silently dropped file.
…tachments

The tool result media rules apply to org servers too, not only inline
servers, so they move to a Tool results section. The section names the
one image or PDF limit per result and the size notes.

The File attachments text said that a rejected part is downgraded to
text. Only text files are. Other rejected files, such as PDFs on Azure
OpenAI and Google, reach the model as a note that names the file.
The OpenAI transport bullet said that the file part rule gates only attachments and that a mismatch drops text attachments. The rule now also gates tool-result media, and neither OpenAI transport takes text file parts, so audio is the only type that the transports treat differently. The compaction bullet now also names tool-result media, which the compaction prompt already replaces.
@ethanndickson
ethanndickson force-pushed the codagt-1430-media-rule branch from 3d5c67b to 409b224 Compare November 9, 2026 06:48
@ethanndickson
ethanndickson merged commit 7a12022 into main Nov 10, 2026
42 of 64 checks passed
@ethanndickson
ethanndickson deleted the codagt-1430-media-rule branch November 10, 2026 13:06
ethanndickson added a commit that referenced this pull request Nov 10, 2026
Part of [CODAGT-1430](https://linear.app/codercom/issue/CODAGT-1430).
Stacked on #31384.

Workspace MCP tool results go through the agent's `convertResult`, which
turns an embedded resource into a `[embedded resource: ...]` string and
drops structured content, so a PDF from a workspace MCP server never
reaches the model. Remote MCP tools already go through `mcpclient`'s
converter, which keeps both.

When a call-tool request sets `include_result`, the agent now returns
the MCP `CallToolResult` as-is in `Result` and leaves `Content` empty.
chattool always sets it and converts the result with
`mcpclient.ConvertCallResult`, so workspace and remote MCP tools share
one converter. Older agents ignore the flag and send the legacy
`Content`, which we adapt to MCP content and pass to the same converter.
The workspace MCP docs get a short Tool results section too.
ethanndickson added a commit that referenced this pull request Nov 10, 2026
Part of CODAGT-1430
Relates to CODAGT-1541

Stacked on #31743.

This bumps fantasy to pick up coder/fantasy#84 and coder/fantasy#85.

A Chat Completions tool message can only hold text, so fantasy sends
tool result media (images, audio, and now PDFs from #31384) in a `role:
"user"` message right after it. That gives whatever the tool returned
the authority of the user's own words, which Codex flagged on #31383 as
a prompt injection risk for MCP tools that return untrusted files.

coder/fantasy#84 starts that user message with a text part naming the
tool call and saying the content is tool output, not instructions from
the user. It only changes the Chat Completions paths (`openai` without
the Responses API, `openaicompat`, OpenRouter and Vercel); Anthropic and
Responses keep the media inside the tool result. AI Bridge records the
last user message's text as the prompt, so on the step after a tool
returns media, it'll now record the label. I think that's fine for now.

coder/fantasy#85 fixes the output token limit field. The OpenAI provider
sent `max_tokens` unless the model ID looked like a reasoning model, and
reasoning models reject it:

```
Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.
```

That breaks any reasoning model whose ID fantasy doesn't recognise, e.g.
an Azure deployment name, which doesn't say what model it serves (Azure
providers start using this client in #31952). The OpenAI provider now
sends `max_completion_tokens` on every Chat Completions request, which
is what OpenAI deprecated `max_tokens` in favour of. `openaicompat`,
OpenRouter and Vercel keep `max_tokens`, so self-hosted servers behind
the OpenAI Compatible provider are unaffected. I've added a test that
checks the field on each route, and the models docs no longer say Max
Completion Tokens is only for reasoning models.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants