Skip to content

feat!: choose the computer use model per organization - #31300

Merged
ibetitsmike merged 48 commits into
mainfrom
mike/computer-use-ai-provider
Oct 29, 2026
Merged

ibetitsmike merged 48 commits into
mainfrom
mike/computer-use-ai-provider

Conversation

@ibetitsmike

@ibetitsmike ibetitsmike commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Note

Xum opened this PR for Mike (Coder agent). Mike reviews it before merge.

Fixes CODAGT-1472.

Computer use now runs on a chat model that each organization picks. Before, it ran a fixed model on the first enabled AI provider of a deployment-wide provider type, sorted by name. On dogfood that provider's key is rejected, so every computer use subagent failed.

What changed

  • Setting. A new computer_use organization model override, set in the Computer use row of Organization settings on the Coder Agents admin page. Without an override, computer use runs on the organization's default model. If the chosen model is disabled, deleted, or unusable for the chat owner, computer use falls back to the default model, as other overrides do. It also falls back when the chosen model moves to an AI provider that is not Anthropic or OpenAI, and the row then says that the saved model is ignored.
  • API. PUT /api/v2/organizations/{organization}/chats/model-overrides/computer_use returns 404 while the chat-virtual-desktop experiment is off, and 400 unless the model's AI provider type is Anthropic or OpenAI.
  • chatd. The computer_use subagent runs on that model, and the computer tool flavor follows the model's provider type. OpenAI computer use always uses the Responses API, so a model that sets use_responses_api: false gets a configuration error. When no usable model exists, computer_use is unavailable, and the reason says whether a saved computer use model was skipped and why the default model cannot be used. The default model is read under the chat owner's model ACLs, as chat turns do.
  • UI. The row lists only enabled Anthropic and OpenAI models. A warning icon with the tooltip "Make sure that the selected model supports computer use." appears when the model in effect (the selected one, or the default) is not on a curated list of canonical model IDs known to accept the computer tool (knownComputerUseModels in knownModels/catalog.ts, exact match only). It also appears when the model's AI provider type is OpenAI and its Use Responses API override is Off, because OpenAI computer use needs the Responses API. The warning does not block saving. I built the list from a live probe on 2026-10-26 that sent each Anthropic and OpenAI model one request with the computer tool.
  • Docs. virtual-desktop.md, models.md, organizations.md, and architecture.md describe the new setting, and chatd's ARCHITECTURE.md describes the computer use model routing.

The Computer use row with an uncurated model selected shows the warning icon and its tooltip

The Computer use model picker lists only Anthropic and OpenAI models

Breaking change

This removes the deployment-wide computer use provider setting (agents_computer_use_provider), its experimental GET/PUT /api/experimental/chats/config/computer-use-provider endpoints (shipped since v2.35), the Virtual desktop settings card, and its audit field. Migration 000635 deletes the stored value. After the upgrade, computer use runs on each organization's default model until an admin picks a computer use model. Computer use subagents created before the upgrade keep the model they inherited from their parent chat, so a follow-up message to one of them can fail; a new computer use subagent uses the organization's computer use model. Telemetry now reports the per-organization overrides instead of the provider, and null when it cannot read them.

Testing

  • make pre-commit passes. go test passes for ./coderd/x/chatd/..., ./coderd/database/migrations/, and the targeted coderd, telemetry, dbauthz, enterprise audit, and codersdk tests. Vitest and the touched Storybook stories pass.
  • Each new Go test fails when its target behavior is removed from production code.
  • Remote dogfood UAT ran on an isolated Coder instance built from the PR head, with real Anthropic and OpenAI models:
    • 110234e: the row with the experiment on and off, the picker, the warning icon cases, save and clear, the 400 for other provider types, which model and computer tool each computer use subagent used (Anthropic, OpenAI, default, and disabled override), a real desktop session on an Anthropic and an OpenAI model, the unavailable reasons, the upgrade through migration 000634, and the removed endpoints.
    • 09ce818: an override whose model moved to an unsupported AI provider.
    • 7925be9: the alert for a moved and a disabled override, and a computer use subagent on an Anthropic model.
    • 2a01eaf: the row, and computer use subagents on the override and on the default model.
    • 47f5bd1: the warning icon for an OpenAI model with Use Responses API set to Off, On, and Default. With Off, a computer use subagent on that model fails with a configuration error before any provider request. With Default, it runs with the computer tool through the Responses API.
    • 81d33b4: the unavailable reasons for a disabled override, a cleared override, and missing credentials, and a computer use subagent on an Anthropic override. This round found that the docs named the wrong section of the model form for Use Responses API. aeb3d02 fixes that step, and a re-read of the docs against the recorded UI passed.

@linear-code

linear-code Bot commented Oct 26, 2026

Copy link
Copy Markdown

CODAGT-1472

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Docs preview

Check off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here.

@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-29T11:52:40.991954Z c763bb8 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a02e48e8da

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/computer_use.go Resolved
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 7c033330f7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike ibetitsmike changed the title feat: let admins choose the AI provider used for computer use feat!: choose the computer use model per organization Oct 27, 2026
@github-actions github-actions Bot added the release/breaking This label is applied to PRs to detect breaking changes as part of the release process label Oct 27, 2026
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

1 similar comment
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 53907308a4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/computer_use.go Resolved
Comment thread coderd/chat_routes.go Resolved
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

1 similar comment
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7925be9d53

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/telemetry/telemetry.go Resolved
Comment thread coderd/x/chatd/turn_environment.go Resolved
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2a01eafecf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/telemetry/telemetry.go Resolved
Comment thread coderd/x/chatd/turn_environment.go Resolved
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 47f5bd11ba

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/computer_use.go Resolved
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

1 similar comment
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: 31c7336ce7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit: aeb3d0298e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike
ibetitsmike marked this pull request as ready for review October 27, 2026 15:25
@ibetitsmike
ibetitsmike requested review from a team and nickvigilante as code owners October 27, 2026 15:25
Store agents_computer_use_ai_provider_id next to the computer-use provider
type. The PUT endpoint validates that the selected AI provider exists, is
enabled, and matches the type, and writes both settings in one audited
transaction. chatd routes computer-use turns and the computer_use subagent
key check to the selected provider, and reports computer_use unavailable
when the selection is unusable. Without a selection, behavior is unchanged.

@DanielleMaywood DanielleMaywood left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was performed by ai-review, an automated review bot.

8 inline findings.

Coverage
Coverage: 47 of 63 files reviewed, 11 unsupported, 4 generated, 1 deleted; 22 rules.
Candidates: 2572 found, 2517 outside changed lines, 3 already discussed, 52 evaluated.
Findings: 8 (20 sites, 3 pre-existing sites), 0 over budget, 0 failed.
Model: deepseek/deepseek-v4.1-flash; 34 questions, 34 answered, 0 abstained, 34 cached; 0 uncalibrated findings held back; 0 input and 0 output tokens.
Experiments: 1 run, 0 passed, 1 vetoed, 0 unverified, 0 skipped for time.
  vetoed: TYPE-1 site/src/pages/AISettingsPage/CoderAgentsPage/OrganizationAgentSettingsView.tsx:129: 1 new type error, first src/pages/AISettingsPage/CoderAgentsPage/OrganizationAgentSettingsView.tsx: TS2345 Argument of type 'string 

Comment thread coderd/x/chatd/subagent_catalog.go Resolved
Comment thread coderd/x/chatd/computer_use.go Resolved
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

Xum acted for Mike.

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 👍

Reviewed commit: 39e6850ccf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@DanielleMaywood DanielleMaywood left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was performed by ai-review, an automated review bot.

1 inline finding.

Coverage
Coverage: 47 of 63 files reviewed, 11 unsupported, 4 generated, 1 deleted; 22 rules.
Candidates: 2557 found, 2517 outside changed lines, 4 already discussed, 36 evaluated.
Findings: 3 (4 sites, 0 pre-existing sites), 0 over budget, 0 failed.
Model: deepseek/deepseek-v4.1-flash; 32 questions, 32 answered, 0 abstained, 32 cached; 0 uncalibrated findings held back; 0 input and 0 output tokens.
Experiments: 1 run, 0 passed, 1 vetoed, 0 unverified, 0 skipped for time.
  vetoed: TYPE-1 site/src/pages/AISettingsPage/CoderAgentsPage/OrganizationAgentSettingsView.tsx:129: 1 new type error, first src/pages/AISettingsPage/CoderAgentsPage/OrganizationAgentSettingsView.tsx: TS2345 Argument of type 'string 

Renumber 000634_chat_model_override_computer_use to 000635 because main
added 000634_aibridge_interception_organization. The migration test now
steps to 634 before applying the renumbered migration.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

Xum posted this comment on behalf of @ibetitsmike.

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: b4e4065dbb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

Xum posted this comment on behalf of @ibetitsmike.

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: 127c2acb3b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/telemetry/telemetry.go Resolved
Comment thread coderd/x/chatd/ARCHITECTURE.md Resolved
Comment thread coderd/x/chatd/computer_use.go Resolved
Comment thread codersdk/chats.go Resolved
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

Xum posted this comment on behalf of @ibetitsmike.

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: 9006d98048

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

Xum posted this comment on behalf of @ibetitsmike.

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: ee05ca41b7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

Xum posted this comment on behalf of @ibetitsmike.

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: c763bb8c6d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike
ibetitsmike merged commit 3697bb3 into main Oct 29, 2026
29 checks passed
@ibetitsmike
ibetitsmike deleted the mike/computer-use-ai-provider branch October 29, 2026 12:00
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 29, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

release/breaking This label is applied to PRs to detect breaking changes as part of the release process

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants