Repository navigation
feat: warn when a Bedrock InvokeModel provider ignores the chat model identifier #30752
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
56db41e
e31b28e
3afd978
ea6c843
d6b2c6e
ba52822
171380c
2f2f390
d953de4
8b5299d
6bb3302
5c84baf
5c7e6e7
afadeea
833333b
3cbbe13
594f993
b26343a
a3966ac
03e8f40
8797de6
a20bde0
4c855c9
b3ab7f9
8a304ae
8c6e8a9
0a8d536
af60986
e789391
075e220
01c2be9
06b57d8
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -32,6 +32,8 @@ import ( | |
|
|
||
| "cdr.dev/slog/v3" | ||
| "github.com/coder/coder/v2/agent/agentssh" | ||
| aibridgeconfig "github.com/coder/coder/v2/aibridge/config" | ||
| agplaibridge "github.com/coder/coder/v2/coderd/aibridge" | ||
| "github.com/coder/coder/v2/coderd/audit" | ||
| "github.com/coder/coder/v2/coderd/cryptokeys" | ||
| "github.com/coder/coder/v2/coderd/database" | ||
|
|
@@ -8402,13 +8404,15 @@ func (api *API) chatModelProviderDescriptors( | |
| } | ||
| } | ||
| hasUserKey := userKeyStatus[provider.ID] | ||
| settings, settingsErr := db2sdk.AIProviderSettings(provider.Settings) | ||
| // Some provider configurations support ambient credentials rather than a | ||
| // stored key. Actual credential availability is checked by the gateway. | ||
| // Bedrock always supports them, so its settings are only needed for the | ||
| // InvokeModel hint, which corrupt settings leave false. | ||
| supportsAmbientCredentials := provider.Type == database.AIProviderTypeBedrock | ||
| if !supportsAmbientCredentials { | ||
| settings, err := db2sdk.AIProviderSettings(provider.Settings) | ||
| if err != nil { | ||
| return nil, xerrors.Errorf("decode AI provider settings: %w", err) | ||
| if settingsErr != nil { | ||
| return nil, xerrors.Errorf("decode AI provider settings: %w", settingsErr) | ||
| } | ||
| supportsAmbientCredentials = aiProviderSupportsAmbientCredentials(settings) | ||
| } | ||
|
|
@@ -8422,11 +8426,21 @@ func (api *API) chatModelProviderDescriptors( | |
| HasUserAPIKey: hasUserKey, | ||
| HasEffectiveAPIKey: hasKey || hasUserKey || supportsAmbientCredentials, | ||
| AllowUserAPIKey: api.DeploymentValues.AI.BridgeConfig.AllowBYOK.Value(), | ||
| BedrockInvokeModel: provider.Type == database.AIProviderTypeBedrock && | ||
| bedrockUsesInvokeModel(settings.Bedrock), | ||
| }) | ||
| } | ||
| return out, nil | ||
| } | ||
|
|
||
| // bedrockUsesInvokeModel reports whether the gateway serves the provider over | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P3 [CRF-13] The The
|
||
| // the Bedrock InvokeModel protocol, which sends the provider's configured | ||
| // models upstream in place of the requested model. | ||
| func bedrockUsesInvokeModel(settings *codersdk.AIProviderBedrockSettings) bool { | ||
| cfg := agplaibridge.BedrockConfig("", settings) | ||
| return cfg != nil && cfg.ResolvedProtocol() == aibridgeconfig.BedrockProtocolInvokeModel | ||
| } | ||
|
|
||
| // aiProviderSupportsAmbientCredentials reports whether the provider's settings | ||
| // support ambient credentials. It does not verify that credentials are available. | ||
| func aiProviderSupportsAmbientCredentials(settings codersdk.AIProviderSettings) bool { | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2821,6 +2821,10 @@ type ChatModelProviderDescriptor struct { | |
| AllowUserAPIKey bool `json:"allow_user_api_key"` | ||
| Available bool `json:"available"` | ||
| UnavailableReason ChatModelProviderUnavailableReason `json:"unavailable_reason,omitempty"` | ||
| // BedrockInvokeModel reports whether the provider uses the AWS Bedrock | ||
| // InvokeModel protocol. That protocol ignores the model a client requests | ||
| // and sends the provider's configured models upstream instead. | ||
| BedrockInvokeModel bool `json:"bedrock_invoke_model,omitempty"` | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P3 [CRF-10] The same name is used for the JSON field,
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit [CRF-11]
|
||
| } | ||
|
|
||
| // OrganizationChatModelsResponse is the org chat model config collection: | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -124,6 +124,8 @@ Configure: | |
| - An **endpoint** in the format the protocol requires: `https://bedrock-runtime.<region>.amazonaws.com` for InvokeModel or `https://bedrock-mantle.<region>.api.aws/anthropic` for Mantle. | ||
| The AWS region is read from the endpoint host. | ||
| - The **model** and **small fast model** identifiers, for InvokeModel only. | ||
| Use Bedrock identifiers, not Anthropic API model names. | ||
| Refer to [How InvokeModel selects the model](#how-invokemodel-selects-the-model). | ||
| Mantle providers do not set them; the client chooses the model on each request and AI Gateway forwards it upstream. | ||
|
|
||
| #### InvokeModel | ||
|
|
@@ -132,11 +134,49 @@ The legacy Bedrock runtime API. | |
| AI Gateway translates each request into Bedrock's InvokeModel format and sends it to `https://bedrock-runtime.<region>.amazonaws.com`. | ||
| Still supported, but Mantle is recommended for new deployments. | ||
|
|
||
| ##### How InvokeModel selects the model | ||
|
|
||
| Clients and Bedrock name models differently. | ||
| Anthropic clients, including Claude Code and Coder Agents, request models by their Anthropic API names, such as `claude-sonnet-4-5`. | ||
| Bedrock does not accept those names. | ||
| It identifies a model by one of the following: | ||
|
|
||
| - A Bedrock model ID, such as `anthropic.claude-sonnet-4-5-20250929-v1:0`. | ||
| - A cross-Region inference profile ID, which adds a geographic prefix to the model ID, such as `global.anthropic.claude-sonnet-4-5-20250929-v1:0`. | ||
| - An [application inference profile](#application-inference-profiles) ARN. | ||
|
|
||
| AI Gateway does not translate Anthropic names into Bedrock identifiers. | ||
| Instead, an InvokeModel provider ignores the model that the client requests and sends one of the two Bedrock identifiers configured on the provider: | ||
|
|
||
| | Requested model identifier | Bedrock identifier sent to AWS | | ||
| |----------------------------------------------|-------------------------------------| | ||
| | Contains `haiku`, such as `claude-haiku-4-5` | The provider's **small fast model** | | ||
| | Anything else, such as `claude-opus-4-5` | The provider's **model** | | ||
|
|
||
| An InvokeModel provider therefore serves at most two Bedrock models, and the requested model only chooses between them. | ||
| AI Gateway records each request, its token usage, and its cost against the Bedrock model that ran, not the requested model. | ||
| For an application inference profile, that is the model behind the profile. | ||
| To offer another Bedrock model, add another InvokeModel provider configured with that model, or use a [Mantle](#mantle) provider. | ||
|
|
||
| The same applies to Coder Agents models. | ||
| The model identifier of an Agents model that uses an InvokeModel provider is not sent to AWS; it only selects the configured **model** or **small fast model**. | ||
| For example, an Agents model with the identifier `claude-opus-4-5` on a provider whose **model** is `global.anthropic.claude-sonnet-4-5-20250929-v1:0` runs Claude Sonnet 4.5. | ||
| The dashboard warns about this when you add or edit such a model, and the Agents chat input warns when such a model is selected. | ||
|
|
||
| The model identifier you give an Agents model on an InvokeModel provider does not change which Bedrock model runs, beyond the `haiku` rule above. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit [CRF-15] Line 166 repeats line 162: both say the Agents model identifier only chooses between the configured model and small fast model. (Leorio) Delete line 166 and start line 167 with "To avoid confusion, match the Agents model identifier to the configured model that it selects, ...".
|
||
| To avoid confusion, match it to the configured model that it selects, such as `claude-sonnet-4-5` for a configured **model** of `global.anthropic.claude-sonnet-4-5-20250929-v1:0`. | ||
| Keep the `claude-` prefix: Coder Agents uses it to send requests in the Anthropic Messages format. | ||
|
|
||
| #### Mantle | ||
|
|
||
| The newer Anthropic-compatible Bedrock endpoint, recommended by AWS for new deployments. | ||
| AI Gateway serves Anthropic models through the native Messages API, forwarding the request body **unchanged** and only applying AWS SigV4 signing with the provider's base identity. | ||
|
|
||
| Because the model is forwarded as requested, clients and Coder Agents models on a Mantle provider must use Mantle model IDs, such as `anthropic.claude-sonnet-5`, rather than Anthropic API names. | ||
| Use the bare model ID, without a cross-Region inference profile prefix such as `us.` or `global.`. | ||
| To list the IDs that Mantle serves in a Region, call its `/v1/models` endpoint. | ||
| Refer to [Get list of models](https://docs.aws.amazon.com/bedrock/latest/userguide/models-get-info.html) in the AWS documentation. | ||
|
|
||
| To route Claude Code through a Mantle provider, run it in mantle mode with client-side signing disabled so the gateway signs centrally: | ||
|
|
||
| ```sh | ||
|
|
||
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
P4 [CRF-12] The added comment calls
BedrockInvokeModel"the InvokeModel hint", which is a second name for the field. (Gon)Replace the two added lines with:
// Bedrock always supports them, so undecodable Bedrock settings only leave BedrockInvokeModel false.