UN-4020 [FIX] Route AWS Bedrock Mantle models (GPT-5.6 Terra) via bedrock_mantle - #2248
Conversation
…_mantle AWS serves the OpenAI GPT-5.x, Google Gemma 4 and xAI Grok families from Bedrock Mantle (bedrock-mantle.<region>.api.aws), an OpenAI-compatible endpoint separate from the classic bedrock-runtime Converse/Invoke surface. The adapter prefixed every model with `bedrock/`, so those models were sent to the wrong endpoint with the wrong body schema and never worked. Route them as `bedrock_mantle/<model>` instead. Membership is an exact lookup in LiteLLM's registry rather than a family prefix, because the two id namespaces are disjoint but look alike: `openai.gpt-oss-120b-1:0` is Converse while `openai.gpt-oss-120b` is Mantle, and the `us.`/`global.` cross-Region profile ids belong to bedrock-runtime. An explicit prefix still wins. Bump litellm 1.90.3 -> 1.96.2. 1.90.3 already knew the provider but ignored explicit aws_access_key_id/aws_secret_access_key on that route, so per-adapter Access Keys auth silently failed; explicit keys are honoured from 1.95.1. Also: - Strip Bedrock Guardrails and Application Inference Profile ARNs for Mantle models with a warning. Both are Converse-only features that LiteLLM discards silently, which would leave an operator believing a guardrail is enforced when it is not. - Map "Enable Extended Thinking" to `reasoning_effort` for non-Anthropic families (Mantle GPT-5.x, gpt-oss, Nova 2). The switch emitted Anthropic's `thinking` block for every model, which LiteLLM drops for everything except Claude -- so the control had never done anything outside Claude. - Keep the cohere embed timeout patch, re-pointed at 1.96.2. Upstream fixed only the async path; the synchronous embedding() that Bedrock Cohere embeddings actually use still builds an untimed HTTPHandler. - Add boto3 to the sdk1 test group: LiteLLM signs Bedrock requests with SigV4 via botocore, which the test venv needs to exercise signing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
| Filename | Overview |
|---|---|
| unstract/sdk1/src/unstract/sdk1/adapters/base1.py | Adds exact-registry Mantle routing, endpoint-specific parameter filtering, and family-aware Bedrock reasoning configuration; no eligible follow-up finding was established. |
| unstract/sdk1/src/unstract/sdk1/adapters/llm1/static/bedrock.json | Documents Mantle routing and unsupported settings while updating the extended-thinking configuration guidance. |
| unstract/sdk1/src/unstract/sdk1/patches/litellm_cohere_timeout.py | Retargets the synchronous Cohere embedding timeout patch to the upgraded LiteLLM implementation. |
| unstract/sdk1/pyproject.toml | Upgrades LiteLLM to 1.96.2 and adds boto3 to the test dependency group for SigV4 request tests. |
| unstract/sdk1/tests/test_bedrock_mantle_routing.py | Exercises Mantle URL selection, request body mapping, and SigV4 or bearer authentication at the mocked transport boundary. |
| unstract/sdk1/tests/test_bedrock_adapter.py | Covers model routing, unsupported Mantle parameter removal, reasoning shapes, and validation fixed-point behavior. |
| unstract/sdk1/uv.lock | Resolves the SDK dependency graph for LiteLLM 1.96.2 and the updated test dependencies. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Bedrock adapter metadata] --> B{Explicit provider prefix?}
B -->|bedrock/| C[Classic Bedrock Converse or Invoke]
B -->|bedrock_mantle/| D[Bedrock Mantle OpenAI-compatible endpoint]
B -->|Bare model ID| E{Exact Mantle registry match?}
E -->|Yes| D
E -->|No| C
D --> F[Remove guardrailConfig and model_id]
C --> G[Retain Converse-specific parameters]
F --> H[Resolve AWS authentication]
G --> H
H --> I[LiteLLM completion]
Reviews (4): Last reviewed commit: "UN-4020 [FIX] Drop the no-op yield from ..." | Re-trigger Greptile
johnyrahul
left a comment
There was a problem hiding this comment.
Standardized review — 17-lens rubric
Verdict: REQUEST CHANGES — Critical: 0 · High: 2 · Medium: 12 · Low: 8 · Lenses run: 17/17
Solid PR — the exact-registry lookup over prefix matching is the right call given openai.gpt-oss-120b vs openai.gpt-oss-120b-1:0, and the bedrock/ escape hatch keeps stored configs working. These are review comments, not requested changes I have made — nothing has been pushed. Please take or leave each on its merits; several may be deliberate choices I can't see from the diff, and I've flagged where I think the call is genuinely yours.
Claims I verified rather than assumed: 563 passed on 1.96.2 (exact match) · reverting the routing breaks 15 tests, not 14 (the extra is test_mantle_cost_is_resolvable) — they genuinely pin the behaviour · all 9 lockfiles consistent, nothing beyond litellm 1.90.3→1.96.2, aiohttp 3.14.1→3.14.3, +pydantic-settings 2.15.0 · boto3~=1.34.131 matches the existing aws extra exactly · the vendored cohere patch genuinely applies under 1.96.2 (_SKIP_PATCH=False, handler.embedding is _patched_embedding) and all six private litellm imports still resolve · validate() is a true fixed point.
Unanchored findings
Both sit on lines this diff does not touch, so GitHub can't thread them.
[Medium] [Lens 1 · 7] — bedrock.json:184-197 still makes budget_tokens required for the families this PR says it doesn't apply to
The allOf block makes budget_tokens (minimum 1024) required whenever enable_thinking is true, for every Bedrock model, with no family condition. This PR extends that switch to GPT-5.x / GPT-OSS / Nova 2 and its own description at :69 now says "the budget below does not apply" for exactly those — while :190-191 still describes the field as "Claude's internal reasoning". So two descriptions in the same file disagree, and an operator enabling reasoning on openai.gpt-5.6-terra is blocked by form validation until they type a number the adapter then discards.
The honest limit here is that JSON-Schema can't branch on a free-text model id. Minimally: drop budget_tokens from required (keep default) and retitle it Claude-only. Flagging rather than fixing because changing the form contract is your call.
[Medium] [Lens 17] — The repo's own adapter-ops skill restates the Bedrock rules this PR changed, in six places
These matter more than ordinary stale docs: adapter-ops is the documented path for "adding/removing models to existing adapters", so an engineer or agent following it will reproduce the defect this PR fixes.
| Site | Now says | Problem |
|---|---|---|
.claude/skills/adapter-ops/SKILL.md:47 |
| Bedrock | bedrock/ | bedrock/anthropic.claude-v2 | |
Prefix rule is no longer unconditional |
references/provider_capabilities.md:111 |
| AWS Bedrock | bedrock/ | bedrock/ | |
Same |
references/provider_capabilities.md:12 |
| AWS Bedrock | ❌ | ✅ Claude 3.7+ | … |
Reasoning column ❌; the file's own Feature Definitions (L20-21) define Reasoning as reasoning_effort, which Bedrock now has. Compare the VertexAI row |
references/provider_capabilities.md:64-66 |
model, max_tokens, enable_thinking, budget_tokens |
Missing reasoning_effort; every sibling that has it lists it |
references/adapter_patterns.md:269-350 |
"Thinking Configuration Pattern (Anthropic, VertexAI, Bedrock)" | The code block is, line for line, the pre-change AWSBedrockLLMParameters.validate this PR deleted — a generative site handing the next engineer the old shape |
SKILL.md:205-207 |
"Add conditional thinking config in validate()" |
Same rule, same staleness |
Not a finding: provider_capabilities.md:32 is the embedding prefix table, and Mantle is LLM-only, so bedrock/ there is still correct.
Two claims I want to retract before you read the comments
I'd rather flag these than let them stand:
- There is no CI flake. Some intermediate analysis suggested the routing suite failed 15 tests spontaneously "under network contention". It does not: I measured 5/5 clean runs, and it still passes with egress blocked, because the bundled map carries all 13 Mantle entries — so no registry state can produce
bedrock/openai.gpt-5.6-terra. That failure only appears when the routing code is reverted. The registry risk in my first comment is latent, not active. - The
os.environwrite attest_bedrock_mantle_routing.py:254does not leak.monkeypatch.setenvalready recorded the pre-test value, so teardown restores it. It's merely dead.
Lens checklist
| # | Lens | Result |
|---|---|---|
| 1 | Spec & intent | See unanchored #1 |
| 2 | Architectural fit | See comments (registry as routing oracle) |
| 3 | Correctness & edge cases | See comments |
| 4 | Security | See comment on guardrail strip |
| 5 | Data integrity & migrations | N/A — no migrations, schema, or persisted-state change |
| 6 | Concurrency | Clean — no new shared mutable state |
| 7 | API & contract compatibility | See comments |
| 8 | Reliability & resilience | See comments |
| 9 | Performance & cost | Clean |
| 10 | Observability | See comments |
| 11 | Operational safety | See comment on silent reasoning activation |
| 12 | LLM/agent | See comments; cost resolves for the new prefix, audit.py prefix-strip correct |
| 13 | Testing | See comments |
| 14 | Dependencies & build | Clean — verified live under 1.96.2 |
| 15 | Code quality | See comments |
| 16 | Doc & comment accuracy | See comments |
| 17 | Cross-document consistency | See unanchored #2 |
The two decisions I think are actually yours
- The routing oracle. Pin
LITELLM_LOCAL_MODEL_COST_MAP=Trueso routing follows the pinned wheel, or keep an in-repo id set with the registry as fallback? Both have real costs and it's a deployment-shape decision, not a code cleanup. - Guardrail on a Mantle model — warn or hard-fail?
llm.py:313already turnsValueErrorinto a UI-visibleSdkError. Degrading a moderation control on a stdout warning seems the wrong default to me, but it's a product call.
Also worth confirming: were the non-Bedrock providers (Vertex, Azure, Mistral, Ollama, x2text) exercised against 1.96.2 anywhere? The bump crosses six minors and touches every adapter, justified by a Bedrock fix.
Happy to be wrong on any of these — please push back where I've misread the intent.
…meticity Correctness - `_apply_bedrock_reasoning_config` now clears `reasoning_effort` in the Anthropic branch, mirroring the pop in the other branch. Previously an Anthropic model carrying a stray `reasoning_effort` emitted *both* shapes, contradicting the comment and resting on `litellm.drop_params` to discard the extra key on the Converse route. - Dropped the `reasoning_effort` field declaration: it is stripped from `validation_metadata` unconditionally, so Pydantic never saw it and the "otherwise Pydantic would drop it" rationale was unreachable. Survival across re-validation comes from the explicit re-attach after `model_dump()`, which is now what the comment says. Docs that were wrong in a way that misleads operators - The Mantle family lists omitted GPT-OSS -- 4 of the 13 Mantle models. The guardrail description is the compliance-relevant one: an operator running `openai.gpt-oss-120b` read that their family was unaffected and would have shipped believing moderation was enforced when it is silently dropped. - `enable_thinking` claimed models without reasoning "ignore this setting". They do not: the `temperature = 1` write precedes the family branch, so the toggle changes sampling for every Bedrock model. Said so instead, in the schema and in the docstring. - `budget_tokens` is no longer `required` when thinking is enabled. The switch now reaches families the budget does not apply to, so requiring it blocked those operators on a value the adapter discards. Retitled Claude-only. - Narrowed the "the two model gates cannot drift apart" claim: `llm.py` keeps its own markers and does not call the helper. - Corrected "ONLY CHANGE" in the cohere patch, which understated two inert deviations from upstream and is the baseline the next re-diff relies on. Tests - The registry pin now lives at module scope in `tests/conftest.py`. The autouse fixture set it after litellm was already imported, so it was inert and the routing tests were silently reading the *network* registry (3054 models) rather than the bundled one (2980). The fixture now asserts the pin took effect instead of pretending to set it. - The routing harness asserts `litellm.drop_params` rather than setting it. The whole Mantle path depends on `unstract.sdk1.llm` setting it globally -- setting it locally would keep this suite green through a refactor that dropped it while every GPT-5.x Bedrock completion broke in production. - Added the registry-miss case (a Mantle model the loaded registry lacks falls back to `bedrock/`) and pinned that the Mantle strip warns once, not on every completion. Repo docs - The `adapter-ops` skill is the documented path for adding models, and it restated the unconditional `bedrock/` rule in its prefix tables and shipped the pre-change `validate()` body verbatim as a copyable pattern. Updated so it no longer hands the next engineer the shape this PR removed. Declined, with reasoning in the PR thread: pinning the cost map at deploy time (routing oracle is a deployment decision with a cost-tracking trade-off), hard-failing on a Mantle guardrail (product call), the `enable_thinking=False` latch (house pattern, repo-wide or not at all), and the cohere version-mismatch timeout (pre-existing mechanism, separate change). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Thanks — this was a genuinely useful review. I verified every claim before acting on it and found them all accurate; replies are on each thread. All changes are in a single commit ( The two unanchored findings — both taken
Your closing question: non-Bedrock providers against 1.96.2Fair challenge — a six-minor bump justified by a Bedrock fix. The 565-test suite is the coverage, and it exercises the adapter layer for OpenAI, Azure, VertexAI/Gemini, Mistral, Anthropic, Ollama, the branded OpenAI-compatible adapters, and the embedding adapters, all green on 1.96.2. What that does not cover is live provider traffic — the tests validate parameter shaping and mocked wire behaviour, not real completions against each vendor. I also verified the whole platform boots on the bump: built all images and brought up 24 containers, with the routing confirmed live inside both the backend and a worker. So: adapter-layer regression risk is covered, live cross-provider behaviour is not, and I'd flag that as the thing worth a smoke pass during RC rather than something I can close out here. Declined, with reasoning on each thread
Two corrections to my own PR description while I'm here: it's 15 tests that fail on revert, not 14, and the hermeticity claim I made for the routing tests was false until this commit — your measurement of the inert fixture was the most valuable finding in the review, since it meant those tests were silently reading the network registry. |
There was a problem hiding this comment.
Approving — verified against ee8f2f5d2
Re-reviewed the follow-up rather than taking the "fixed in" claims at face value. 15 of the findings are fixed and independently confirmed; 5 were declined or deferred with reasoning I accept.
Verified behaviourally, not by reading the diff:
- Both-shapes bug —
BOTH PRESENT = Falseon the original repro and on a Mantle→Claude family switch. Idempotence still holds andreasoning_effortstill round-trips. - Test hermeticity — the one that mattered: the pin now takes in a full-suite run (
source=local, is_env_forced=True, 2980 models, previouslyremote/False/3054). Self-verifying assert means it fails loudly if that ever regresses. - Dead field — genuinely removed, not repaired in comment only.
- Lens 17 — all six
adapter-opssites corrected, including dropping Bedrock from the thinking-pattern heading and adding the "check before copying" callout.provider_capabilities.md:32correctly left alone, since the embedding table is right as it stands. budget_tokens—requireddropped and retitled, picked up from the summary body without a thread prompting it.
Suite green at 565 passed (+2 new tests, both pinning things that were genuinely unpinned).
On the registry finding — your counter-argument is better than my recommendation, and I'm withdrawing it. I framed litellm.model_cost as a routing oracle; you're right that it is simultaneously the pricing oracle, and that pinning it would freeze audit.py pricing and silently record $0 for any model shipped after the pin, across every provider. Trading a latent routing miss for a guaranteed ongoing revenue miss is the worse default, and it isn't a decision a Bedrock routing PR should make. Pinning it in tests while leaving production live is the right split.
One option neither of us raised, for whenever that issue gets written up: the two oracles could be decoupled — resolve the routing membership check against the bundled map (cached once in a module-level frozenset) while leaving litellm.model_cost on the live fetch for pricing. That gets deterministic, version-pinned routing without freezing cost data. Worth weighing there rather than here.
You were also right to reject my breadcrumb suggestion — "not in the Mantle registry" isn't a signal on its own, since Claude, Titan and Nova take that same path.
Open, by agreement — approving does not close these
- Guardrail runtime signal — the config-time text is now honest, but a model switch that silently disables an already-configured guardrail still produces no tenant-visible signal. Detail in the thread. Note that thread is marked resolved — that's your call and I've left it that way — so this list is the durable record of the open question, not the thread state.
- Silent reasoning activation — stored adapters with
enable_thinking: trueon non-Claude models gain reasoning, latency and token spend on deploy. Pending the production count and release notes; gate rather than log if the count is non-trivial. - Cohere patch unbounded-timeout degradation on version mismatch — pre-existing, re-armed by the pin move.
us./global.detection warning — agreed shape, deferred to its own change.
Approving on the strength of the code in this PR. Nice work on the follow-up — the replies engaged with the substance rather than just the asks, and pushing back on two of them was the right call.
|
@pk-zipstack Please handle the sonar issue. |
`_isolated_litellm_env` had nothing to do after the yield -- `monkeypatch` undoes its own `delenv` calls at teardown -- so being a generator bought nothing and read as an empty teardown block. Made it a plain function returning None and dropped the now-unused `Iterator` import. Behaviour is unchanged: a non-generator autouse fixture still runs, verified by breaking its precondition (`LITELLM_LOCAL_MODEL_COST_MAP=False`), which still fails every test in the file on the registry-pin assertion. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Unstract test resultsPer-group results
Critical paths
|



What
AWS serves the OpenAI GPT-5.x, Google Gemma 4 and xAI Grok families from Bedrock Mantle (
bedrock-mantle.<region>.api.aws) — an OpenAI-compatible endpoint separate from the classicbedrock-runtimeConverse/Invoke surface. Our adapter prefixed every model withbedrock/, so those models were sent to the wrong endpoint with the wrong body schema and never worked. Reported by an on-prem customer trying to useopenai.gpt-5.6-terra.This routes them as
bedrock_mantle/<model>instead. No UI change — users keep entering the bare model ID (openai.gpt-5.6-terra); the adapter picks the provider itself, exactly as it already does forbedrock/.How the route is decided
Membership is an exact lookup in LiteLLM's own model registry, not a family/prefix match, because the two id namespaces are disjoint but look alike:
openai.gpt-5.6-terrabedrock_mantle/openai.gpt-5.6-terraopenai.gpt-oss-120bbedrock_mantle/openai.gpt-oss-120bopenai.gpt-oss-120b-1:0bedrock/openai.gpt-oss-120b-1:0anthropic.claude-…bedrock/anthropic.claude-…us.openai.gpt-5.6-terrabedrock/us.openai.gpt-5.6-terraAn explicit prefix is honoured verbatim — that keeps already-stored configs working and makes
validate()a fixed point, which matters becauseLLM.complete()re-validates kwargs on every call.litellm 1.90.3 → 1.96.2
Required, but not for the reason you'd guess: 1.90.3 already knew the
bedrock_mantleprovider — it just ignored explicitaws_access_key_id/aws_secret_access_keyon that route, so per-adapter Access Keys auth silently failed. Explicit keys are honoured from 1.95.1.Also in this PR
reasoning_effortfor non-Anthropic families (Mantle GPT-5.x, gpt-oss, Nova 2). The switch emitted Anthropic'sthinkingblock for every model, which LiteLLM drops for everything except Claude — so the control had never done anything outside Claude.embedding()that Bedrock Cohere embeddings actually use still builds an untimedHTTPHandler.boto3added to sdk1'stestgroup — LiteLLM signs Bedrock requests with SigV4 via botocore, which the test venv needs to exercise signing.Testing
563 passedon litellm 1.96.2. 32 new tests: unit tests for routing, the Mantle strip, and the per-family reasoning shape; plustest_bedrock_mantle_routing.py, which feeds the adapter's output throughlitellm.completion()with the HTTP transport patched and asserts the real wire request — URL, SigV4/Bearer header and body. No AWS credentials, network or spend involved.Reverting the routing makes 14 of the new tests fail, so they genuinely pin the behaviour.
Verified end to end in Docker (all 24 containers), in both the backend and a worker:
Cost resolves for
bedrock_mantle/openai.gpt-5.6-terra=(4.4, 19.8)per 1M, so usage rows won't silently record $0.Known limitations (documented, not fixed here)
us./global.cross-Region inference profile IDs belong tobedrock-runtimeand still don't work; noted in the model field's help text.mode: responses, e.g.openai.gpt-oss-120b) still ignores explicit access keys — an upstream bug. GPT-5.x is unaffected as it uses the Responses path; bearer token and IAM role work everywhere.🤖 Generated with Claude Code