fix(llm): filter internal params at provider request boundaries - #41025

TLDR

Problem this solves:

• Internal parameters cause Bedrock chat and embedding HTTP 400s
• Shared provider handlers can serialize the same internal parameters

How it solves it:

• Strip registered internal keys at provider request boundaries
• Preserve fallback credentials, streaming controls, and Converse caching
• Cover synchronous and asynchronous transforms before request signing

Continues Mateo Wang's implementation in #30774 on current main. I cannot push to that upstream branch. This version resolves its conflicts, handles the newer asynchronous transform path, preserves Nova Invoke tool caching, and prevents embedding extra_body from reintroducing internal keys

User Flow

Before: requests carrying internal parameters fail against Bedrock

1. The admin configures the two Bedrock models shown below
2. The caller sends POST http://127.0.0.1:4017/v1/embeddings with the embedding payload below
3. The caller receives HTTP 400 naming cache_control_injection_points
4. The caller sends POST http://127.0.0.1:4017/v1/chat/completions with the chat payload below
5. The caller receives HTTP 400 naming skip_mcp_handler

After: the same payloads return embeddings and a completion

1. The admin starts the fixed proxy with the same model configuration
2. The caller sends POST http://127.0.0.1:4018/v1/embeddings with the same embedding payload
3. The caller receives HTTP 200 with one 1,024-dimensional embedding
4. The caller sends POST http://127.0.0.1:4018/v1/chat/completions with the same chat payload
5. The caller receives HTTP 200 with assistant content and token usage

Relevant issues

Fixes #30371
Fixes #30314
Addresses #30301
Continues #30774

Affected release

Linear ticket

Pre-Submission checklist

• I have added meaningful tests
• The handful of test files covering my change pass locally: 600 tests across ten affected files
• My PR passes all required CI/CD checks: all 27 required contexts pass
• My PR's scope is as isolated as possible; it only solves one specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review: 5/5 at 9e1c16b

Local validation at 9e1c16ba57: 600 affected tests passed, the complete repository quality gate passed, and four fresh-process import-order checks passed. Filters now live in an independent utility module to remove the provider import cycles reported by CodeQL. All 27 required CI contexts pass. Patch coverage is 93.18% against an 80.91% target. Greptile scored this tip 5/5; Bugbot and Veria found no issues. CodeQL and its Python analysis pass with no new alerts. The separate OSV dependency finding is documented below

Screenshots / Proof of Fix

Real Bedrock calls through local proxies on 14 September 2026, using bearer-token credentials supplied through the environment. No mocked provider responses. Both proxies use this configuration:

model_list:
  - model_name: bedrock-titan-embed
    litellm_params:
      model: bedrock/amazon.titan-embed-text-v2:0
      aws_region_name: us-east-1
      api_key: os.environ/AWS_BEARER_TOKEN_BEDROCK
  - model_name: bedrock-invoke-mistral
    litellm_params:
      model: bedrock/mistral.mistral-7b-instruct-v0:2
      aws_region_name: us-east-1
      api_key: os.environ/AWS_BEARER_TOKEN_BEDROCK

The payloads intentionally use allowed_openai_params to exercise the provider boundary with internal keys present, matching the original PR's reproduction

Before (30f33a9)

Titan embeddings

1. Send:

   curl -s -w '\n%{http_code}\n' http://127.0.0.1:4017/v1/embeddings \
     -H 'Content-Type: application/json' \
     -d '{"model":"bedrock-titan-embed","input":"hello world","allowed_openai_params":["cache_control_injection_points"],"cache_control_injection_points":[{"location":"message","role":"user"}]}'

2. HTTP 400, provider error excerpt:

   Malformed input request: extraneous key [cache_control_injection_points] is not permitted, please reformat your input and try again.
   

Mistral Invoke chat

1. Send:

   curl -s -w '\n%{http_code}\n' http://127.0.0.1:4017/v1/chat/completions \
     -H 'Content-Type: application/json' \
     -d '{"model":"bedrock-invoke-mistral","messages":[{"role":"user","content":"Say hello"}],"max_tokens":20,"allowed_openai_params":["skip_mcp_handler"],"skip_mcp_handler":true}'

2. HTTP 400, provider error excerpt:

   Validation Error: 1 validation error for TextToTextCompletionRequest
   skip_mcp_handler
     Extra inputs are not permitted [type=extra_forbidden, input_value=True, input_type=bool]
   

After (9e1c16b)

Titan embeddings

1. Repeat the embedding curl command above using port 4018
2. HTTP 200. Response summary: one embedding with 1,024 dimensions, prompt_tokens: 3, total_tokens: 3

Mistral Invoke chat

1. Repeat the chat curl command above using port 4018
2. HTTP 200, response excerpt:

   {
     "choices": [
       {
         "finish_reason": "length",
         "message": {
           "content": "\nHello there! How's your day going? Is there anything specific you'd like to",
           "role": "assistant"
         }
       }
     ],
     "usage": {
       "completion_tokens": 20,
       "prompt_tokens": 13,
       "total_tokens": 33
     }
   }

Type

Bug Fix

Caveats (if any)

Medium

• Dependency scan flags MLflow 3.15.0 in unchanged upstream lockfile

The OSV run reports PYSEC-2026-3865 / GHSA-h7x2-h6g9-p789, with no fixed version listed. uv.lock, the dashboard lockfile, and osv-scanner.toml have identical Git blob IDs at main and this PR tip. This is an existing dependency finding, not a passed check; no vulnerability ignores or dependencies were changed

Low

• Nova tool-cache preservation has regression coverage; no live Nova caching measurement

• Legacy handlers outside these boundaries are not comprehensively audited

• Live verification covers Bedrock chat and embeddings; other paths have unit coverage

• Historical reviews on fix(llm): strip LiteLLM-internal optional_params from provider request bodies #30774 do not validate this new tip

• Generated API types remove two stale documentation lines

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR