fix(llm): strip LiteLLM-internal optional_params from provider request bodies by mateo-berri · Pull Request #30774 · BerriAI/litellm

mateo-berri · 2026-06-18T18:11:50Z

Relevant issues

Fixes #30371 (Bedrock invoke leaks internal params), fixes #30314 (Titan embeddings rejected on cache_control_injection_points), and addresses #30301 (harden transforms against internal optional_params leaking). Design writeup in #30769.

Linear ticket

N/A

Pre-Submission checklist

I have added meaningful tests
My PR passes all unit tests on make test-unit (touched suites green; see below)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have requested a Greptile review and received a Confidence Score of at least 4/5

Changes

optional_params is an untyped dict that carries both provider inference params and LiteLLM-internal control knobs (skip_mcp_handler, stream_chunk_size, fake_stream, cache_control_injection_points), and the request transforms splat the whole dict into the wire payload. Strict-schema providers reject unknown fields, so a leaked knob fails the whole request with a 400.

This introduces a typed registry, LiteLLMInternalParam in litellm/types/internal_params.py, as the single source of truth, and a strip_internal_params_from_request_body filter derived from it. The filter runs at the serialization boundary: the common HTTP handlers (llm_http_handler chat and embedding paths, aiohttp_handler, which cover every provider that routes through them) and the bespoke Bedrock invoke, embedding, and image transforms. A new internal knob is now covered by adding one enum member instead of remembering to pop it at each splat site.

The embedding path through llm_http_handler.embedding() strips at the same boundary as chat, so providers that splat optional_params into the embedding body (e.g. VoyageAI builds {"input": ..., "model": ..., **optional_params}) no longer leak. Inside the Bedrock invoke transform a single sanitized copy now feeds both the inference_params branches and the anthropic/nova/twelvelabs/openai delegate sub-transforms, so a direct transform_request call strips consistently regardless of provider; this also drops the now-redundant stream_chunk_size pop and the caller-dict mutation.

The chat boundary needs one nuance. cache_control_injection_points is also a body-shaping directive that AmazonConverseConfig reads to append a cachePoint to the Bedrock tool list for location: "tool_config", on the converse_like/ and claude-platform routes that share this handler. So the chat handler keeps that one key in optional_params before transform_request (via strip_internal_params_from_chat_request_body), then re-applies the full strip_internal_params_from_request_body to the body transform_request returns. Splat-style chat transforms that never pop it (OpenAI, Anthropic) therefore cannot leak it into the wire payload, while converse still consumes it. Embeddings, Bedrock invoke and the image transforms have no such consumer, so they keep the full strip throughout.

filter_internal_params keeps its MCP-only behavior for fallback re-dispatch, so cache_control_injection_points is still preserved on a retried call; only the body boundary strips the full set. Titan image generation already builds a structured body so it needed no change.

Screenshots / Proof of Fix

Live proxy (litellm/proxy/proxy_cli.py) against real Bedrock (bearer-token auth, us-east-1), forcing the internal keys into optional_params with allowed_openai_params. Same proxy, same requests, run on the pre-fix tree and then the fixed tree.

#30314, Titan embeddings, cache_control_injection_points forced:

curl -s http://localhost:4000/v1/embeddings -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d '{"model":"bedrock-titan-embed","input":"hello world","allowed_openai_params":["cache_control_injection_points"],"cache_control_injection_points":[{"location":"message","role":"user"}]}'

Before:

{"error":{"message":"litellm.BadRequestError: BedrockException - {\"message\":\"Malformed input request: extraneous key [cache_control_injection_points] is not permitted, please reformat your input and try again.\"}. ...","code":"400"}}
HTTP 400

After:

{"model":"bedrock-titan-embed","data":[{"embedding":[-0.0206023...,0.0566126...],"index":0,"object":"embedding"}], ...}
HTTP 200

#30371, Mistral invoke, skip_mcp_handler (+ cache_control_injection_points) forced:

curl -s http://localhost:4000/v1/chat/completions -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d '{"model":"bedrock-invoke-mistral","messages":[{"role":"user","content":"say hi in exactly 3 words"}],"max_tokens":20,"allowed_openai_params":["skip_mcp_handler","cache_control_injection_points"],"skip_mcp_handler":true,"cache_control_injection_points":[{"location":"message","role":"user"}]}'

Before:

{"error":{"message":"litellm.BadRequestError: BedrockException - {\"message\":\"Validation Error: 1 validation error for TextToTextCompletionRequest\\nskip_mcp_handler\\n  Extra inputs are not permitted [type=extra_forbidden, ...]\"}. ...","code":"400"}}
HTTP 400

After:

{"id":"chatcmpl-...","model":"bedrock-invoke-mistral","object":"chat.completion","choices":[{"finish_reason":"length","index":0,"message":{"content":"\nHello, I'm here!\n...","role":"assistant"}}],"usage":{"completion_tokens":20,"prompt_tokens":17,"total_tokens":37}}
HTTP 200

Regression tests covering the chat, embedding and Bedrock invoke paths (including the four invoke delegate providers) fail when strip_internal_params_from_request_body is made a passthrough and pass with the fix.

Type

🐛 Bug Fix

Note

Medium Risk
Touches the shared request path for all providers routed through HTTP handlers and several Bedrock transforms; incorrect stripping could drop valid provider params or break Bedrock tool-config caching, though tests target both leak and preserve behavior.

Overview
Adds a central LiteLLMInternalParam registry and request-body filters so LiteLLM-only keys in optional_params (e.g. skip_mcp_handler, stream_chunk_size, fake_stream, cache_control_injection_points) are not splatted into provider payloads, fixing Bedrock and other strict-schema 400 rejections (#30371, #30314, #30301).

strip_internal_params_from_request_body runs at the serialization boundary on shared HTTP handlers (chat, embedding, aiohttp) and on Bedrock invoke/embed/image transforms. Chat uses a two-step flow: strip_internal_params_from_chat_request_body before transform_request keeps cache_control_injection_points for AmazonConverseConfig, then the full strip on the returned body blocks splat-style transforms from leaking it. filter_internal_params stays MCP-only for fallback re-dispatch.

Regression tests cover registry stripping, invoke delegate paths, Titan embeddings, and handler chat/embedding boundaries.

^{Reviewed by Cursor Bugbot for commit 0fee6ac. Bugbot is set up for automated code reviews on this repo. Configure here.}

…t bodies optional_params mixes provider inference params with LiteLLM-internal control knobs (skip_mcp_handler, stream_chunk_size, fake_stream, cache_control_injection_points), and the request transforms splat the whole dict into the wire payload. Strict-schema providers reject unknown fields, so a leaked knob fails the whole request with a 400. Introduce a typed registry (LiteLLMInternalParam) as the single source of truth and a strip_internal_params_from_request_body filter derived from it, applied at the serialization boundary: the common HTTP handlers (covering every provider that routes through them) and the bespoke Bedrock invoke, embedding, and image transforms. filter_internal_params keeps its MCP-only behavior for fallback re-dispatch so cache_control_injection_points is not dropped from retries. Fixes #30371 Fixes #30314 Addresses #30301

mateo-berri · 2026-06-18T18:12:09Z

@greptileai

codecov · 2026-06-18T18:15:36Z

Codecov Report

❌ Patch coverage is 92.10526% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/litellm_core_utils/core_helpers.py	90.90%	1 Missing ⚠️
...age_generation/amazon_stability1_transformation.py	50.00%	1 Missing ⚠️
litellm/llms/custom_httpx/aiohttp_handler.py	50.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

greptile-apps · 2026-06-18T18:26:28Z

Greptile Summary

This PR fixes a class of bugs where LiteLLM-internal control knobs (skip_mcp_handler, cache_control_injection_points, stream_chunk_size, etc.) were leaking into provider request bodies through optional_params splats, causing strict-schema providers (Bedrock Titan embeddings, Bedrock invoke Mistral) to reject requests with 400 errors.

Introduces LiteLLMInternalParam enum as a single source of truth and derives two boundary-strip functions: strip_internal_params_from_request_body (full strip) and strip_internal_params_from_chat_request_body (preserves cache_control_injection_points so Bedrock Converse tool-config caching still works before transform, then the full strip runs on the body returned by transform_request).
Wires the filters into the shared llm_http_handler (chat + embedding), aiohttp_handler (chat), and the bespoke Bedrock invoke/embed/image transforms; all nine Bedrock invoke provider branches now receive a sanitized copy of optional_params, covering the four delegate sub-transforms (anthropic, nova, twelvelabs, openai) that were previously missed. Regression tests cover the registry strip, every invoke branch, the Titan embedding path, and the two-pass chat-boundary behavior.

Confidence Score: 5/5

Safe to merge — all previously identified gaps (embedding path, four delegate invoke paths, redundant manual pop) have been addressed in this version, and the narrowing of filter_internal_params to MCP-only is a behavioral no-op since that function already filtered exactly those three keys.

The change is well-scoped: a new typed registry drives the strip, the strip is applied at all identified splat boundaries, a deep copy is used in the Bedrock invoke path to avoid mutating the caller's dict, and the chat two-pass pattern correctly handles the cache_control_injection_points carve-out. Regression tests are comprehensive across all provider branches.

No files require special attention — all modified paths have corresponding regression tests.

Important Files Changed

Filename	Overview
litellm/types/internal_params.py	New file introducing the LiteLLMInternalParam enum registry and three derived frozensets for request-body stripping; well-structured and correctly parameterized for Python 3.10+.
litellm/litellm_core_utils/core_helpers.py	Adds strip_internal_params_from_request_body and strip_internal_params_from_chat_request_body; narrows filter_internal_params to MCP-only (behavioral no-op since it already filtered those same three keys).
litellm/llms/bedrock/chat/invoke_transformations/base_invoke_transformation.py	All provider branches now receive sanitized_params; the redundant stream_chunk_size pop is removed; a deep copy is made before stripping to preserve caller's dict.
litellm/llms/custom_httpx/llm_http_handler.py	Chat path uses two-pass strip (chat variant before transform, full strip on the body after); embedding path now strips before transform_embedding_request; both previously-flagged gaps are closed.
litellm/llms/custom_httpx/aiohttp_handler.py	Applies same two-pass strip pattern as llm_http_handler for the async chat path.
litellm/llms/bedrock/embed/embedding.py	Titan/Cohere embedding now strips internal params from the deep-copied optional_params before building inference_params, closing the #30314 regression.
litellm/llms/bedrock/image_generation/amazon_nova_canvas_transformation.py	Strips internal params before splatting optional_params into image_generation_config.
litellm/llms/bedrock/image_generation/amazon_stability1_transformation.py	Applies strip before copying optional_params into inference_params for Stability v1 image bodies.
litellm/llms/bedrock/image_generation/amazon_stability3_transformation.py	Strips internal params before splatting into AmazonStability3TextToImageRequest constructor.
tests/test_litellm/litellm_core_utils/test_core_helpers.py	Adds TestInternalParamFiltering with four unit tests; existing tests reformatted but not weakened.
tests/test_litellm/llms/bedrock/chat/invoke_transformations/test_base_invoke_transformation.py	Adds two parametrized regression tests covering all nine invoke provider branches.
tests/test_litellm/llms/bedrock/embed/test_bedrock_embedding.py	Adds regression test for #30314 verifying no internal params appear in the Titan request body.
tests/test_litellm/llms/custom_httpx/test_llm_http_handler.py	Adds three handler-level tests covering the embedding strip and chat two-pass boundary behavior.

_{Reviews (5): Last reviewed commit: "fix(llm): strip internal params from cha..." | Re-trigger Greptile}

greptile-apps · 2026-06-18T18:28:45Z

Greptile Summary

This PR introduces a typed registry (LiteLLMInternalParam) as the single source of truth for LiteLLM-internal optional_params keys and a strip_internal_params_from_request_body filter that runs at the serialization boundary to prevent these knobs from reaching strict-schema providers like Bedrock.

Adds litellm/types/internal_params.py with the enum registry and two frozensets (LITELLM_INTERNAL_REQUEST_BODY_PARAMS for body stripping, MCP_INTERNAL_PARAMS for fallback re-dispatch), then wires the filter into llm_http_handler, aiohttp_handler, and all relevant Bedrock bespoke transforms (invoke, embed, image generation).
Regression tests cover Titan embedding (fixes [Bug]: Malformed input request: #: extraneous key [cache_control_injection_points] is not permitted #30314), Bedrock invoke models (fixes [Bug]: Bedrock invoke path leaks LiteLLM-internal params into the request body (Converse already filters them) #30371), and the filter_internal_params / strip_internal_params_from_request_body distinction; the PR description includes live-proxy before/after evidence.

Confidence Score: 4/5

Safe to merge for the targeted Bedrock use-cases; two gaps in coverage (embedding path in llm_http_handler, and four delegate branches in base_invoke_transformation) are non-regressions and low-risk.

The core fix (registry, filter function, Bedrock bespoke paths) is solid and well-tested. The two gaps noted are incomplete coverage of the stated 'all providers that route through the common handlers' goal rather than new regressions — neither introduces behaviour worse than the pre-PR state, and Bedrock (the reported culprit) is fully protected.

litellm/llms/custom_httpx/llm_http_handler.py (embedding method missing the strip) and litellm/llms/bedrock/chat/invoke_transformations/base_invoke_transformation.py (anthropic/nova/twelvelabs/openai delegate paths pass unstripped optional_params).

Important Files Changed

Filename	Overview
litellm/types/internal_params.py	New typed registry (LiteLLMInternalParam enum + two frozensets) as the single source of truth for internal optional_params keys — clean design, no issues found.
litellm/litellm_core_utils/core_helpers.py	Adds strip_internal_params_from_request_body and narrows filter_internal_params to MCP-only params; docstrings clearly distinguish the two functions.
litellm/llms/custom_httpx/llm_http_handler.py	Strip applied correctly to the chat-completion transform_request call but missing from the embedding path (transform_embedding_request, line 895-900).
litellm/llms/custom_httpx/aiohttp_handler.py	Strip correctly applied to the chat-completion transform_request call; no embedding path in this handler.
litellm/llms/bedrock/chat/invoke_transformations/base_invoke_transformation.py	Strip applied correctly for inference_params paths (cohere/mistral/ai21/amazon/meta/etc.) but the anthropic, nova, twelvelabs, and openai delegate paths pass the unstripped optional_params.
litellm/llms/bedrock/embed/embedding.py	Correctly applies strip_internal_params_from_request_body before building inference_params, directly fixing the Titan embedding 400-error regression.
litellm/llms/bedrock/image_generation/amazon_nova_canvas_transformation.py	Strips internal params before merging optional_params into image_generation_config — correct fix.
litellm/llms/bedrock/image_generation/amazon_stability1_transformation.py	Strip applied correctly to inference_params deep copy before building the Stability 1 request body.
litellm/llms/bedrock/image_generation/amazon_stability3_transformation.py	Strip applied correctly before constructing the Pydantic AmazonStability3TextToImageRequest — prevents unknown-field validation errors.
tests/test_litellm/litellm_core_utils/test_core_helpers.py	Adds comprehensive unit tests for strip_internal_params_from_request_body, including full registry coverage, pass-through of native params, and non-dict handling; formatting reformats to existing tests are benign.
tests/test_litellm/llms/bedrock/chat/invoke_transformations/test_base_invoke_transformation.py	Good regression tests for the mistral/cohere/titan/llama/ai21 invoke paths; anthropic, nova, twelvelabs, and openai invoke delegates are not tested (they pass unstripped optional_params to sub-transforms).
tests/test_litellm/llms/bedrock/embed/test_bedrock_embedding.py	Solid regression test covering all three Titan embedding models; verifies the full internal param set is absent from the POST body while inputText survives.

Comments Outside Diff (2)

litellm/llms/custom_httpx/llm_http_handler.py, line 895-900 (link)

Embedding path misses the body-boundary strip

transform_embedding_request receives the raw, unstripped optional_params while the chat-completion path right above it (line 450) was given strip_internal_params_from_request_body(optional_params). Any strict-schema provider that routes through llm_http_handler.embedding for embeddings would still receive internal control knobs (e.g., cache_control_injection_points, skip_mcp_handler) in its request body and fail with a 400. Bedrock is protected by its own bespoke handler, but the generic path is inconsistent with the stated goal of covering "every provider that routes through" these common handlers.
litellm/llms/bedrock/chat/invoke_transformations/base_invoke_transformation.py, line 195-254 (link)

Four delegate paths bypass the bespoke strip

The fix at line 168-170 builds inference_params as a stripped copy and protects the cohere, ai21, mistral, amazon, meta/llama, and deepseek_r1 branches. However, the anthropic, nova, twelvelabs, and openai branches pass the original (un-deep-copied, unstripped) optional_params to their delegate transform_request calls. When called through llm_http_handler, the params are pre-stripped so there is no immediate issue, but calling AmazonInvokeConfig().transform_request() directly with internal params (as the new regression test does for the first five providers) would still leak them through these four branches. Adding strip_internal_params_from_request_body to the delegate calls — or passing inference_params where possible — would close this gap and make the defense consistent regardless of call site.

_{Reviews (2): Last reviewed commit: "fix(llm): strip LiteLLM-internal optiona..." | Re-trigger Greptile}

…delegates The request-body filter was wired into the chat path but not into llm_http_handler.embedding(), so providers that splat optional_params into the embedding body (e.g. VoyageAI builds {"input": ..., "model": ..., **optional_params}) still leaked internal knobs and could 400 on a strict schema. Apply strip_internal_params_from_request_body there too. In AmazonInvokeConfig.transform_request the inference_params branches were stripped but the anthropic, nova, twelvelabs and openai delegate branches forwarded the raw optional_params to their sub-transforms. Build one sanitized copy up front and use it for both the inference_params branches and the delegates; this also drops the now-redundant stream_chunk_size pop and the caller-dict mutation. Adds regression coverage for the embedding path and the four invoke delegate providers.

mateo-berri · 2026-06-19T04:05:33Z

@greptileai

mateo-berri · 2026-06-19T04:11:14Z

bugbot run

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

✅ Fixed: Strips cache injection before Converse
- Added a chat-completion variant of the strip helper that preserves cache_control_injection_points so AmazonConverseConfig.transform_request can still append a tool_config cachePoint on the converse_like/ route, while the embedding boundary and splat-style transforms keep the full strip.

_{You can send follow-ups to the cloud agent here.}

…boundary AmazonConverseConfig.transform_request reads cache_control_injection_points to append a cachePoint to the Bedrock tool list for location: tool_config. Stripping that key at the shared HTTP handler boundary silently disabled tool-config prompt caching on the bedrock/converse_like/ route, which is dispatched through base_llm_http_handler.completion. Split the strip set: chat-completion paths use a smaller set that preserves provider-consumed params, while the embedding boundary and splat-style transforms keep the full strip via strip_internal_params_from_request_body.

CLAassistant · 2026-06-19T04:22:42Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ mateo-berri
❌ cursoragent
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

mateo-berri · 2026-06-19T04:31:37Z

@greptileai

mateo-berri · 2026-06-19T04:36:36Z

bugbot run

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

✅ Fixed: Internal param leaks on chat splat
- Confirmed the leak: OpenAIGPTConfig.transform_request and similar splat-style configs return {**optional_params, ...} without popping cache_control_injection_points, so the chat-only strip would let it reach the wire body; fixed by re-applying strip_internal_params_from_request_body to the dict returned by transform_request inside the shared HTTP/aiohttp handlers, which is a no-op for AmazonConverseConfig (it already pops the key) but closes the splat leak.

_{You can send follow-ups to the cloud agent here.}

The chat-completion boundary preserves cache_control_injection_points in optional_params so AmazonConverseConfig.transform_request can pop it (used for Bedrock tool_config cachePoint injection on the converse_like/ route). However, splat-style transforms such as OpenAIGPTConfig and AnthropicConfig build the wire body with ```**optional_params``` and never pop that key, so the preserved key leaked straight into the wire payload and re-introduced the extraneous-field 400s on strict-schema providers. Apply strip_internal_params_from_request_body to the dict returned by transform_request inside the shared HTTP handler. For Converse the key was already popped, so the post-transform strip is a no-op; for splat transforms it closes the leak at the serialization boundary without touching every provider's transform_request.

mateo-berri · 2026-06-19T04:46:41Z

@greptileai

mateo-berri · 2026-06-19T05:00:11Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 0fee6ac. Configure here.}

cursor Bot reviewed Jun 19, 2026

View reviewed changes

Comment thread litellm/llms/custom_httpx/llm_http_handler.py Outdated

cursor Bot reviewed Jun 19, 2026

View reviewed changes

Comment thread litellm/llms/custom_httpx/llm_http_handler.py

cursor Bot reviewed Jun 19, 2026

View reviewed changes

yuneng-berri deleted the branch main September 13, 2026 04:25

yuneng-berri closed this Sep 13, 2026

yuneng-berri reopened this Sep 13, 2026

mateo-berri changed the base branch from litellm_internal_staging to main September 13, 2026 07:46

mateo-berri requested a review from a team September 13, 2026 07:46

fangkangmi mentioned this pull request Sep 13, 2026

fix(llm): filter internal params at provider request boundaries #41025

Closed

6 tasks

Uh oh!

Uh oh!

Conversation

mateo-berri commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Relevant issues

Linear ticket

Pre-Submission checklist

Changes

Screenshots / Proof of Fix

Type

Uh oh!

mateo-berri commented Jun 18, 2026

Uh oh!

codecov Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

greptile-apps Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

greptile-apps Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Comments Outside Diff (2)

Uh oh!

mateo-berri commented Jun 19, 2026

Uh oh!

mateo-berri commented Jun 19, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

CLAassistant commented Jun 19, 2026

Uh oh!

mateo-berri commented Jun 19, 2026

Uh oh!

mateo-berri commented Jun 19, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mateo-berri commented Jun 19, 2026

Uh oh!

mateo-berri commented Jun 19, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mateo-berri commented Jun 18, 2026 •

edited

Loading

codecov Bot commented Jun 18, 2026 •

edited

Loading

greptile-apps Bot commented Jun 18, 2026 •

edited

Loading

greptile-apps Bot commented Jun 18, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading