fix(caching): honor supported_call_types for embeddings and router functions (#41003) - #41008
fix(caching): honor supported_call_types for embeddings and router functions (#41003)#41008yuefdev wants to merge 4 commits into
Conversation
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Greptile SummaryCaching now honors
Confidence Score: 5/5This PR appears safe to merge. The cache checks now trust the authoritative call type before request fields. Router prefixes still match after leading underscores are removed. All previous threads are resolved.
|
| Filename | Overview |
|---|---|
| litellm/caching/caching.py | should_use_cache picks one authoritative call type and removes leading router underscores before checking supported_call_types. |
| litellm/caching/caching_handler.py | Cache reads and writes check the explicit call type first. Embedding results also use the correct sync or async call type. |
| tests/test_litellm/caching/test_caching_handler.py | Tests cover excluded embeddings, router prefixes, stable cache keys, and conflicting call type fields. |
Reviews (3): Last reviewed commit: "fix(caching): use authoritative call_typ..." | Re-trigger Greptile
…prevent kwargs key pollution
|
@greptileai please re-review this pull request with the latest updates to call type resolution and cache key immutability. |
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 1 · PR risk: 0/10 |
|
@greptileai please re-review this pull request with the authoritative call type resolution fix preventing potential bypass. |
TLDR
Problem this solves:
supported_call_typesexcludedembedding/aembedding._aembedding) failed exact string match againstsupported_call_types.How it solves it:
supported_call_typesinCache.should_use_cacheand caching handler methods..lstrip("_")).call_typeandoriginal_functioninto cache lookup and pipeline write tasks.User Flow
Before: setting
supported_call_types: ["completion", "acompletion"]continues to cache/v1/embeddingsrequests.cache_params.supported_call_types: ["completion", "acompletion"]to only cache chat completions.POST /v1/embeddingswith input"hello".After: embedding requests bypass cache completely when excluded from
supported_call_types.cache_params.supported_call_types: ["completion", "acompletion"].POST /v1/embeddingswith input"hello".Relevant issues
Fixes #41003
Pre-Submission checklist
uv run pytest tests/test_litellm/<your_test_file>.py -v.Type
🐛 Bug Fix
Final Attestation