Skip to content

fix(caching): honor supported_call_types for embeddings and router functions (#41003) - #41008

Open
yuefdev wants to merge 4 commits into
BerriAI:mainfrom
yuefdev:fix/embedding-cache-supported-call-types
Open

fix(caching): honor supported_call_types for embeddings and router functions (#41003)#41008
yuefdev wants to merge 4 commits into
BerriAI:mainfrom
yuefdev:fix/embedding-cache-supported-call-types

Conversation

@yuefdev

@yuefdev yuefdev commented Sep 13, 2026

Copy link
Copy Markdown

TLDR

Problem this solves:

  • Embedding calls were still being cached when supported_call_types excluded embedding/aembedding.
  • Router call types starting with an underscore (e.g. _aembedding) failed exact string match against supported_call_types.

How it solves it:

  • Enforce supported_call_types in Cache.should_use_cache and caching handler methods.
  • Match candidate function/call types with and without leading underscores (.lstrip("_")).
  • Pass call_type and original_function into cache lookup and pipeline write tasks.

User Flow

Before: setting supported_call_types: ["completion", "acompletion"] continues to cache /v1/embeddings requests.

  1. Proxy admin configures cache_params.supported_call_types: ["completion", "acompletion"] to only cache chat completions.
  2. User sends POST /v1/embeddings with input "hello".
  3. Gateway stores embeddings in cache and subsequent calls return cached embedding responses instead of executing upstream.

After: embedding requests bypass cache completely when excluded from supported_call_types.

  1. Proxy admin configures cache_params.supported_call_types: ["completion", "acompletion"].
  2. User sends POST /v1/embeddings with input "hello".
  3. Gateway does not look up or write embeddings into cache; calls are executed upstream as intended.

Relevant issues

Fixes #41003

Pre-Submission checklist

  • I have added meaningful tests
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v.
  • My PR passes all required CI/CD checks
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Type

🐛 Bug Fix

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

@yuefdev
yuefdev requested a review from a team September 13, 2026 14:00
@codecov

codecov Bot commented Sep 13, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.23077% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
litellm/caching/caching_handler.py 93.33% 3 Missing ⚠️

📢 Thoughts on this report? Let us know!

@codspeed-hq

codspeed-hq Bot commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing yuefdev:fix/embedding-cache-supported-call-types (62a81d4) with main (30f33a9)

Open in CodSpeed

@greptile-apps

greptile-apps Bot commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Caching now honors supported_call_types for reads and writes, including embeddings and router functions. It passes the real call type through cache lookups and stores, and matches names with or without leading underscores.

  • Blocks excluded embedding calls from cache reads and writes.
  • Matches router function names after removing leading underscores.
  • Keeps supported async completion lookups and writes using consistent cache keys.
  • Adds coverage for filtering, router prefixes, cache writes, and key consistency.

Confidence Score: 5/5

This PR appears safe to merge.

The cache checks now trust the authoritative call type before request fields. Router prefixes still match after leading underscores are removed. All previous threads are resolved.

Important Files Changed

Filename Overview
litellm/caching/caching.py should_use_cache picks one authoritative call type and removes leading router underscores before checking supported_call_types.
litellm/caching/caching_handler.py Cache reads and writes check the explicit call type first. Embedding results also use the correct sync or async call type.
tests/test_litellm/caching/test_caching_handler.py Tests cover excluded embeddings, router prefixes, stable cache keys, and conflicting call type fields.

Reviews (3): Last reviewed commit: "fix(caching): use authoritative call_typ..." | Re-trigger Greptile

Comment thread litellm/caching/caching_handler.py Outdated
Comment thread litellm/caching/caching_handler.py Outdated
Comment thread tests/test_litellm/caching/test_caching_handler.py Outdated
@yuefdev

yuefdev commented Sep 13, 2026

Copy link
Copy Markdown
Author

@greptileai please re-review this pull request with the latest updates to call type resolution and cache key immutability.

Comment thread litellm/caching/caching_handler.py Outdated
@veria-ai

veria-ai Bot commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

Comment thread litellm/caching/caching_handler.py Outdated
@yuefdev

yuefdev commented Sep 13, 2026

Copy link
Copy Markdown
Author

@greptileai please re-review this pull request with the authoritative call type resolution fix preventing potential bypass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: cache_params.supported_call_types does not disable caching for /v1/embeddings

1 participant