[Bug]: /v1/embeddings returns duplicate "index" values when a batch mixes cached and uncached inputs #41002

What happened?

When a single /v1/embeddings request contains a mix of inputs already in the cache and new inputs, the proxy returns HTTP 200, valid JSON and the correct number of vectors — but with duplicate / incorrect index values.

Any OpenAI-compatible client that maps data[i].embedding back to input[data[i].index] either rejects the response or silently assigns embeddings to the wrong texts.

Minimal reproduction

Proxy config:

litellm_settings:
  cache: true
  cache_params:
    type: local

Model: text-embedding-3-small (OpenAI upstream).

const URL = "http://<proxy>/v1/embeddings";
const KEY = process.env.LITELLM_KEY;
const tag = Math.random().toString(36).slice(2, 10);   // keep inputs unique per run
const T = (n) => `probe ${tag} control text number ${n} — filler`;

const send = async (label, input) => {
  const res = await fetch(URL, {
    method: "POST",
    headers: { Authorization: `Bearer ${KEY}`, "Content-Type": "application/json" },
    body: JSON.stringify({ model: "text-embedding-3-small", input }),
  });
  const j = await res.json();
  const idx = JSON.stringify(j.data.map((e) => e.index));
  const expected = JSON.stringify([...Array(input.length).keys()]);
  console.log(`${label.padEnd(32)} n=${j.data.length} index=${idx}${idx === expected ? "" : "   <-- WRONG"}`);
};

await send("1. all fresh",            [T(1), T(2), T(3)]);
await send("2. same batch (all cached)", [T(1), T(2), T(3)]);
await send("3. two cached + one fresh",  [T(1), T(2), T(9)]);
await send("4. fresh first",             [T(10), T(1), T(2)]);
await send("5. interleaved",             [T(1), T(11), T(2), T(12)]);
await send("6. four fresh (control)",    [T(20), T(21), T(22), T(23)]);

Relevant log output

1. all fresh                     n=3 index=[0,1,2]
2. same batch (all cached)       n=3 index=[0,1,2]
3. two cached + one fresh        n=3 index=[0,1,0]     <-- WRONG
4. fresh first                   n=3 index=[0,1,2]
5. interleaved                   n=4 index=[0,0,2,1]   <-- WRONG
6. four fresh (control)          n=4 index=[0,1,2,3]

All-fresh is fine. All-cached is fine. Only the mixed batches are corrupted, and reproducibly so — the same cases fail on repeated runs.

Counter-proof: it is the cache

Same patterns, same run, with per-request cache control:

body: JSON.stringify({ model: "text-embedding-3-small", input, cache: { "no-cache": true } })

with cache:    2/4 wrong
with no-cache: 0/4 wrong

Identical inputs, identical batch shapes — the only difference is whether the cache participates.

Expected behaviour

data[*].index must be a permutation of 0 … input.length - 1, with no duplicates and no gaps, regardless of which entries were served from the cache.

Impact

This is silent data corruption for any batching client. A strict client rejects the response and the whole operation fails; a permissive client stores embeddings against the wrong source text, which is worse because nothing surfaces.

We hit it while building a semantic index: batches of chunks kept failing validation with no obvious pattern, because whether a batch is "mixed" depends on cache state, not on the data.

Are you a ML Ops Team?

No

What LiteLLM version are you on?

v1.99.0