fix: preserve cache_control markers when proxying to CC API - #5
Open
BUKOWSKIREAL wants to merge 1 commit into
Open
fix: preserve cache_control markers when proxying to CC API#5BUKOWSKIREAL wants to merge 1 commit into
BUKOWSKIREAL wants to merge 1 commit into
Conversation
The proxy dropped cache_control on system/user/assistant messages and tools when converting OpenAI/Anthropic requests to CC's internal format (system was flattened to a string, message parts and tools were rebuilt without cache_control). As a result the upstream CC API never created a prompt cache, and cached_tokens / cache_read_input_tokens were always 0 despite the README advertising cache-hit metrics. - buildCcRequest: keep cache_control on system blocks, user text/image parts, assistant text and tools - convertAnthropicToOpenAI: preserve cache_control on system blocks, user text blocks and tools before forwarding to buildCcRequest
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Prompt caching never works through this proxy:
cached_tokens(OpenAI) /cache_read_input_tokens(Anthropic) are always0, even though the README advertises "缓存命中指标 / cache-hit metrics".Root cause
The proxy only reads back cache usage from the CC API response (
event.usage.cachedInputTokens) but never sends thecache_controlmarkers upstream, so the CC API has nothing to cache:buildCcRequestflattenssystemmessages withjoin('\n')into a plain string, droppingcache_controlon system blockstext/image_url) and assistant text are rebuilt withoutcache_controlcache_controlFix
Preserve
cache_controlend-to-end on both conversion paths:buildCcRequest):Arraycontent withcache_control, emit as[{ type: 'text', text, cache_control }]blocks instead of a flattened stringmsg.cache_controlfor string content; keeppart.cache_controlontext/image_urlarray partsmsg.cache_control/part.cache_controlon text blockst.cache_controlonto the CC tool objectconvertAnthropicToOpenAI):cache_controlbefore being handed tobuildCcRequestNo behavior change for requests that don't send
cache_control(system is still flattened to a string as before).Verification
node --check proxy.mjspassescached_tokens > 0(previously always 0)