Skip to content

feat(aibridge): add shared extract package with OpenAI Responses extractor - #30211

Closed
ThomasK33 wants to merge 10 commits into
tk/ai-gateway-responses-ws-experimentfrom
tk/ai-gateway-extract-responses
Closed

ThomasK33 wants to merge 10 commits into
tk/ai-gateway-responses-ws-experimentfrom
tk/ai-gateway-extract-responses

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

Adds aibridge/extract, a shared package of read-only traffic extractors, and aibridge/extract/responses, an OpenAI Responses extractor. The interceptors do not use them yet. Two upcoming consumers need them: the AI Gateway reverse proxy (AIGOV-616 and AIGOV-618 will build their Anthropic and Chat Completions extractors on these interfaces) and the OpenAI Responses WebSocket relay. The interface shape was reviewed with the reverse proxy's owner, Paweł.

The extractors never touch, block or alter traffic. They write recorder records as they observe traffic, and a parse failure only reduces the recorded detail: it is logged as a bounded parse note, never returned as a traffic error. The interceptor code (aibridge/intercept/...) is unchanged.

Interface

package extract

const MaxEventBytes = 4 << 20 // per streamed event
const MaxBodyBytes = 8 << 20  // per complete body, after decoding

type RequestFacts struct {
	Model                 string
	Streaming             bool
	Prompt                string // last user prompt; passed to the response extraction
	CorrelatingToolCallID string // e.g. call_id of a trailing function_call_output
}

type RequestExtractor interface {
	ExtractRequest(body []byte) (RequestFacts, error) // error = parse note, facts partial
}

// Not safe for concurrent use: call sequentially, feeding events in stream order.
type ResponseExtraction interface {
	OnEvent(eventType string, raw []byte)       // SSE event data or a raw WebSocket frame
	ProcessBlocking(statusCode int, raw []byte) // complete decoded body: 2xx response or 4xx/5xx error
	Outcome() Outcome
}

type Outcome struct {
	ResponseID string
	Terminal   Terminal // Status: "", completed, incomplete, failed; Reason: incomplete reason or error code
	Err        error    // provider-shaped (*intercept.ResponseError) for interceptionerror.Categorize
}

func DecodeBody(contentEncoding string, raw []byte) ([]byte, error) // gzip, 8 MiB bound
func NewSSEStream(ctx context.Context, logger slog.Logger, ext ResponseExtraction) *SSEStream // io.WriteCloser

responses.NewResponseExtraction(ctx, logger, recorder.Recorder, interceptionID, prompt) records online. Recorder calls run synchronously and are not bounded per call, so the caller supplies a recorder (or context) that enforces a deadline. responses.Relevant(eventType) reports which event types the extraction acts on, so callers can skip deltas.

  • The prompt is recorded once, as soon as the first response ID is seen.
  • Token usage, tool usage (function, custom and hosted) and model thoughts (reasoning summaries and commentary) are recorded once, from the terminal response object. They keep the interceptor's field semantics:
    • MsgID is the provider response ID.
    • Non-cached input is max(0, input - cached - cache_write).
    • output_reasoning and total_tokens are extra token types.
    • service_tier is recorded as metadata.

Design notes:

  • Bodies are []byte. One reusable buffer is filled once and shared between forwarding and extraction. A streaming io.Reader input is a possible future direction.
  • The SSE splitter is new. SSEStream does not reuse aibridge.SSEParser. That parser lives in the root package, has a single test caller, reads the whole stream before returning, buffers every event, and fails on lines over 64 KiB. SSEStream splits events incrementally, follows the same field rules, and bounds each event (event name included) at 4 MiB.
  • Provider errors are classified in the extractor. 4xx/5xx bodies, SSE error events and response.failed become *intercept.ResponseError, so interceptionerror.Categorize keeps classifying through Provider.CategorizeError. The first terminal event decides the outcome: a later error is a parse note, and a terminal response after an error is recorded but keeps the error outcome.
  • The SSE splitter follows the specification: CR, LF and CRLF line endings (a CRLF may be split across writes) and one leading UTF-8 BOM, stripped.

Intentional differences from the interceptor

The parity test encodes these; the responses package doc lists them too.

  • Mid-stream errors are errors. On streaming/stream_error and streaming/stream_failure, the extractor returns a provider-shaped error; the interceptor records success. This follows the AIGOV-616 spec, which requires error and response.failed events to be terminal errors.
    • error events carry no HTTP status, so the error has status 0 and categorizes as unknown, the same as the interceptor's SDK stream errors. A WebSocket error frame that includes a status keeps it.
    • response.failed maps its error code to a status, as the Responses WebSocket relay does: rate_limit_exceeded is 429, server_error is 500, other codes are 400, and no code is 0. stream_failure (server_error) therefore categorizes as a server error.
  • A malformed 2xx body (blocking/wrong_response_format) is only a parse note, where the interceptor returns an unknown decode error.
  • Streamed response.incomplete and response.failed record usage, tools and thoughts from their response object, as the blocking path already does.
  • Missing data records nothing. A response with no usage object records no token row, and an empty prompt is not recorded.
  • No injected tools. The extractor does not inject MCP tools or run the inner tool loop, so the injected-tool fixtures run without an MCP proxy.

Testing

  • Parity: aibridge/internal/integrationtest/extract_parity_internal_test.go replays every aibridge/fixtures/openai/responses/{blocking,streaming}/*.txtar fixture (33 today; new ones are picked up automatically). Each fixture goes through the real interceptor and through the extractor, and the test compares prompt, token, tool and thought records plus error categorization.
  • Unit tests: transports (WebSocket frames against SSE split byte by byte, CRLF, comments, multi-line data), terminal-error categorization against real openai.Error values, fail-open cases (malformed, oversized and truncated input, gzip errors, duplicate or late terminal events), SSE and body bounds, note bounds, and request facts.
  • Remote dogfood UAT passed on 5e4cef1f03, before the late test and fix commits and the X1 merge:
    • go test -race on the new packages and the parity test passed in a fresh workspace.
    • A live OpenAI Responses request, blocking and streaming, went through ./scripts/develop.sh and appeared with usage on the AI Gateway Sessions page (the interceptors are unchanged).
    • Two low-severity findings from its probe tests are fixed in fix(aibridge/extract): settle outcome after terminal and bound note size. Later commits fix Codex findings (terminal-after-error outcome, bare CR, leading BOM) and add the response.failed code mapping, each with a test that failed before the change.

Stacked on tk/ai-gateway-responses-ws-experiment.


Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-30T22:29:51.304128Z 6fd7d19 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a83a8aafc9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread aibridge/extract/responses/response.go Resolved
Comment thread aibridge/extract/sse.go Resolved
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9d1e1370a1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread aibridge/extract/sse.go Resolved
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

CI note: pushes of 66bcf55 and 5023477 did not start a pull_request ci run, so ci was dispatched manually for head 5023477: https://github.com/coder/coder/actions/runs/36784150269 (it will not appear in this PR's checks rollup).

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5023477424

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread aibridge/extract/responses/response.go Resolved
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

CI note: ci was dispatched manually for head 6fd7d19 (no pull_request run started): https://github.com/coder/coder/actions/runs/36785521522

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit: 6fd7d1957c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Add a read-only extraction layer that derives recording facts from AI
provider traffic without taking part in the HTTP request lifecycle. It is
the shared base for the AI Gateway reverse proxy (AIGOV-616) and the
OpenAI Responses WebSocket relay.

- RequestExtractor and ResponseExtraction interfaces. ResponseExtraction
  takes OnEvent(eventType, raw) for streamed events (SSE or WebSocket
  frames), OnBody(status, raw) for complete bodies (non-streaming
  responses and error responses to any request), and Result().
- SSEStream: an io.Writer that splits an SSE byte stream into events
  incrementally, with a 4 MiB per-event bound. The existing
  aibridge.SSEParser is not reused: it buffers every event, is not
  incremental, is capped by bufio.Scanner's 64 KiB line limit, and lives
  in the root aibridge package, which a proxy inside aibridge could not
  import from here without a cycle. Field parsing follows the same rules.
- FromBody: bounded (8 MiB after decoding), gzip-aware body adapter. For
  error statuses a body that cannot be read still reports the status.
- NewRecords/Records.Record: convert facts into the existing recorder
  records with the interceptor field semantics (MsgID is the provider
  response ID, prompts need a response ID, tool calls are not injected).
- Parse failures become bounded parse notes, never traffic errors.
…test

Add aibridge/extract/responses, an extractor for OpenAI Responses API
traffic that mirrors what aibridge/intercept/responses records, reading
raw JSON with gjson so SSE, complete bodies, and WebSocket frames share
one implementation. It lives in a provider subpackage so later Anthropic
Messages and Chat Completions extractors sit beside it under
aibridge/extract. The interceptor code is unchanged.

- Request facts: model, stream flag, last user prompt (string input or
  input_text parts of a trailing user item), function_call_output
  correlation.
- Response facts from the terminal response object (completed,
  incomplete, failed, or a 2xx body): response ID, model, service tier,
  token usage with the interceptor math, function/custom/hosted tool
  calls, reasoning summaries and commentary. Truncated streams fall back
  to response.output_item.done items.
- Terminal errors are intercept.ResponseError values, so
  provider.OpenAI categorizes them unchanged. HTTP 4xx/5xx bodies keep
  their status and categorize exactly as the SDK errors the interceptor
  returns. Mid-stream "error" and "response.failed" events carry no HTTP
  status (unless a WebSocket error event reports one) and categorize as
  unknown, as the nested stream errors the interceptor reports today do.

TestResponsesExtractorRecordingParity replays every Responses fixture
through the interceptor and the extractor and requires identical prompt,
token, tool, and model thought records. Intentional differences, all in
the categorized terminal error:
- streaming/stream_error, streaming/stream_failure: the interceptor
  records success; the extractor reports an unknown provider error.
- blocking/wrong_response_format: the interceptor records an unknown SDK
  decode error; the extractor only adds a parse note.
Other behavior differences with no fixture: a streamed
response.incomplete records usage (as the blocking path already does), a
terminal response without a usage object records no zero-valued token
usage, and an empty prompt is not recorded. Injected-tool fixtures run
without an MCP proxy, since the extractor never injects tools or runs
the inner agentic loop.
… bytes

Apply the reverse-proxy owner's interface feedback:

- ResponseExtraction is constructed with a recorder.Recorder, the
  interception ID, a logger, and the request prompt. OnEvent and
  ProcessBlocking write prompt, token, tool, and model thought records as
  they observe the traffic, with the interceptor's field semantics. The
  prompt is recorded once the response ID is known; usage, tools, and
  thoughts are recorded once, from the terminal response object.
- The fact structs and the facts-to-records helper are gone. Callers read
  only Outcome(): response ID, terminal status and reason, and the
  provider-shaped error for interceptionerror.Categorize.
- ProcessBlocking(status, raw []byte) replaces OnBody. DecodeBody turns a
  wire body ([]byte, gzip-aware, 8 MiB bound) into the bytes it takes;
  bodies stay []byte, with a streaming io.Reader noted as a future option.
- Parse notes are logged (bounded per response) instead of returned.
- SSEStream takes a context and logger and gains Close, which reports a
  partial trailing event. Its doc explains why it does not reuse
  aibridge.SSEParser.
- ResponseExtraction documents that it is not safe for concurrent use and
  must be fed in stream order. A stream without a terminal event no longer
  records tool calls from response.output_item.done, matching the
  interceptor.
- The responses package doc lists the intentional differences from the
  interceptors.

The parity test now compares the records written online against the
interceptor's records for every fixture.
Drop the promptRecorded flag: observeResponseID only reaches the prompt
recording on the first response ID, so the flag was never set when read.
Remove a speculative note from the extract package doc, and move the
duplicated withoutTime test helper into testutil.WithoutCreatedAt.
Move the SSE oversized-event and DecodeBody encoding and size cases to
TestSSEStream and TestDecodeBody in the extract package, where they
fail when the per-event and per-body bounds are removed; through the
Responses extractor they passed either way because its own parse limit
hid the missing bound. Keep one decode-failure case in the extractor
fail-open table.

Add cases for behavior no test caught when mutated: a repeated terminal
event is recorded once, a blocking body sets the terminal status, an
empty prompt is not recorded, a trailing function_call does not set the
correlating tool call ID, and an unterminated SSE event is not
dispatched on Close.

_Generated with [xum](https://github.com/coder/xum) • Model: anthropic:claude-opus-5-5 • Thinking: high_

Signed-off-by: Thomas Kosiewski <tk@coder.com>
Fixes two low-severity defects found in remote UAT:

- An error event or error body after a terminal event no longer changes
  a settled outcome; it is logged as a parse note.
- Parse notes are cut to 256 bytes, and the SSE event name counts toward
  the per-event bound, so traffic-controlled event names cannot grow
  log lines or memory unbounded.

Also restores the package doc note that a streaming io.Reader input is a
possible future direction.

_Generated with [xum](https://github.com/coder/xum) • Model: anthropic:claude-opus-5-5 • Thinking: high_
- A terminal response after an error event is still recorded, but the
  earlier error stays the outcome instead of being overwritten with
  completed or incomplete.
- SSEStream accepts CR, LF, and CRLF line endings as the SSE
  specification requires, including a CRLF split across writes.

_Generated with [xum](https://github.com/coder/xum) • Model: anthropic:claude-opus-5-5 • Thinking: high_
The SSE specification strips one leading byte-order mark. Without it, a
BOM-prefixed first field name was ignored, losing the first event. The
BOM may arrive split across writes.

_Generated with [xum](https://github.com/coder/xum) • Model: anthropic:claude-opus-5-5 • Thinking: high_
- response.failed carries no HTTP status, so its error code now maps to
  one (rate_limit_exceeded 429, server_error 500, other codes 400, no
  code 0), matching the Responses WebSocket relay. Rate limits and server
  failures now categorize as such instead of unknown.
- Add responses.Relevant so callers can skip events the extraction
  ignores, such as output deltas, without duplicating the list.
- Document that recorder calls are not bounded per call.

_Generated with [xum](https://github.com/coder/xum) • Model: anthropic:claude-opus-5-5 • Thinking: high_
4xx and 5xx bodies larger than MaxBodyBytes are no longer parsed; the
HTTP status still classifies the error, with the status text as the
message.

_Generated with [xum](https://github.com/coder/xum) • Model: anthropic:claude-opus-5-5 • Thinking: high_
@ThomasK33
ThomasK33 force-pushed the tk/ai-gateway-extract-responses branch from 6fd7d19 to 5d867ce Compare October 2, 2026 08:16
@github-actions github-actions Bot added the stale This issue is like stale bread. label Nov 9, 2026
@github-actions github-actions Bot closed this Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stale This issue is like stale bread.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant