Skip to content

feat(aibridge/x/proxy/responsesws): record through the response extractor off the read path - #30218

Closed
ThomasK33 wants to merge 4 commits into
tk/ai-gateway-responses-ws-corefrom
tk/ai-gateway-responses-ws-accounting
Closed

ThomasK33 wants to merge 4 commits into
tk/ai-gateway-responses-ws-corefrom
tk/ai-gateway-responses-ws-accounting

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

Moves recording in aibridge/x/proxy/responsesws off the read path and onto the shared aibridge/extract Responses extractor from #30211. It also fixes the three findings deferred from #30198. Resolves AIGOV-697. Nothing imports this package yet; the WebSocket endpoint comes in a later PR and depends on this one.

What changes:

  • The built-in observer (observer.go, Options.Observer) is removed. Each interception gets one responses.NewResponseExtraction, which records prompts, token usage and the terminal outcome. Model, prompt and correlating tool call ID come from responses.RequestExtractor.
  • The reader goroutine never calls the recorder. It binds frames to interceptions, queues accounting jobs for the event types responses.Relevant accepts, and forwards every frame at once. A slow or blocked recorder no longer delays frames or stalls other streams on the connection.
  • One accountant goroutine per session runs the jobs in order: it records the start of interceptions the server opened, feeds each response's extraction, and ends the interception from Outcome(). A terminal event the extractor skipped (oversized or invalid) still ends the interception, with a warning.
  • The queue is bounded at 256 jobs and 16 MiB of frame bytes. On overload the reader drops the accounting job, never the frame, logs accounting queue full: dropped event at error level, and the interception ends with an overload error.
  • An upstream error that arrives while a create's write is still in flight is kept, and becomes that create's end cause once the write succeeds. Before, it was dropped and the interception was recorded as successful.
  • Each interception end record gets its own bounded context after extraction finishes, and each of the extraction's record calls is bounded too. Shutdown drains the queue under one shared cleanup deadline that starts when the reader exits, and Close waits for the accountant.

Tests: synchronized regression tests for a blocked recorder not stalling frames, overload logging, a raced error keeping its cause, unknown frames relayed unchanged, a fresh end-record context, and a bounded Close with queued work. The seeded recording invariant test also runs with a slow recorder and checks that each end is recorded exactly once. go test -race -count=3 ./aibridge/x/proxy/... ./aibridge/extract/... and golangci-lint pass, and each new test passed 500 runs.

Follow-ups before the WebSocket endpoint PR merges, tracked in AIGOV-698: a WebSocket read limit on the upstream connection (the real memory bound), skipping the accounting copy for frames over extract.MaxEventBytes, a byte bound or documented bound for the client frame channel, and a comment fix on which goroutines run extractions.


Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high

@ThomasK33
ThomasK33 added this pull request to stack #30212 September 30, 2026 23:41
@linear-code

linear-code Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

AIGOV-697

AIGOV-698

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 58dcec8c93

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-01T01:10:28.574232Z 34ec977 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d635f55538

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread aibridge/x/proxy/responsesws/session.go Resolved
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 185081a3dc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread aibridge/x/proxy/responsesws/session.go Resolved
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 34ec9773b3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ThomasK33

Copy link
Copy Markdown
Member Author

No pull_request CI run started for the last three pushes, so I dispatched ci manually for head 34ec9773b3: https://github.com/coder/coder/actions/runs/36799892391 (headSha 34ec9773b3fa64d07186f8bc1e73473040f72817). It will not show in this PR's check list.


Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high

…ctor off the read path

Replace the built-in observer with the E1 extractor (extract/responses) and
move all recording driven by server events off the reader, so a slow
recorder no longer holds up forwarded frames.

- The reader binds frames to interceptions and forwards them. It makes no
  recorder calls: it queues the events responses.Relevant accepts for one
  accountant goroutine, which feeds each response's extraction in order
  and ends the interception from the extraction's outcome. Interceptions
  the server opens are registered by the reader and recorded by the
  accountant before their events.
- The accounting queue is bounded by job count and frame bytes. On
  overload the reader drops the accounting job, never the frame, logs the
  drop, and the interception ends with an overload error.
- Session shutdown arms one cleanup deadline when the reader exits. It also
  bounds a record call already in progress; queued jobs past it are
  dropped with a log. Close waits for the accountant.
- An error event that raced a create's write is kept and becomes the
  create's end cause, instead of ending it without an error.
- Each end record gets its own bounded context after extraction finished.
  Extraction record calls are bounded per call.
- Steer prompts and CorrelatingToolCallID come from the request extractor.

Regression tests cover: a blocked recorder not stalling later frames,
logged overload, a raced error keeping its cause, unknown and oversized
frames forwarded unchanged, bounded shutdown with blocked records, a fresh
end record context, and the randomized invariants with a held recorder and
exactly-once end records.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
…dependent

Address review round 1 on the accounting path:

- An interception that lost an accounting job ends with the overload
  error on every end path, including when its terminal event is recorded
  later, instead of reporting success.
- Interceptions the server opens start when their frame arrived: the
  reader captures the start time and model on the interception, and the
  accountant records them later.
- A failed start of such an interception no longer ends it. It keeps its
  response mapping, and the start is retried before each later job, so its
  terminal usage lands on the same interception whatever the timing. If
  the start still fails at the terminal event, the response is dropped
  with a warning naming it and its state is released; session end also
  skips end records for interceptions never started.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
…ue delay and overload

- Records and terminal-driven ends carry the frame's arrival time. The
  reader stamps each job; the extraction's recorder overwrites CreatedAt
  on prompt, token, tool, and thought records with it, and an end driven
  by a terminal event uses it as EndedAt. Shutdown and overload ends keep
  the current time.
- A terminal job dropped by a full queue no longer leaves its
  interception open until the connection ends. The reader forgets the
  interception's bookkeeping at once and hands it to the accountant
  through a list under the session lock plus a wake channel. The
  accountant ends it with the overload error once every job queued
  before the drop ran; session end covers anything left.
- Jobs carry their event type. A terminal frame the extractor skips,
  such as an oversized response.failed, ends by its type: failures with
  an upstream error of unknown category, others without an error. The
  warning stays.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
…ents and own queued frames

- Timestamp contract: every StartedAt, CreatedAt, and EndedAt is the time
  of the event that caused the record, captured where it was observed.
  The reader stamps frames and the session end it reads, Send stamps the
  frames it sends and the failures it sees, and Close stamps a close.
  The accountant never reads the clock for a record.
- Overload hand-offs end at the dropped terminal event's arrival, and
  session end ends every open interception at the observed end time.
  Creates start when Send received them, steer prompts carry the steer's
  send time, and creates retired by a raced event end when that event
  arrived.
- Frames queued for accounting, and raced error events, are copies, so a
  client modifying a frame Recv returned cannot change what is recorded.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
@ThomasK33
ThomasK33 force-pushed the tk/ai-gateway-responses-ws-accounting branch from 34ec977 to dbd473d Compare October 2, 2026 08:16
@github-actions github-actions Bot added the stale This issue is like stale bread. label Nov 9, 2026
@github-actions github-actions Bot closed this Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stale This issue is like stale bread.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant