Skip to content

feat: add Responses WebSocket admission, socket ownership and session bounds - #30245

Closed
ThomasK33 wants to merge 5 commits into
tk/ai-gateway-responses-ws-accountingfrom
tk/ai-gateway-responses-ws-plumbing
Closed

ThomasK33 wants to merge 5 commits into
tk/ai-gateway-responses-ws-accountingfrom
tk/ai-gateway-responses-ws-plumbing

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

Adds the server-side contracts the AI Gateway Responses WebSocket endpoint (next layer) will use. Nothing becomes reachable: WebSocket upgrades to the gateway still return 501.

Part of AIGOV-698 and AIGOV-410. Stacked on #30218.

Changes

  1. Per-create admission. Every authorized gateway request carries an aibridge.CreateAdmissionFunc (context helpers in aibridge/context; responsesws.AdmitFunc is now an alias). Each response.create counts one request against the same rate limiter instance and key as HTTP (httpmw.RateLimitByAuthToken now exposes a RateLimitConsumer on the request context; HTTP responses are unchanged), then re-checks the AI budget over DRPC. Refusals are *intercept.ResponseErrors, so the client gets an error event and keeps its socket: 429 rate_limit_exceeded, 403 with the HTTP budget message (type and code insufficient_quota), and 500 with the ErrBudgetCheck message when the check fails (fail closed).
  2. Socket ownership. aibridged.Server owns a SocketRegistry (Server.Sockets()): Acquire(ctx, actorID, provider) returns a lease whose context keeps the request's values but not its cancellation, so per-user bridge eviction (pool TTL, provider reloads) does not end sockets. Caps are constants (16 per actor, 512 per replica) and count sockets from Acquire, so connecting sockets count. A lease ends after 60 minutes (OpenAI's own WebSocket limit) with ErrSocketLifetime; Server.Shutdown first refuses new leases, cancels all with ErrShutdown, and waits up to its deadline for releases, and only then cancels its lifecycle, so closing sockets can still record through the coderd connection. Detached sockets do not re-authorize: they are bounded by the lifetime, and every create runs admission. New metrics responses_websockets_open and responses_websockets_refusals_total.
  3. Early release of the MaxConcurrency slot. httpmw.ConcurrencyLimit holds a slot for the handler's lifetime, which for a socket would be up to 60 minutes. Admitted requests now carry an agplaibridge.ConcurrencySlotRelease (released once; later calls and the release on return do nothing). The next layer calls it after SocketRegistry.Acquire succeeds, so the handshake, authorization and budget check stay bounded by MaxConcurrency, and only the open socket moves to the socket caps. No request skips the limit in this PR.
  4. responsesws session bounds. MaxClientFrameBytes = routing.MaxRequestBodyBytes and MaxUpstreamFrameBytes = 16 MiB. Frames read ahead of Recv are bounded by 8 MiB as well as 16 frames, with backpressure on the upstream reader. Accounting jobs for frames over extract.MaxEventBytes are queued without the frame bytes, so they cannot trigger the queue overload error. The package doc states the worst-case memory, and the accountLoop comment now explains the publish extraction.

Why detaching the lease context is enough

RequestBridge.Shutdown only cancels its in-flight request contexts and shuts down its MCP proxies. The recorder, providers, key pools and logger it holds stay usable after shutdown, so a session that keeps them only needs a context that bridge shutdown does not cancel. No refcount on the bridge is needed.

Validation

  • go test -race on the touched packages, golangci-lint, make fmt, make gen, and the full pre-push hook on push.
  • Remote dogfood UAT through Coder Agents on a dev deployment: HTTP Responses still 200, rate limited (429) and budget checked (403) as before; WebSocket upgrades to /api/v2/ai-gateway/openai/v1/responses still 501 with the ai-gateway-responses-websocket experiment on and off. An earlier path-based exemption was replaced by the early release above after UAT and review findings.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high

@linear-code

linear-code Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

AIGOV-698

AIGOV-410

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Docs preview

Check off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here.

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-01T12:17:22.150750Z d6c39cc Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d79281d1be

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread enterprise/coderd/aibridge.go Resolved
Comment thread coderd/aibridged/aibridged.go Resolved
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: d6c39cc353

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

… bounds

Adds the server-side contracts the Responses WebSocket endpoint will use.
Nothing becomes reachable: WebSocket upgrades still get 501.

- Per-create admission: every authorized gateway request carries an
  aibridge.CreateAdmissionFunc. Each response.create counts against the
  same rate limiter and key as the HTTP request (exposed by
  httpmw.RateLimitByAuthToken as a context value) and rechecks the AI
  budget. Refusals are OpenAI-shaped errors: 429 rate_limit_exceeded,
  403 for an exceeded budget, 500 when the check fails.
- Socket ownership: aibridged.Server owns a SocketRegistry with per-actor
  (16) and per-replica (512) caps that count connecting sockets, a 60
  minute maximum lifetime, and shutdown that ends every lease. Lease
  contexts are detached from the per-user bridge, so bridge eviction does
  not end sockets. Adds open and refusal metrics.
- MaxConcurrency no longer applies to WebSocket upgrades to /v1/responses,
  since a socket would hold a slot for its lifetime; the socket caps bound
  them instead.
- responsesws: frame size limits, a byte bound with backpressure on frames
  read ahead of Recv, and accounting of frames over extract.MaxEventBytes
  without copying them.
…Concurrency

The exemption cleaned the path before matching, so an upgrade to
/v1/responses/ or /v1/responses/{id}, a passthrough route the reverse
proxy tunnels upstream, skipped the concurrency limit for the tunnel's
lifetime. Match the clean escaped path exactly instead.
Signed-off-by: Thomas Kosiewski <tk@coder.com>
…ockets first

Upgrades to /v1/responses skipped MaxConcurrency before authentication,
so upgrades with fake tokens could pile up handlers waiting on coderd.
Drop the path exemption: every request holds its slot as before, and
ConcurrencyLimit now carries an aibridge.ConcurrencySlotRelease that a
Responses WebSocket handler calls once its socket holds a lease, so only
the open socket leaves the limit and the socket caps bound it.

Server.Shutdown now ends the sockets before it cancels the lifecycle,
which closes the coderd connection that a closing socket's last records
need.
@ThomasK33
ThomasK33 force-pushed the tk/ai-gateway-responses-ws-plumbing branch from d6c39cc to 7b846eb Compare October 2, 2026 08:16
@github-actions github-actions Bot added the stale This issue is like stale bread. label Nov 10, 2026
@github-actions github-actions Bot closed this Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stale This issue is like stale bread.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant