Skip to content

feat: wire AI Gateway proxy routing and recorder dependencies - #29725

Merged
pawbana merged 6 commits into
mainfrom
pawel/aigov-615-reverse-proxy-handler-with-recorder-lifecycle-and-key-8
Oct 5, 2026
Merged

pawbana merged 6 commits into
mainfrom
pawel/aigov-615-reverse-proxy-handler-with-recorder-lifecycle-and-key-8

Conversation

@pawbana

@pawbana pawbana commented Sep 22, 2026 •

Copy link
Copy Markdown
Collaborator

Adds forwardingHandler for "bridged" routes in proxy mode. Wires provider snapshots to a shared mux and recorder, tracing, circuit breakers, and the server-owned inflight gate.

Construction lifecycle

Server, created by aibridged.New during embedded or standalone gateway startup:

  • Owns the DRPC connection lifecycle and holds the logger, tracer, metrics, and recording options.
  • Creates inflight once in New. The same gate tracks requests across all router snapshots.
  • Added in this PR: Creates recorder once in initializeBackend when proxy mode is selected. All proxy handlers reuse it. Interception mode is unchanged: CachedBridgePool.Acquire creates a recorder on a cache miss, cache hits reuse the bridge and its recorder.

Server holds the Router through backend.proxyRouter:

  • Created by Server.ReplaceProviders through proxy.NewRouter.
    • The provider loader calls ReplaceProviders on the initial provider load and each subsequent reload.

For each enabled provider, NewRouter creates:

  • One passthrough handler, reusing aibridge.NewPassthroughHandler, mounted on the provider’s passthrough routes with provider-prefix stripping and shared inflight middleware.
  • One forwardingHandler mounted on every entry in that provider’s BridgedRoutes(). Each forwardingHandler holds:
    • The provider, logger, tracer, and provider-specific circuit breakers.
    • References the server-owned recorder and inflight gate rather than creating its own.
    • The same instance handles all requests to that provider’s bridged routes.

Router.ServeHTTP limits request-body reads before dispatching through the shared mux.

Request processing

forwardingHandler.ServeHTTP runs these stages:

  1. inflight.Admit and inflight.RequestContext: reject new requests with 503 after shutdown begins and link admitted requests to forced shutdown. The handler also starts a Proxy trace span.
  2. checkRequest: rejects WebSocket upgrades and declared oversized bodies, validates Agent Firewall headers. Actor, recorder, and credential validation and interception record metadata population remain unimplemented.
  3. breaker.Execute: apply the provider's circuit breaker for the matched route, rejecting an open circuit before outbound preparation or forwarding.
  4. prepareForwarding: clone the request. Provider-header preparation and a replayable request body will be added here.
  5. forward: returns the 404 stub for now. This stage will perform forwarding to upstream once lifecycle recording is connected.
  6. finishForwarding: run through a defer after the breaker returns, including when it rejects the request. Currently a no-op, it will finalize the outcome and record interception ended.

Deferred cleanup ends the trace span, cancels the derived request context, and releases inflight admission.

Generated by Coder Agents on behalf of @pawbana.

@linear-code

linear-code Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

AIGOV-615

AIGOV-676

@pawbana pawbana removed their assignment Sep 22, 2026
@pawbana
pawbana added this pull request to stack #29736 September 22, 2026 16:14
@pawbana
pawbana removed this pull request from stack #29736 September 24, 2026 17:36
@pawbana
pawbana force-pushed the pawel/aigov-615-reverse-proxy-handler-with-recorder-lifecycle-and-key-7 branch 11 times, most recently from f070c19 to 116c8ff Compare September 29, 2026 12:50
@pawbana
pawbana force-pushed the pawel/aigov-615-reverse-proxy-handler-with-recorder-lifecycle-and-key-8 branch 7 times, most recently from 1cdb6f2 to a9b1a1a Compare September 29, 2026 21:31
@pawbana
pawbana force-pushed the pawel/aigov-615-reverse-proxy-handler-with-recorder-lifecycle-and-key-7 branch from 9c5c959 to 98ccbfd Compare September 29, 2026 21:31
@pawbana
pawbana force-pushed the pawel/aigov-615-reverse-proxy-handler-with-recorder-lifecycle-and-key-8 branch 2 times, most recently from b62dd7e to 09ec334 Compare September 29, 2026 22:46
@pawbana
pawbana force-pushed the pawel/aigov-615-reverse-proxy-handler-with-recorder-lifecycle-and-key-7 branch from 98ccbfd to 7dc2cf5 Compare September 30, 2026 09:18
@pawbana
pawbana force-pushed the pawel/aigov-615-reverse-proxy-handler-with-recorder-lifecycle-and-key-8 branch from 09ec334 to 5341ac5 Compare September 30, 2026 09:18
@pawbana
pawbana requested review from SasSwart, mtojek and ssncferreira and removed request for johnstcn October 2, 2026 11:43
Comment thread aibridge/context/context.go Resolved
@pawbana
pawbana changed the base branch from pawel/issue-29726-actor to pawel/issue-29726-passthrough-handler October 2, 2026 13:00
@pawbana
pawbana force-pushed the pawel/aigov-615-reverse-proxy-handler-with-recorder-lifecycle-and-key-8 branch from e8d590a to afb06f1 Compare October 2, 2026 20:38
@pawbana
pawbana force-pushed the pawel/issue-29726-passthrough-handler branch from 5521e7b to 208c1c7 Compare October 2, 2026 20:38
@pawbana
pawbana force-pushed the pawel/aigov-615-reverse-proxy-handler-with-recorder-lifecycle-and-key-8 branch 3 times, most recently from ae07413 to 0d07390 Compare October 5, 2026 10:55
Comment thread coderd/aibridged/aibridged.go Outdated
cancelFn context.CancelCauseFunc

// poolOptions configures the interception pool created at startup.
// poolOptions configures the interception pool and recording in both modes.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "both" sounds vague, which modes exactly?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, updated comment.

_ = pool.Shutdown(context.Background())
return s.Err()
}
s.backend.Store(&backend{pool: pool, keyPools: pool.KeyPools})

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

require.NotNil(t, rec, "proxy mode initializes the recorder before providers load")

providers := []aibridge.Provider{aibridge.NewOpenAIProvider(config.OpenAI{})}
for range 2 {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: worth adding a comment on why 2?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest it is not necessarily needed. Pass 1 checks nil to router, and pass 2 checks router to a new router. It is more defensive check that even router is definitely changed:

require.NotSame(t, previous, server.backend.Load().proxyRouter)

regardless of what was store there before but recorder stays the same.

Comment thread aibridge/x/proxy/router_test.go Outdated
Passthrough: []string{"/v1/models"},
URL: upstream.URL,
Bridged: []string{"/v1/chat/completions", "/v1/models/bridged"},
Passthrough: []string{"/v1/models", "/v1/models/", "/v1/tokens"},

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: do we need both /v1/models and /v1/models/ (with slash)?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/models/ covers whole /models/* sub tree while/models cover exact /models path.
Changes path names to better reflect their meaning, made test cases even more extensive.

Comment thread aibridge/x/proxy/handler.go Outdated

func (*forwardingHandler) forward(w http.ResponseWriter, r *http.Request) error {
// TODO: forward only when lifecycle recording is connected.
http.NotFound(w, r)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: return 501 implemented?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

}

func (*forwardingHandler) finishForwarding(context.Context, *recorder.InterceptionRecord, error) {
// TODO: finalize the outcome and lifecycle records, including breaker rejection.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: return 501 implemented?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is just cleanup doesn't return anything.

return nil, err
}

snapshot := slices.Clone(providers)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do need to clone all providers not just ones we need?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All providers are needed. Router holds all providers, mounts newForwardingHandler per provider.

Clone in general is a bit unnecessary, it is shallow copy so it only protects from slice manipulation, which never happens, it doesn't protect from modifying provider instances directly that is in slice, which never happens even more.
In theory if some code would change KeyPool for the same provider instance hold in that slice then it would change the Router.KeyPools() return value but this is very unlikely scenario.
The providers argument slice is created by initialization / reloader and consumed only here in aibridged.Server.

}

bridged := newForwardingHandler(prov, logger, m, tracer, inflight, rec)
for _, path := range prov.BridgedRoutes() {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: would it be useful debug log all routes? (same for passthrough)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added debug log.

@pawbana
pawbana force-pushed the pawel/issue-29726-passthrough-handler branch 2 times, most recently from 1a3d015 to df88311 Compare October 5, 2026 14:23
Base automatically changed from pawel/issue-29726-passthrough-handler to main October 5, 2026 14:54
@pawbana
pawbana enabled auto-merge (squash) October 5, 2026 16:22
@pawbana
pawbana force-pushed the pawel/aigov-615-reverse-proxy-handler-with-recorder-lifecycle-and-key-8 branch 2 times, most recently from 89bd18b to 74bc42d Compare October 5, 2026 17:12
@pawbana
pawbana force-pushed the pawel/aigov-615-reverse-proxy-handler-with-recorder-lifecycle-and-key-8 branch from 74bc42d to 60b2e22 Compare October 5, 2026 17:13
@pawbana
pawbana merged commit 7cfc38d into main Oct 5, 2026
48 of 51 checks passed
@pawbana
pawbana deleted the pawel/aigov-615-reverse-proxy-handler-with-recorder-lifecycle-and-key-8 branch October 5, 2026 17:33
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 5, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants