Repository navigation
Conversation
|
Coder Agents is running the automated review loop on behalf of @pawbana. This PR will remain draft and unassigned; no human reviewers will be requested. |
|
/coder-agents-review |
|
Chat: Review in progress (28/28 reviewers complete) | View chat deep-review v0.9.0 | Round 4 | Status: Panel review (25 reviewers) Last posted: Round 3, 73 findings (2 P1, 19 P2, 41 P3, 6 Nit, 5 Note), REQUEST_CHANGES. Review Finding inventoryFinding inventory, PR #29656Law analysis
Findings
Contested and acknowledgedCRF-9 (P3, aibridge/x/proxy/observed_body.go:7) - onRead hook dead in production
CRF-17 (Note, aibridge/x/proxy/handler.go:210) - X-Forwarded-For and User-Agent dropped
CRF-24 (P3, aibridge/client.go:12) - clientmeta re-export shim
Law round-1 mandatory split (not a CRF) - unresolved
Round logRound 3Churn guard: PROCEED. 11 of 13 round-2 findings addressed in Cross-check decisions, round 3
Round 2Churn guard: PROCEED. 16 of 18 findings addressed in Round 1Netero plus Law (effective additions 3370 > 1000). Law verdict: Split, Mandatory, so no panel ran. Netero: 5 P2, 8 P3, 2 Nit, 3 Note, no P0 or P1. Orchestrator verified CRF-1 (h.logger referenced once, at handler.go:136), CRF-3 (every proxy.NewRouter call site in tests passes nil for metrics), CRF-4 (maxRecordedErrorMessageBytes defined in both aibridge/interception_error.go:15 and aibridge/x/proxy/handler.go:34), CRF-5 (old aibridge/client.go carried the docs-sync and Mux-rename comments; clientmeta/client.go carries neither), CRF-8 (only self-references for the aibridge wrapper), and Law's C3b/C4/C5 independence by reading the diffs. Reviewed against 29ae50d..1a30fde. About deep-reviewCRF = Coder Review Finding (P0-P4, Nit, Note)
|
There was a problem hiding this comment.
The reverse-proxy handler is well built for the shape of the problem. record bool cleanly separates bridged from passthrough behavior, observedBody gives the recorder a real EOF and read-error signal instead of guessing at stream termination, the body-limit path distinguishes a declared oversize Content-Length from a mid-read overflow, and the tests assert observable outcomes (upstream headers received, dump files written, recorder start/end IDs paired, panic propagation) rather than mock returns. TestHandlerBodyLimitBoundaries genuinely allocates MaxRequestBodyBytes and checks the upstream got exactly that many bytes. 68.9% test density on 1049 production lines is not padding.
This round ran the two infrastructure reviewers only. Law returned a mandatory split verdict, so the review panel has not looked at this PR yet and will run once the split lands.
Split request (Law, mandatory). Effective diff is +3370 across 46 files, and four of the nine concerns inside it are independently mergeable. Law verified three of them by revert-and-test rather than by argument: the helper extraction builds standalone at its own commit 09d2561904, reverting aibridge/keypool/failover.go breaks no proxy test, and reverting coderd/aibridged/transport.go fails exactly its own two dedicated tests. Proposed landing order:
refactor(aibridge): hoist mode-independent routing helpers intoproviderandrouting. This is commit09d2561904unchanged, +246 -243.refactor(aibridge/clientmeta): extract client and session identification. Not purely mechanical,GuessSessionIDwas restructured and rationale comments were dropped (CRF-5, CRF-6, CRF-7, CRF-8).fix(aibridge/intercept): strip all Coder headers from upstream requests.PrepareClientHeadersnow drops everyCoder-*andX-Coder-*header instead of the two Agent Firewall headers, on the interception path that serves production today. Its entire dedicated review surface here is twoassert.Emptylines.fix(aibridge/keypool): replay failover bodies viaGetBody. Alters retry behavior in interception mode.fix(coderd/aibridged): scope the in-process served request to its own context. Changes cancellation and stream-termination semantics for every chatd LLM call today.feat(aibridge/x/proxy): the forwarding handler itself, roughly +560 production.
Law's reason for making this mandatory rather than advisory is not the line count: "A reviewer approving 'add reverse-proxy handler' is being asked to simultaneously approve a change to what headers leave the deployment toward Anthropic and OpenAI, a change to how failed requests are retried, and a change to when chatd's in-process LLM calls are canceled. Those are three different risk domains with three different blast radii, and the one with the largest blast radius has the smallest diff." The compliance cost is low: item 1 is already its own commit, and items 4 and 5 each touch one production file plus one test file with no reference into the feature.
Netero, 18 findings, no P0 or P1: 5 P2, 8 P3, 2 Nit, 3 Note. Build, vet, golangci-lint, and -race tests on the changed trees are all clean.
Two of the P2s compound. CRF-1 and CRF-2 together mean h.logger is referenced exactly once in the whole handler, to build a config, so every rejection (400 no actor, 501 websocket, 400 bad firewall headers, 413, 500 record failure, 502 upstream) is silent, and on passthrough routes the upstream error is not recorded either because finalize returns before touching it. The trace does not name the upstream either, since rewriteRequest dropped the PassthroughUpstreamURL span attribute that aibridge/passthrough.go:94 sets. A 502 in proxy mode is currently undiagnosable from logs, records, or traces at once. The handler this replaces logged both cases.
CRF-3 is the one I would fix first alongside those, because it is the cheapest: no test anywhere constructs the handler with a non-nil *metrics.Metrics, so inflight Inc/Dec balance is unverified and a future label change panics inside the request handler with no CI signal. One test passing metrics.NewMetrics(prometheus.NewRegistry()) closes it.
CRF-5 through CRF-7 and CRF-13 are all the same class: the extraction moved code and left its constraints behind. The docs-sync link, the Mux-rename note, the fail-closed rationale, and the VARCHAR(15) width now sit somewhere other than the definition that has to satisfy them. These are cheap to restore and expensive to rediscover.
Netero on the dead export, verbatim: "Dead export on arrival; delete it."
CRF-17 is a question, not a defect claim. The rewrite drops X-Forwarded-For forwarding and no longer sets the aibridge User-Agent, both of which aibridge/passthrough.go did deliberately. The tests suggest this is intentional, but it is worth confirming against whatever the providers key on.
🤖 This review was automatically generated with Coder Agents.
|
Coder Agents follow-up on behalf of @pawbana: the logging, metrics, rejection coverage, moved comments, constructor validation, and shared error-classification findings are being addressed. The extractor onRead seam, empty model facts, and header/User-Agent behavior are intentional requirements of AIGOV-615 and its approved plan, not final protocol parity. The suggested split includes prerequisite #29573, which already merged at 44eb71f before this review. This branch carries the pre-merge version, causing overlap/conflicts; I am reconciling the merged base without rewriting published history. Creating additional PRs requires author approval and is awaiting that decision. This PR remains draft and unassigned. |
Restore proxy diagnostics and metadata rationale, share terminal error classification, validate recorder construction, and cover metrics and request rejection paths. Generated by Coder Agents on behalf of @pawbana.
Reconcile the already-merged AIGOV-674 prerequisite while preserving the reverse-proxy implementation. Generated by Coder Agents on behalf of @pawbana.
|
Coder Agents follow-up on behalf of @pawbana: pushed review fixes in f7cdbcd and merged the already-landed prerequisite/main in 96b98e7 without force-pushing. CRF-1-8, 10-16, and the actionable classification coverage in CRF-18 are addressed. Logging/tracing, metrics/preflight tests, recorder construction validation, shared error categorization, moved rationale comments, and constant documentation are updated. CRF-9 (explicit extractor seam) and CRF-17 (approved header/User-Agent contract) remain intentionally unchanged as explained in-thread. Validation passed: make fmt; make lint/go; go test ./aibridge/... ./coderd/aibridged/... ./coderd/x/chatd/...; affected-package race tests; TestAIBridgeTransportFactory; enterprise TestIntegration; git diff --check. #29573 is now part of the merge base, reducing the review to the remaining implementation rather than reviewing the prerequisite twice. Additional PR creation remains pending author authorization. Please re-evaluate the current diff; keep this PR draft and unassigned. |
|
/coder-agents-review |
There was a problem hiding this comment.
Strong turnaround. 16 of the 18 round-1 findings are fixed in f7cdbcdada, and the fixes are real rather than cosmetic: every rejection path now logs, ErrorHandler logs and sets a span status, rewriteRequest restores the PassthroughUpstreamURL attribute, NewRouter rejects a nil recorder instead of deferring a 500 to request time, the duplicated classifier became aibridge/interceptionerror with a Categorizer interface that restores the provider delegation, and the deleted rationale comments in clientmeta are back with the per-client cases spelled out again. aibridge/x/proxy is at 91.6% statement coverage and TestHandlerRecordsSharedErrorClassifications now pins timeout and rate_limited rather than merely asserting non-empty. CRF-17 is closed on the author's AIGOV-615 answer, and CRF-9 is closed: Netero re-examined the seam and declined to re-raise the dead-code claim, since a nil function pointer and one nil check per Read with a named consumer ticket is a real cost of nearly zero.
This was the second pre-panel round. Netero found a new P2, which gates panel selection again, so the panel still has not reviewed this PR. Round 3 hits the consecutive-first-pass cap and will run the full panel regardless of what Netero finds.
13 new findings: 2 P2, 7 P3, 2 Nit, 2 Note. Build, vet, golangci-lint at the pinned version, make lint/architecture, and -race -count=3 on ./aibridge/x/proxy/ are all clean.
I upgraded one finding. Netero rated the route metric label P3; I am posting it as P2. On subtree passthrough routes the label is the raw client URL path, and Copilot's PassthroughRoutes() returns /, so every distinct path a client invents becomes a permanent Prometheus series in the coderd registry. Netero verified it empirically (50 paths, 50 passthrough_total series). That is client-driven unbounded memory growth with no eviction, reachable by any authenticated gateway user, which is an availability property rather than a hygiene one. The identical class exists at aibridge/passthrough.go:70 today; that does not make it acceptable in the mode meant to replace it, and a fix should cover both.
The other P2 is the same class this PR keeps hitting: stripProxyHeaders re-implements intercept.IsActorHeader with the literal "x-ai-bridge-actor", and the proxyHeaders set with inline string comparisons. Three copies of the actor prefix now exist (the intercept constant, the proxy handler, and a test regex), and if the prefix is ever renamed the proxy silently forwards a client-supplied X-AI-Bridge-Actor-Id upstream. go list -deps ./aibridge/intercept shows no cycle, so intercept.IsActorHeader(name) is available today.
Four of the P3s are duplication that survived the round-1 cleanup: the transport tuning block copied from passthrough.go, the circuit-breaker onChange closure copied verbatim from bridge.go, the clientmeta re-export shim in aibridge/client.go whose only consumers are tests, and GuessSessionID hardcoding ClientClaudeCode as the sole payload-consuming client while the switch below owns that fact. The last one has a consequence beyond tidiness: when a second client gains body-based extraction, the proxy path finds the session ID and the interception path returns nil, with no failing test.
Two round-1 fixes are incomplete rather than wrong. CRF-13's VARCHAR(15) comment moved to recorder/types.go where it annotates only CredentialHintFailoverKey, leaving hintAWSChainKey as the one hint literal in the tree with no length constraint documented anywhere near it. And the class CRF-5 through CRF-7 addressed in clientmeta was not applied to interceptionerror, which dropped all four rationale comments it moved, including the one explaining why key-pool errors must be checked before provider delegation.
Netero on the seam contract, verbatim: "Buffering a response stream is exactly what a response extractor does." onRead receives p[:n] where p is the caller's buffer, and httputil.ReverseProxy reuses one 32 KiB buffer for the whole copy loop. The one existing consumer happens to copy via append; nothing says it must.
Process note on the split. Law's round-1 verdict was mandatory, and it is still unresolved. Item 1's prerequisite #29573 merged to main independently, which removes it from this diff, but the author's response states that creating the remaining PRs "requires author approval and is awaiting that decision," with no linked ticket. Neither I nor the author-agent can accept that bundling as permanent; this needs a human decision. The PR still asks one approval to cover three behavior changes on the interception path that serves production today (the Coder-* header strip in intercept.PrepareClientHeaders, the GetBody replay factory in keypool, and the served-context change in inMemoryRoundTripper) alongside an experiment-gated feature. Worth recording: Netero reviewed all three on their own terms this round and found no defect in the keypool factory or the served-context change, so the split argument is about review surface and revert granularity, not about latent bugs found.
🤖 This review was automatically generated with Coder Agents.
Use registered passthrough routes for metrics, share header and streaming transport helpers, preserve recorder and session contracts, and update the standalone proxy-mode expectation. Generated by Coder Agents on behalf of @pawbana.
|
Coder Agents follow-up on behalf of @pawbana: f4310a5 addresses CRF-19 through CRF-31 except CRF-24, whose pre-existing public compatibility aliases are retained as explained in-thread. Route/method metric labels are bounded in both forwarding modes; legacy routes still share one provider transport. Actor-header predicates, streaming transport construction, and breaker observability are shared without changing the deliberately deferred breaker accounting. End recording uses a direct detached bounded call with error logging. Borrowed-buffer and persistence constraints are documented; session extraction has one payload-needed decision; the review-named tests are merged into handler_test.go. Also fixed the CI-only standalone test that still expected the pre-handler 404 instead of forwarding/recording success. Full make pre-commit passed, along with gateway/chatd/consumer integration tests, affected race tests, and the standalone experiment test in normal and race modes. Additional PR creation still requires author authorization; this PR remains draft and unassigned. |
|
/coder-agents-review |
There was a problem hiding this comment.
First panel round, and the code it met is materially better than the code round 1 saw. 29 of the 31 earlier findings are fixed, and the round-3 commit did the harder version of most of them: interceptionerror is a real shared package with a Categorizer interface rather than a copied function, NewProviderCircuitBreakersWithObservability collapsed two duplicated closures into one, sessionIDFromInputs replaced a hardcoded client check with a switch that owns the decision, and utils.NewStreamingTransport carries the rationale comment that was previously lost. The metric-cardinality work is genuinely done where it was aimed: Hisoka went looking to break the bounded passthrough label and could not, and both modes now have a 50-distinct-path test. Several reviewers independently named the same test as the best thing in the diff, TestProxyRecorderUsesAuthenticatedAPIKeyIDPerRequest, which runs three identities concurrently and asserts the negative cases. Kurapika traced the whole recorder identity chain and reported it correct, which is the place cross-request attribution bugs usually live.
42 new findings from a 25-reviewer panel: 2 P1, 13 P2, 25 P3, 2 Nit. Nine reviewers settled claims with throwaway probes rather than reasoning, and every one cleaned up after itself. CRF-24 is closed: six reviewers evaluated your compatibility defense independently and all six declined to re-raise, 6/6 accept.
The two P1s are both about proxy mode being a governance gateway that does not govern yet, without saying so.
CRF-32. There is no RecordTokenUsage, RecordPromptUsage, or RecordToolUsage call anywhere in aibridge/x/proxy, and spend only accrues inside RecordTokenUsage on the coderd side. Server.ServeHTTP still calls IsBudgetExceeded on every request, against a total that can never grow. So a user with a configured monthly cap can spend without bound on the org's centralized keys while the control plane shows the budget check running and passing. I accept that extractors are staged behind AIGOV-615 and AIGOV-616. What is not a sequencing decision is that this PR replaced Warn "reverse proxy routing is not yet functional" with Info "selected reverse proxy routing", which removed the only operator-visible statement that the mode is incomplete. Five reviewers reached this independently from five different angles. Whether to ship a mode that silently disables a documented, entitled control is a human decision, not one a reviewer or a follow-up ticket can make.
CRF-33. captureRequestBody runs above every h.record branch, so passthrough routes that stream today are fully buffered before the upstream is dialed. Copilot is always BYOK with PassthroughRoutes() of {"/"} and a nil key pool, so today every Copilot request streams and buffers nothing; under this handler all of them buffer up to 32 MiB. Hisoka probed it with an open io.Pipe body and confirmed the upstream is not contacted until the client's last byte. Killua measured io.ReadAll holding 48 MiB live per 32 MiB body. CODER_AI_GATEWAY_MAX_CONCURRENCY and CODER_AI_GATEWAY_RATE_LIMIT both default to 0, and the in-memory chatd transport does not pass through that middleware at all, so nothing bounds how many buffers exist at once. Ten reviewers rated this P2, citing the experiment gate and the per-request cap; I am posting P1 because experiment gating is a probability argument and a per-request cap bounds one request, not the process. The fix is four lines and the passthrough path reads none of the bytes.
Those two are not the whole of it, and one compound effect deserves saying because no single reviewer said it. CRF-34 and CRF-33 interact. The breaker's Execute closure always returns nil, ErrorHandler writes 502, and DefaultIsFailure counts only 503 and 504, so the circuit cannot trip on a dial, DNS, or TLS failure. Netero and Hisoka each proved it separately against http://127.0.0.1:1 with a threshold of one: three requests, three 502s, no state change, while the same setup with an upstream 503 opens the circuit on the first failure. So during a hard provider outage proxy mode keeps dispatching at full rate, every request pays the whole dial timeout, and each of those parked requests is holding its buffered body. The one scenario the breaker exists for becomes a coderd memory event instead of shed load. Bisky then found why nothing caught it: the only breaker test asserts a 503 status that an upstream 503 and a breaker-generated 503 both satisfy, and it stays green with the breaker wiring deleted from router.go entirely.
Four findings land on the production interception path rather than behind the experiment, so they need fixing whether or not this PR is split. CRF-35: InterceptionCount still passes raw r.Method, which eight reviewers flagged and two reproduced end to end, minting a permanent series per method token. CRF-37: Knuckle ran inserts against a real migrated schema and found client_session_id is VARCHAR(256) with no clamp on either side, so a client sending a 300-character session ID gets a 500 on every request until it changes, while the neighbouring error_message is truncated. CRF-48 and CRF-49: the PR taught two of three upstream paths to strip Coder headers and left interception-mode passthrough forwarding them, and PrepareClientHeaders strips Coder-* but not the actor headers the proxy path does strip.
Kurapika's CRF-36 is the one I would look at next after the P1s. Encoded dot segments survive rewriteRequest untouched: a request for /copilot/%2e%2e/%2e%2e/admin/keys reached the upstream as /tenant-a/%2e%2e/%2e%2e/admin/keys, because net/http.cleanPath runs on the escaped path so %2e%2e never becomes .. and the subtree pattern matches. On Anthropic and OpenAI the gateway injects the org's pooled key into that request, so the user picks the upstream path and Coder supplies the credential. Kurapika was explicit that the second half is unverified: whether the request lands outside the allow-listed path depends on the upstream normalizing %2e%2e, which edge proxies commonly do but which was not tested against a real provider. The gateway-side break is the half Coder controls and it is confirmed. What would settle the rest is one request to each configured provider with an encoded segment.
Ryosuke's structural read is worth more than any individual P3 and I am relaying it whole rather than as a comment: the record bool is read five times and each read marks a place where the two modes want different code. The unconditional buffer exists because one entry point serves both. The dead metricRoute on bridged handlers and the dead route on passthrough handlers exist because one struct carries both modes' fields. Round 3 extracted the leaves and left the sequencing duplicated in two files, which is exactly why bounded labels landed on one path and not the other. His suggestion is that the next extraction be the recording lifecycle as one component both bridged handlers call, not another helper in utils.
Verbatim, Hisoka opening: "Four probes thrown, four landed. The proxy handler is well-built where it's been looked at. Nobody looked at what it does to a request body, and nobody asked the circuit breaker what it counts."
Two process items. The PR description is still the single word "WIP" on a 45-file, +3866/-992 change, and Leorio and Mafu-san independently noted that the broadest production behavior change in it, widening PrepareClientHeaders to drop every Coder-* header, is described nowhere a reader would look: not the PR body, not the commit that introduced it, not the commit that touched it again. Second, Law's mandatory split from round 1 has now gone two rounds with no linked ticket and no round-3 response. The author-agent's position is that creating the PRs awaits author authorization; that is a reasonable thing for an agent to say and it is also why this needs a human. Three behavior changes on the path serving production today are still bundled with an experiment-gated feature, and four of this round's findings land inside them. Either authorize the split or explicitly accept the bundling; neither I nor the author-agent can accept it as permanent.
aibridge/bridge.go:271
P2 [CRF-35] The CRF-20 fix bounded the method label on the new path and left the same metric unbounded on the interception path that serves production today. (Mafuuu P2, Meruem P2, Melody P2, Ryosuke P2, Mafu-san P2, Robin P2, Zoro P2, Razor P2)
This PR introduced
routing.MetricMethodand applied it atpassthrough.go:65,106andx/proxy/handler.go:145,293, butInterceptionCounton the bridged interception path still passesr.Methodverbatim atbridge.go:271(failed) andbridge.go:277(completed). One unbounded writer poisons the series set for every reader of that metric, so bounding one of the two producers changes nothing. (Ryosuke)
Verified, not reasoned: a request with method
FROBNICATE1234567890to/anthropic/v1/messagesreturned 200 and producedinterceptions_total{...,method="FROBNICATE1234567890",status="completed"} 1. Every distinct token an authenticated user sends creates a new series withinitiator_idalready in the label set, so one request per series is the cost of growing coderd's registry without bound. (Meruem)
In coderd the handler is mounted under chi, which rejects non-standard method tokens with 405. The standalone gateway mounts it on an
http.ServeMux, which matches on path only, so an authenticated client sendingFOO /anthropic/v1/messageswith a valid body reaches the bridged handler. (Melody)
Eight reviewers found this independently and two reproduced it end to end through aibridge/internal/integrationtest. Melody's mount-point analysis narrows the blast radius to the standalone gateway rather than coderd, which is why this is P2 and not higher, but it is the same remote resource-exhaustion vector the panel rated P2 for the route label and it is not behind the experiment flag. Fix: routing.MetricMethod(r.Method) at both call sites.
🤖
coderd/aibridged/aibridged.go:388
P3 [CRF-52] Graceful shutdown kills the DRPC connection before it drains requests, so every interception still in flight is recorded as started and never ended. (Takumi)
Sequence in
Shutdown:s.cancelFn(ErrShutdown),s.wg.Wait(), thens.inflight.Shutdown(ctx)for proxy mode orpool.Shutdown(ctx)for interception mode.s.wgtracks the connect goroutine, which is the only sender ons.clientChand closes the DRPC connection on its way out. So by the time the drain begins, no sender remains andServer.Clienthas only<-s.Done()ready.Verified: with
lifecycleCtxcanceled,s.Client(ctx)returnsErrShutdownin under a millisecond against a 5s call context.Every handler that finishes during the drain therefore fails its end record with
acquire client: ..., logged at warn and discarded. The interception rows for those requests keep a start and never get an end, on every coderd restart that has gateway traffic. Interception mode has the identical exposure throughbridge.go:282.
The ordering fix is local: stop admission and drain first, cancel the lifecycle context after, so the drain still has a client. This interacts with CRF-51: fixing the ordering here also removes the 5s-per-request stall during a restart.
🤖
aibridge/circuitbreaker/circuitbreaker.go:73
P3 [CRF-56] After the CRF-27 extraction, NewProviderCircuitBreakers has zero production callers and exists as an exported hook for tests only. (Knov, Zoro)
Count, not intuition: production callers of
NewProviderCircuitBreakersWithObservabilityarebridge.go:78andx/proxy/router.go:88, two of two. Production callers ofNewProviderCircuitBreakersare zero; the only call sites are the wrapper at line 66 and four incircuitbreaker_test.go.The wide constructor takes an arbitrary
onChange. Every real caller wants exactly one: log the transition and move the metrics. Keeping both exported means the next caller can register breakers with no observability at all, which is the state this refactor was raised to remove, and CI will not notice. This is the shape CRF-27 was about: one behavior, two entry points, drifting independently. (Knov)
The CRF-27 fix is right; this is the residue of it. Unexport the narrow constructor, keep the observable one as the only exported entry point, and rename it back to NewProviderCircuitBreakers since there is no longer an alternative. Gon separately noted the naming consequence: the constructor every production caller must use currently has the longer, mechanism-shaped name.
🤖
🤖 This review was automatically generated with Coder Agents.
| }, | ||
| } | ||
|
|
||
| execErr := h.breaker.Execute(route, "", w, func(rw http.ResponseWriter) error { |
There was a problem hiding this comment.
P2 [CRF-34] The breaker closure always returns nil, so upstream transport failures never count as circuit-breaker failures. (Netero P2, Hisoka P2, both reproduced)
h.breaker.Execute(route, "", w, func(rw) error { ...; proxy.ServeHTTP(writer, r); return nil })swallows every failure. TheErrorHandlerwrites 502, andcircuitbreaker.DefaultIsFailurecounts only 503 and 504, so a dial failure, TLS failure, or DNS failure is invisible to gobreaker. The interception path does the opposite:bridge.go:261-263returnsinterceptor.ProcessRequest's error, which gobreaker counts. (Netero)
Bridged route,
FailureThreshold: 1, base URLhttp://127.0.0.1:1, three requests:502 502 502, no state change logged. Same provider, same threshold, upstream answering 503 instead:circuit breaker state change provider=openai endpoint=/v1/chat from=closed to=open, second request{"error":"circuit breaker is open"}. So the breaker works, and refuses the one case that matters. (Hisoka)
Two reviewers reproduced this separately with the same setup and the same result. Verified independently: the closure returns a bare nil, and DefaultIsFailure accepts only StatusServiceUnavailable and StatusGatewayTimeout.
This compounds with CRF-33. During a hard provider outage the circuit never opens, so every request is dispatched, pays the full dial or TLS timeout, and holds its fully buffered body while it waits. The scenario the breaker exists for becomes a coderd memory event instead of shed load.
Fix is one line: return state.transportErr from the closure. ErrorHandler has already written the 502 so the client response does not change, and terminalError already prefers execErr over transportErr, so the recorded error type does not change either.
🤖
There was a problem hiding this comment.
First panel round, and the code it met is materially better than the code round 1 saw. 29 of the 31 earlier findings are fixed, and the round-3 commit did the harder version of most of them: interceptionerror is a real shared package with a Categorizer interface rather than a copied function, NewProviderCircuitBreakersWithObservability collapsed two duplicated closures into one, sessionIDFromInputs replaced a hardcoded client check with a switch that owns the decision, and utils.NewStreamingTransport carries the rationale comment that was previously lost. The metric-cardinality work is genuinely done where it was aimed: Hisoka went looking to break the bounded passthrough label and could not, and both modes now have a 50-distinct-path test. Several reviewers independently named the same test as the best thing in the diff, TestProxyRecorderUsesAuthenticatedAPIKeyIDPerRequest, which runs three identities concurrently and asserts the negative cases. Kurapika traced the whole recorder identity chain and reported it correct, which is the place cross-request attribution bugs usually live.
42 new findings from a 25-reviewer panel: 2 P1, 13 P2, 25 P3, 2 Nit. Nine reviewers settled claims with throwaway probes rather than reasoning, and every one cleaned up after itself. CRF-24 is closed: six reviewers evaluated your compatibility defense independently and all six declined to re-raise, 6/6 accept.
The two P1s are both about proxy mode being a governance gateway that does not govern yet, without saying so.
CRF-32. There is no RecordTokenUsage, RecordPromptUsage, or RecordToolUsage call anywhere in aibridge/x/proxy, and spend only accrues inside RecordTokenUsage on the coderd side. Server.ServeHTTP still calls IsBudgetExceeded on every request, against a total that can never grow. So a user with a configured monthly cap can spend without bound on the org's centralized keys while the control plane shows the budget check running and passing. I accept that extractors are staged behind AIGOV-615 and AIGOV-616. What is not a sequencing decision is that this PR replaced Warn "reverse proxy routing is not yet functional" with Info "selected reverse proxy routing", which removed the only operator-visible statement that the mode is incomplete. Five reviewers reached this independently from five different angles. Whether to ship a mode that silently disables a documented, entitled control is a human decision, not one a reviewer or a follow-up ticket can make.
CRF-33. captureRequestBody runs above every h.record branch, so passthrough routes that stream today are fully buffered before the upstream is dialed. Copilot is always BYOK with PassthroughRoutes() of {"/"} and a nil key pool, so today every Copilot request streams and buffers nothing; under this handler all of them buffer up to 32 MiB. Hisoka probed it with an open io.Pipe body and confirmed the upstream is not contacted until the client's last byte. Killua measured io.ReadAll holding 48 MiB live per 32 MiB body. CODER_AI_GATEWAY_MAX_CONCURRENCY and CODER_AI_GATEWAY_RATE_LIMIT both default to 0, and the in-memory chatd transport does not pass through that middleware at all, so nothing bounds how many buffers exist at once. Ten reviewers rated this P2, citing the experiment gate and the per-request cap; I am posting P1 because experiment gating is a probability argument and a per-request cap bounds one request, not the process. The fix is four lines and the passthrough path reads none of the bytes.
Those two are not the whole of it, and one compound effect deserves saying because no single reviewer said it. CRF-34 and CRF-33 interact. The breaker's Execute closure always returns nil, ErrorHandler writes 502, and DefaultIsFailure counts only 503 and 504, so the circuit cannot trip on a dial, DNS, or TLS failure. Netero and Hisoka each proved it separately against http://127.0.0.1:1 with a threshold of one: three requests, three 502s, no state change, while the same setup with an upstream 503 opens the circuit on the first failure. So during a hard provider outage proxy mode keeps dispatching at full rate, every request pays the whole dial timeout, and each of those parked requests is holding its buffered body. The one scenario the breaker exists for becomes a coderd memory event instead of shed load. Bisky then found why nothing caught it: the only breaker test asserts a 503 status that an upstream 503 and a breaker-generated 503 both satisfy, and it stays green with the breaker wiring deleted from router.go entirely.
Four findings land on the production interception path rather than behind the experiment, so they need fixing whether or not this PR is split. CRF-35: InterceptionCount still passes raw r.Method, which eight reviewers flagged and two reproduced end to end, minting a permanent series per method token. CRF-37: Knuckle ran inserts against a real migrated schema and found client_session_id is VARCHAR(256) with no clamp on either side, so a client sending a 300-character session ID gets a 500 on every request until it changes, while the neighbouring error_message is truncated. CRF-48 and CRF-49: the PR taught two of three upstream paths to strip Coder headers and left interception-mode passthrough forwarding them, and PrepareClientHeaders strips Coder-* but not the actor headers the proxy path does strip.
Kurapika's CRF-36 is the one I would look at next after the P1s. Encoded dot segments survive rewriteRequest untouched: a request for /copilot/%2e%2e/%2e%2e/admin/keys reached the upstream as /tenant-a/%2e%2e/%2e%2e/admin/keys, because net/http.cleanPath runs on the escaped path so %2e%2e never becomes .. and the subtree pattern matches. On Anthropic and OpenAI the gateway injects the org's pooled key into that request, so the user picks the upstream path and Coder supplies the credential. Kurapika was explicit that the second half is unverified: whether the request lands outside the allow-listed path depends on the upstream normalizing %2e%2e, which edge proxies commonly do but which was not tested against a real provider. The gateway-side break is the half Coder controls and it is confirmed. What would settle the rest is one request to each configured provider with an encoded segment.
Ryosuke's structural read is worth more than any individual P3 and I am relaying it whole rather than as a comment: the record bool is read five times and each read marks a place where the two modes want different code. The unconditional buffer exists because one entry point serves both. The dead metricRoute on bridged handlers and the dead route on passthrough handlers exist because one struct carries both modes' fields. Round 3 extracted the leaves and left the sequencing duplicated in two files, which is exactly why bounded labels landed on one path and not the other. His suggestion is that the next extraction be the recording lifecycle as one component both bridged handlers call, not another helper in utils.
Verbatim, Hisoka opening: "Four probes thrown, four landed. The proxy handler is well-built where it's been looked at. Nobody looked at what it does to a request body, and nobody asked the circuit breaker what it counts."
Two process items. The PR description is still the single word "WIP" on a 45-file, +3866/-992 change, and Leorio and Mafu-san independently noted that the broadest production behavior change in it, widening PrepareClientHeaders to drop every Coder-* header, is described nowhere a reader would look: not the PR body, not the commit that introduced it, not the commit that touched it again. Second, Law's mandatory split from round 1 has now gone two rounds with no linked ticket and no round-3 response. The author-agent's position is that creating the PRs awaits author authorization; that is a reasonable thing for an agent to say and it is also why this needs a human. Three behavior changes on the path serving production today are still bundled with an experiment-gated feature, and four of this round's findings land inside them. Either authorize the split or explicitly accept the bundling; neither I nor the author-agent can accept it as permanent.
aibridge/bridge.go:271
P2 [CRF-35] The CRF-20 fix bounded the method label on the new path and left the same metric unbounded on the interception path that serves production today. (Mafuuu P2, Meruem P2, Melody P2, Ryosuke P2, Mafu-san P2, Robin P2, Zoro P2, Razor P2)
This PR introduced
routing.MetricMethodand applied it atpassthrough.go:65,106andx/proxy/handler.go:145,293, butInterceptionCounton the bridged interception path still passesr.Methodverbatim atbridge.go:271(failed) andbridge.go:277(completed). One unbounded writer poisons the series set for every reader of that metric, so bounding one of the two producers changes nothing. (Ryosuke)
Verified, not reasoned: a request with method
FROBNICATE1234567890to/anthropic/v1/messagesreturned 200 and producedinterceptions_total{...,method="FROBNICATE1234567890",status="completed"} 1. Every distinct token an authenticated user sends creates a new series withinitiator_idalready in the label set, so one request per series is the cost of growing coderd's registry without bound. (Meruem)
In coderd the handler is mounted under chi, which rejects non-standard method tokens with 405. The standalone gateway mounts it on an
http.ServeMux, which matches on path only, so an authenticated client sendingFOO /anthropic/v1/messageswith a valid body reaches the bridged handler. (Melody)
Eight reviewers found this independently and two reproduced it end to end through aibridge/internal/integrationtest. Melody's mount-point analysis narrows the blast radius to the standalone gateway rather than coderd, which is why this is P2 and not higher, but it is the same remote resource-exhaustion vector the panel rated P2 for the route label and it is not behind the experiment flag. Fix: routing.MetricMethod(r.Method) at both call sites.
🤖
coderd/aibridged/aibridged.go:388
P3 [CRF-52] Graceful shutdown kills the DRPC connection before it drains requests, so every interception still in flight is recorded as started and never ended. (Takumi)
Sequence in
Shutdown:s.cancelFn(ErrShutdown),s.wg.Wait(), thens.inflight.Shutdown(ctx)for proxy mode orpool.Shutdown(ctx)for interception mode.s.wgtracks the connect goroutine, which is the only sender ons.clientChand closes the DRPC connection on its way out. So by the time the drain begins, no sender remains andServer.Clienthas only<-s.Done()ready.Verified: with
lifecycleCtxcanceled,s.Client(ctx)returnsErrShutdownin under a millisecond against a 5s call context.Every handler that finishes during the drain therefore fails its end record with
acquire client: ..., logged at warn and discarded. The interception rows for those requests keep a start and never get an end, on every coderd restart that has gateway traffic. Interception mode has the identical exposure throughbridge.go:282.
The ordering fix is local: stop admission and drain first, cancel the lifecycle context after, so the drain still has a client. This interacts with CRF-51: fixing the ordering here also removes the 5s-per-request stall during a restart.
🤖
aibridge/circuitbreaker/circuitbreaker.go:73
P3 [CRF-56] After the CRF-27 extraction, NewProviderCircuitBreakers has zero production callers and exists as an exported hook for tests only. (Knov, Zoro)
Count, not intuition: production callers of
NewProviderCircuitBreakersWithObservabilityarebridge.go:78andx/proxy/router.go:88, two of two. Production callers ofNewProviderCircuitBreakersare zero; the only call sites are the wrapper at line 66 and four incircuitbreaker_test.go.The wide constructor takes an arbitrary
onChange. Every real caller wants exactly one: log the transition and move the metrics. Keeping both exported means the next caller can register breakers with no observability at all, which is the state this refactor was raised to remove, and CI will not notice. This is the shape CRF-27 was about: one behavior, two entry points, drifting independently. (Knov)
The CRF-27 fix is right; this is the residue of it. Unexport the narrow constructor, keep the observable one as the only exported entry point, and rename it back to NewProviderCircuitBreakers since there is no longer an alternative. Gon separately noted the naming consequence: the constructor every production caller must use currently has the longer, mechanism-shaped name.
🤖
🤖 This review was automatically generated with Coder Agents.
| }, | ||
| } | ||
|
|
||
| execErr := h.breaker.Execute(route, "", w, func(rw http.ResponseWriter) error { |
There was a problem hiding this comment.
P2 [CRF-34] The breaker closure always returns nil, so upstream transport failures never count as circuit-breaker failures. (Netero P2, Hisoka P2, both reproduced)
h.breaker.Execute(route, "", w, func(rw) error { ...; proxy.ServeHTTP(writer, r); return nil })swallows every failure. TheErrorHandlerwrites 502, andcircuitbreaker.DefaultIsFailurecounts only 503 and 504, so a dial failure, TLS failure, or DNS failure is invisible to gobreaker. The interception path does the opposite:bridge.go:261-263returnsinterceptor.ProcessRequest's error, which gobreaker counts. (Netero)
Bridged route,
FailureThreshold: 1, base URLhttp://127.0.0.1:1, three requests:502 502 502, no state change logged. Same provider, same threshold, upstream answering 503 instead:circuit breaker state change provider=openai endpoint=/v1/chat from=closed to=open, second request{"error":"circuit breaker is open"}. So the breaker works, and refuses the one case that matters. (Hisoka)
Two reviewers reproduced this separately with the same setup and the same result. Verified independently: the closure returns a bare nil, and DefaultIsFailure accepts only StatusServiceUnavailable and StatusGatewayTimeout.
This compounds with CRF-33. During a hard provider outage the circuit never opens, so every request is dispatched, pays the full dial or TLS timeout, and holds its fully buffered body while it waits. The scenario the breaker exists for becomes a coderd memory event instead of shed load.
Fix is one line: return state.transportErr from the closure. ErrorHandler has already written the 502 so the client response does not change, and terminalError already prefers execErr over transportErr, so the recorded error type does not change either.
🤖
|
Coder Agents scope reconciliation on behalf of @pawbana: The demonstrated passthrough buffering regression, cookie/path/upgrade boundaries, method-label cardinality, diagnostics, and shutdown/stream-abort issues are being addressed. Startup warnings and the PR description will explicitly disclose incomplete usage/spend accounting, unknown model facts, unsupported Bedrock, and missing proxy actor injection. Several panel recommendations conflict with the agreed implementation contract and are not being silently converted into new features:
These dispositions do not claim app approval. The mandatory split still needs a human decision; no additional PRs have been created. #29656 remains draft and unassigned. |
Stream passthrough bodies, reject unsafe upstream paths and upgrades, sanitize credential cookies across forwarding modes, preserve recorder access during graceful shutdown, and disclose staged proxy limitations. Generated by Coder Agents on behalf of @pawbana.
|
Coder Agents follow-up on behalf of @pawbana: pushed 71af097 with the bounded panel fixes. Passthrough requests stream again; encoded traversal and unsupported upgrades fail closed; cookies and untrusted identity/proxy headers are sanitized across forwarding modes; method/route labels are bounded; overlong optional session IDs no longer block traffic; logging and exact error tests are improved. Graceful shutdown stops publication/admission and drains requests before closing DRPC; in-process stream aborts no longer masquerade as handler panics. Duplicate app threads for addressed findings are resolved. CRF-32/38/39 are addressed through explicit startup warnings and the expanded PR description, not through unsupported claims of parity. No extractor/spend/model facts or breaker-correctness feature was added. The remaining scoped deferrals in the prior reconciliation comment still stand (CRF-34, 41, 51, 56, 57, 62, 63, 67). Existing public compatibility entry points remain. Validation: full make pre-commit passed; aibridge/aibridged/chatd and consumer integration tests passed; affected-package race tests passed. One broader integration race run failed two breaker-transition assertions; the exact tests passed on rerun, so it is reported rather than claimed proven pre-existing. Please review the current work against these explicit scope decisions. Additional PR creation remains awaiting human authorization. PR remains draft and unassigned. |
|
/coder-agents-review |
|
Coder Agents follow-up on behalf of @pawbana: pushed 71af097 with the bounded panel fixes. Passthrough requests stream again; encoded traversal and unsupported upgrades fail closed; cookies and untrusted identity/proxy headers are sanitized across forwarding modes; method/route labels are bounded; overlong optional session IDs no longer block traffic; logging and exact error tests are improved. Graceful shutdown stops publication/admission and drains requests before closing DRPC; in-process stream aborts no longer masquerade as handler panics. Duplicate app threads for addressed findings are resolved. CRF-32/38/39 are addressed through explicit startup warnings and the expanded PR description, not through unsupported claims of parity. No extractor/spend/model facts or breaker-correctness feature was added. The remaining scoped deferrals in the prior reconciliation comment still stand (CRF-34, 41, 51, 56, 57, 62, 63, 67). Existing public compatibility entry points remain. Validation: full make pre-commit passed; aibridge/aibridged/chatd and consumer integration tests passed; affected-package race tests passed. One broader integration race run failed two breaker-transition assertions; the exact tests passed on rerun, so it is reported rather than claimed proven pre-existing. Please review the current work against these explicit scope decisions. Additional PR creation remains awaiting human authorization. PR remains draft and unassigned. |
|
/coder-agents-review |
There was a problem hiding this comment.
Round 4, against 71af09733f, and the first time the panel has seen this head.
The hardening commit is substantial and mostly right. Request-body capture moved inside if h.record with a MaxBytesError arm, so passthrough streams again and unknown-length oversize bodies get 413. ValidateForwardPath rejects encoded traversal on both forwarding paths, and Kurapika and Razor each probed it: %2e%2e%2fadmin, ..%2f../admin, and ./chat are rejected, a%2Fb survives, and double encoding is not a traversal at the provider. StripSensitiveRequestHeaders is one matcher covering Cookie, actor headers, coder-*, x-coder-*, forwarded, and x-forwarded-*, called from all three upstream paths. Shutdown drains before cancelFn. The breaker test now counts upstream calls and asserts the open-circuit body, so CRF-43's "green with the wiring deleted" property is gone. Netero measured 96.7% coverage on aibridge/x/proxy and Komugi ran the new shutdown and streaming tests at GOMAXPROCS=1 -race -count=30 green. Bisky, who is not generous about tests, called the drain tests honest.
Process note, because it changed what this round did. Law returned Split/Mandatory a second time, which under this review process skips the panel. deep-review post refused that payload: it requires panel reviewers once the panel has reviewed the PR. The tool is right and the process rule was wrong here, because 71af09733f claimed 34 panel findings closed and no panel member had checked any of them. So the panel ran. It found ten P2s, a P1 re-raise with nine reviewers behind it, and five independent reproductions of one defect.
The one that five reviewers reproduced separately
CRF-81: an enabled circuit breaker turns a user pressing Escape into a route-wide outage. requestProxy.ServeHTTP runs inside h.breaker.Execute, httputil.ReverseProxy panics http.ErrAbortHandler when the copy to the client fails, and gobreaker.Execute books any panic as a failure before re-panicking. Hisoka, Mafuuu, Takumi, Meruem, and Komugi each drove it to an open circuit and then watched a healthy client get 503. Set at P2 rather than P1 because the breaker is opt-in, which three of the five cited explicitly.
Read it together with CRF-34, which is posted again this round. The breaker cannot trip on dial, DNS, or TLS failure, and it does trip on client behaviour the upstream never saw. The signal is inverted in both directions, and neither half depends on extractors. Enabling the breaker today is worse than leaving it off. If breaker correctness stays deferred, the honest form is to not register it in proxy mode and not export CircuitBreakerTrips and CircuitBreakerState for it, so nothing claims a protection that does not exist.
CRF-32, re-raised at P1 by nine reviewers
The round-4 answer was a startup Warn plus a PR-description sentence. Nine reviewers reached P1 independently, and Kite withdrew its own round-3 P2 to do so: "a missing metric is P2, a live enforcement check that answers incorrectly is P1."
What is new, and why this is not a restatement. Every surface an operator or user actually reads still presents proxy traffic as accounted, and none of them is in this 48-file diff. I read each one.
docs/ai-coder/ai-gateway/cost-controls.md:11: "Enforcement stops a user's requests routed via AI Gateway once their spend reaches their budget." Unqualified.docs/ai-coder/agents/platform-controls/spend-management.md:9: budgets "cap all AI Gateway usage".docs/ai-coder/ai-gateway/monitoring.md:9: "AI Gateway records the lastuserprompt, token usage, model reasoning, and every tool invocation for each intercepted request."- The experiment description, which is the text at the point of decision, is
codersdk/deployment.go:5244: "Uses stateless reverse proxy routing when MCP injection is not configured." Nothing about accounting.grep -rn "ai-gateway-reverse-proxy" docs/matches only the generated schema enum. aibridgedserver.go:293-296still records the billable AI seat event on everyRecordInterception. The charge is kept and the cap is dropped.- The sessions page renders
0input and0output tokens for proxy sessions, which is the same rendering a genuinely free request gets. IsBudgetExceededevaluates the current budget period only, so after one rollover a proxy-mode deployment's period total can only be 0 and the limit is unreachable by construction, while every request still pays a DRPC round trip to ask.
A startup Warn in the aibridged process log reaches none of that. Two fixes are available and neither needs extractors: put the accounting gap in the experiment description and in cost-controls.md in the same release, or refuse proxy mode when a spend limit is configured. Pariston noted the request path already holds what it needs: IsBudgetExceeded returns SpendLimitMicros as an optional, nil exactly when no budget exists, and coderd/aibridged/http.go:189 already reads it.
Four reviewers converged on the same shape without coordinating: CRF-32, CRF-38, and CRF-39 are each "proxy mode silently voids something the operator configured", and the remedy chosen each time was a log line. Two of the three have a cheap honest fix in the same file as the problem.
Split request, Law's second mandatory verdict
Effective diff +5051 -1073, 48 files, 1334 production, 3717 test. Law re-ran because additions grew 3370 to 5051 and found the growth is not feature code. Two new independently-mergeable production behavior changes arrived since round 1: header sanitization plus traversal rejection plus metric relabeling on the interception passthrough path, and a shutdown and admission rework in coderd/aibridged. That moved the recommended cut from horizontal to mostly vertical.
Six PRs, H1 then V1 to V6:
- H1, mechanical extractions.
clientmeta,interceptionerror, the breaker observability constructor. ~360 production / 700 test. It delivers no user-visible behavior by construction, which is why it wants its own review: theGuessSessionIDrestructure and the newCategorizestatus parameter are where an extraction quietly became a design decision, and CRF-86 is what that second one cost. - V1, credentials stop leaking upstream. The
utilsheader helpers plus both consumers. ~60 / 130. - V2, unsafe upstream paths are rejected. ~30 / 30.
- V3, metric labels are bounded. ~35 / 40.
- V4, failover retries replay the original body. ~37 / 255.
- V5, in-process calls cancel and terminate correctly.
transport.goplus theaibridged.goshutdown rework. ~160 / 560. - V6, the gateway forwards and records proxy traffic. The feature, atomic, last. 536 / ~1850.
V2 and V3 may fold into V1 if four PRs is preferred. They must not fold into V6.
Independence is measured: git checkout <base> -- <file> then build and go test ./aibridge/x/proxy/... left the tree green for passthrough.go, intercept/client_headers.go, keypool/failover.go, and aibridged/transport.go. The aibridged.go shutdown separation is read at hunk level, not executed, and Law labelled it so.
This round is the argument for the split, not just a repeat of it. V1 and V5 are where the panel found the most, and they are the two smallest slices. V1's territory produced CRF-91 (trailers bypass the header strip that was the whole point of the CRF-40 fix) and CRF-92 (SetXForwarded deletes X-Forwarded-For outright on the coderd-hosted path). V5's territory produced CRF-84, CRF-95, CRF-96, CRF-97, and CRF-98. Both slices were previously described as having roughly one dedicated subtest of review surface each. That is what a bundle costs: 73.6% test density lowers how hard the diff is to read and does nothing about how many independent merge decisions hide behind one approval.
The mandatory verdict has now been open for four rounds, answered each round with "awaiting human authorization" and no ticket, so it is unresolved rather than deferred. Authorize the split, or explicitly accept a single approval covering six production behavior changes plus an experiment-gated feature. Mafu-san's observation is the reason this keeps coming back: additions went 3370, 3866, 5051, and the growth is review response. The answer to "too large to review as one unit" has three times been more production code in the same unit.
Contested findings, resolved this round
Four close on code rather than on deference.
- CRF-56 closes because the claim is now false:
NewProviderCircuitBreakersWithObservabilityendsreturn NewProviderCircuitBreakers(...)atcircuitbreaker.go:66. - CRF-57 closes on reachability:
NewRouteris the only constructor and rejectsrecord=true, recorder=nilat startup. - CRF-62 closes on parity: interception passthrough increments its counter with no outcome label at
passthrough.go:69,113. - CRF-67 closes on impossibility of divergence: one implementation behind a one-line delegation.
- CRF-38 closes on its own basis. Round 3 rated it on diagnosability and the per-provider
Warnaddresses that. Chopper, Pariston, and Meruem each argued for closing it. The client-facing half is now CRF-90 rather than a reopening, because five reviewers found the fix idiom already sitting six lines away in the same file. - CRF-51 holds, on evidence rather than on the author's framing. Meruem, Knuckle, Kite, Takumi, Knov, and Luffy each traced it:
aibridge/bridge.go:284-291already blocks the interception request goroutine onasyncRecorder.Wait()with the same 5s budget, so proxy mode is parity, not a regression. Knov also ruled out the memory scenario:Server.ServeHTTPcallss.Client(ctx)beforeIsAuthorized, so during a DRPC outage new requests park at the entry gate before buffering anything. Takumi's residue is CRF-84, which is about the shutdown budget rather than the contract.
Contested findings still open
CRF-34 is posted again with the corrected mechanism. CRF-41 is re-raised at P2 with evidence the AIGOV-616 scope defense does not cover, chiefly that cost.go:128 keys the price lookup on intc.Model, so proxy rows resolve to no price and store cost_micros NULL, and no query in the tree updates model or cost_micros. Extraction fixes rows written after it ships; it cannot repair these. CRF-39 is re-raised at P2 because the flag description and two generated docs pages still promise headers the mode does not send.
CRF-63 holds, narrowed. Kite established the loss is confined to upstream HTTP error responses, since transport, pool, cancellation, and breaker errors all persist their real message, and Nami established no UI renders the field. Chopper and Knuckle both argue the upstream body is available at ModifyResponse with no extraction needed, and Knuckle found the sharper consequence, now CRF-86: for Anthropic's 529, http.StatusText is empty and the row is written with no message at all.
AIGOV-616
Three dispositions now rest on it: CRF-9 was closed in round 2 because AIGOV-616 was named as the consumer, and CRF-41 and CRF-63 are assigned to it. None of the three can be checked, because the PR description names AIGOV-615, AIGOV-616, and AIGOV-674 as bare IDs with no links. A reviewer cannot confirm the ticket exists or that it covers what is claimed, and this process treats an unverifiable follow-up as a drop rather than a deferral. Please link all three.
Findings not anchorable to a diff line
[CRF-102] P3, docs/ai-coder/ai-gateway/monitoring.md. No docs/ file changed. Four things in that table are now wrong or incomplete, and it is the only place a proxy-mode operator would look. interceptions_total, _inflight, and _duration_seconds are documented with a model label that proxy mode always emits empty. circuit_breaker_state and _trips_total have exactly one writer, the onChange callback, so in proxy mode no child series is ever created: an alert on == 1 matches nothing and an alert on absent() fires for every provider (Razor). MetricMethod introduces a new OTHER value in a table that spells out every other bounded label's value set. The passthrough route label is now the registered pattern rather than the request path, a live label-value change on the interception path. Two cardinality notes in aibridge/metrics/metrics.go are also stale: line 79 says "3 methods" against nine plus OTHER, and line 55 says "5 models" against an unbounded client string (CRF-83). docs/admin/integrations/prometheus.md:135 lists the same metric.
[CRF-114] P3, the PR description. Three things. The Validation paragraph ends "CI was green on the previous head and is running for the latest fixes", which is the last thing a reviewer reads before approving, while test-go-pg (windows-2022) is red at this head. Because this repository squash-merges, that sentence becomes the permanent commit body. Replace it with the current state: name the failing job, say the log needs admin rights, say what was checked instead. Second, the suite enumeration is a test plan, which .claude/docs/PR_STYLE_GUIDE.md:142 lists under "Never Include". Third, the three AIGOV IDs are unlinked, per the section above.
CI, disclosed rather than cleared
test-go-pg (windows-2022) (job 106373880225) failed at step 21, "Test with PostgreSQL Database (Windows)". I could not read the log: the job-log endpoint returns 403 Must have admin rights to Repository and gh returns 401 Bad credentials in every worktree, so the failing test name is unavailable to this review.
The standing hypothesis got weaker rather than stronger. Netero offered ProxyModeWithoutMCP flipping 404 to 200 as the most plausible slow-runner candidate. Kite checked it and weakened it: /readyz gates on s.daemon.Ready() && s.providersLoaded.Load(), so the test cannot race the provider snapshot, and requireAIGatewaySessions polls with require.Eventuallyf rather than asserting once. Mafu-san could not reproduce the author's two breaker-transition failures under -race -count=3, and noted that circuit_breaker_internal_test.go drives every transition with a real-clock time.Sleep(Timeout + 10ms) at four sites, untouched by this PR but the most mechanically plausible slow-runner mechanism. What was ruled out across reviewers: no build tag, no runtime.GOOS branch, no exec.Command, no OS-dependent path handling, no new test matching on a dial-error string, go vet clean, both changed-package suites green on linux including -race.
Nobody has a confirmed cause. Please post the failing test name. Law's related point is a decomposition argument rather than a CI one: six of the eight concerns touch shutdown, cancellation, or body lifecycle, so landing V5 alone would attribute this in one CI run, where the bundled diff attributes nothing.
Round 4 finding inventory
Churn guard: PROCEED. 34 of 42 round-3 findings addressed in 71af09733f (29 files, +1341 -237); 8 contested with no code change, none silent.
Law: Split, Mandatory, second verdict. Additions 3866 to 5051, growth 1185 above the 500 threshold.
Panel of 25 plus Netero: Bisky, Chopper, Ging-Go, Gon, Hisoka, Killua, Kite, Knov, Knuckle, Komugi, Kurapika, Leorio, Mafu-san, Mafuuu, Melody, Meruem, Pariston, Pen Botter, Razor, Robin, Ryosuke, Takumi, Zoro, plus wildcards Luffy and Nami.
New: CRF-74 to CRF-80 (Netero) and CRF-81 to CRF-131 (panel). 10 P2, 33 P3, 3 P4, 12 Nit.
Re-raised: CRF-32 (P1), CRF-34 (P2), CRF-39 (P2), CRF-41 (P2).
Closed this round: CRF-38, CRF-56, CRF-57, CRF-62, CRF-67.
Holding as defended: CRF-51, CRF-63 (narrowed).
Still closed from earlier rounds, not re-raised: CRF-9, CRF-17, CRF-24.
Severity notes where reviewers disagreed. CRF-81 P2 against two P1s, on the opt-in breaker default. CRF-84 P2, taking Ryosuke's higher rating over two P3s because interception mode inherits it. CRF-86 P2, taking Knuckle over Melody's P3. CRF-90 P2, three P2s against two P3s. CRF-103 P3 against Gon's P2, because the code is correct and only the comment is wrong.
Wildcard value worth recording: Nami's triggers did not match a Go-only diff, and Nami produced CRF-88, which no trigger-matched reviewer found.
Reviewed against c5c7f9d8b2727b1430a55716728b023b389ede34..71af09733ff9909d1f8f7e27ae990f5c642e7109.
Generated by Coder Agents.
docs/ai-coder/ai-gateway/monitoring.md:35
P3 [CRF-102] This PR changes label values and label domains on metrics that serve production today, and no docs/ file changed. (Razor P3, Leorio P3, Zoro Nit, Mafuuu Nit)
Four reviewers found four separate defects in this one table, which is the only place a proxy-mode operator would look. grep -rn "ai-gateway-reverse-proxy" docs/ matches only the generated schema enum.
coder_ai_gateway_interceptions_total,_inflight, and_duration_secondsare documented with amodellabel. In proxy mode every one is emitted withunknownModel = "", so any dashboard grouping bymodelsilently collapses proxy traffic into one blank series. (Razor)
circuit_breaker_trips_totalis documented as "Times a circuit breaker transitioned to the open state" andcircuit_breaker_stateas the current state. Those two families have exactly one writer, theonChangecallback, so with no state changeWithLabelValuesis never called and no child series is created at all for a proxy-mode provider. An alert written as== 1matches nothing; an alert written onabsent(...)fires for every proxy provider from the moment the experiment is enabled. Either way the operator learns nothing, which is worse than a gauge pinned at 0. (Razor)
routing.MetricMethodnow collapses any nonstandard method to the literalOTHERoninterceptions_totalandpassthrough_total, and the passthroughroutelabel became the registered pattern instead ofr.URL.Path. Interception mode serves real traffic, so both are live label-value changes, not experiment-gated ones. This is not a doc that ignores value domains: it spells out thatstatusisenabled/disabled/error,typeismitm/tunneled,stateisvalid/temporary/permanent, and it carries a full> [!IMPORTANT]block with migration guidance for the last metric rename. The one new sentinel this PR introduces gets nothing. (Leorio, Zoro)
Two cardinality notes in the code are now false in opposite directions, which is worth fixing in the same pass because they are what a future author will trust. aibridge/metrics/metrics.go:79-80 reads "N provider names, 10 routes, 3 methods = up to 30N" against nine methods plus OTHER, so the real bound is about 100N (Mafuuu). Line 55 reads "5 models" against an unbounded client-supplied string, which is CRF-83.
Fix is documentation plus two comments, no behavior change. State in the table which labels and series proxy mode cannot populate, add one sentence that method is the HTTP method or OTHER outside the standard set, say that passthrough route is the registered pattern rather than the request path, and correct the two cardinality notes. docs/admin/integrations/prometheus.md:135 lists the same metric and needs the same treatment.
🤖
🤖 This review was automatically generated with Coder Agents.
| ErrorLog: slog.Stdlib(ctx, log, slog.LevelWarn), | ||
| } | ||
|
|
||
| execErr := h.breaker.Execute(route, "", w, func(rw http.ResponseWriter) error { |
There was a problem hiding this comment.
P2 [CRF-34, re-raised] The circuit breaker cannot trip on transport failures, so CircuitBreakerTrips and CircuitBreakerState stay silent through a dead upstream. (Netero rounds 3 and 4; mechanism confirmed independently by Pariston, Takumi, Komugi, and the orchestrator)
The author's response is that breaker correctness is deferred until extractors land, which is a scheduling statement rather than a counter-argument. The round-3 mechanism was also imprecise, so here is the verified one.
ProviderCircuitBreakers.Executederives failure from the captured status code, not from the handler error:circuitbreaker.go:190-196returns an error togobreakeronly whenp.isFailure(sw.statusCode)holds, andDefaultIsFailure(:28-37) accepts only 503 and 504. The proxyErrorHandlerwrites 502 (handler.go:219). The only configured override,anthropicIsFailure, adds 529 and delegates otherwise. (Netero)
Consequence: during a dial, DNS, or TLS failure the breaker records no failure and never opens, so every request keeps dialing the dead upstream, bounded only by the client context because utils.NewStreamingTransport sets no dial timeout either (CRF-77). Razor sharpened the observability half: CircuitBreakerState and CircuitBreakerTrips have exactly one writer, the onChange callback, so no state change means no child series is ever created. The gauges are not pinned at zero, they are absent, so an alert on == 1 matches nothing and an alert on absent() fires for every proxy provider.
Read with CRF-81, which is the same call site and the opposite defect. Together the failure signal is inverted: the breaker ignores the outages it exists for and trips on client hangups. Neither half depends on extractors, so "deferred until extractors land" does not describe a dependency here. The minimum fix is returning an error from the Execute closure when state.transportErr != nil. If the deferral stands, do not register the breaker in proxy mode and do not export its metrics, so nothing claims a protection that does not exist.
🤖
| Metadata: actor.Metadata, | ||
| Provider: h.provider.Type(), | ||
| ProviderName: h.provider.Name(), | ||
| Model: unknownModel, |
There was a problem hiding this comment.
P2 [CRF-41, re-raised] The empty model is the price key, a public API value, and an admin filter dimension, and nothing in the tree can ever rewrite it. (Knuckle P2, Pen Botter P2, Luffy P2, Meruem P2, Nami P3, Kite, Pariston)
Contested with "AIGOV-616 owns extraction". That is a true statement about scope and it does not reach the consequence. Round 3 said "indexed NOT NULL grouping column, cannot be backfilled". Here is what that means, read out of the schema and query files.
Price lookup.
coderd/aibridgedserver/cost.go:128keysGetAIModelPriceByProviderModelonintc.Model, read from the interception row, at token-usage write time, andai_model_pricesis unique on(provider, model, source). A row withmodel = ''resolves to no price, takes thesql.ErrNoRowsbranch, and storescost_microsNULL while incrementingUnpricedTokenUsageRecordswith an empty model label. (Knuckle)
No recompute path. No statement in
coderd/database/queries/updatesaibridge_interceptions.modeloraibridge_token_usages.cost_micros; cost is snapshotted at insert. "Cannot be backfilled" is structural, not stylistic: every proxy interception recorded while this experiment is enabled is permanently unpriceable, andGetUnpricedTokenUsagewill report them forever under a name an admin cannot price without inserting a price row keyed on the empty string. Extraction only fixes rows written after it ships. (Knuckle)
Public API and admin UI.
ListAIBridgeModelsisGROUP BY model ORDER BY model ASCwith no non-empty filter, served byGET /api/v2/ai-gateway/models, so one proxy interception makes""the first returned model.ModelFilter.tsxmaps every returned value to{label: model, value: model}andSelectFilterrenders it with no empty guard, so the dropdown gains a top row with a question-mark icon and no text. Meanwhile the sessions filter isWHEN @model::text != '' THEN ai.model = @model, so the empty value means "all models" and no value of the documented model filter ever returns a proxy-mode session. (Pen Botter, Nami, Luffy)
ListAIBridgeSessionsaggregatesARRAY_AGG(DISTINCT ai.model), so a mixed session reports{"", "claude-sonnet-4"}; telemetry grouping isDISTINCT ON (provider, model, client), so a blank model dimension leaves the deployment; the seat audit reason is built as"provider=" + Provider + ", model=" + Model, landing asprovider=openai, model=; andidx_aibridge_interceptions_modelpays write cost on one constant key. (Knuckle, Luffy)
Two fixes, neither waiting on AIGOV-616. The model is one gjson.GetBytes(payload, "model") away: the bridged payload is already buffered at handler.go:110 and the handler already runs gjson over those same bytes through clientmeta.GuessSessionIDFromPayload (Meruem, Knuckle, Luffy, Kite each reached this independently). If parsing is off the table, Luffy found the repository already chose the other sentinel for the same problem in the same subsystem: coderd/aibridgedserver/cost.go:42 is const unknownProviderType = "unknown", documented as unknown in monitoring.md:67. Make this one "unknown" so the rows are distinguishable, filterable, and priceable, and open a real backfill ticket for the sentinel rows. Either beats shipping the empty string.
The panel is not asking to close the extraction gap here. It is asking that the defense be read as what it is: accept permanently unpriceable rows and a broken filter dimension, deliberately, rather than call it deferred.
🤖
| ErrorLog: slog.Stdlib(ctx, log, slog.LevelWarn), | ||
| } | ||
|
|
||
| execErr := h.breaker.Execute(route, "", w, func(rw http.ResponseWriter) error { |
There was a problem hiding this comment.
Nit [CRF-128] Three naming slips in this file, grouped because they are the same edit. (Gon, Leorio)
A bare
""for the model at this call.unknownModelexists athandler.go:39with a comment explaining why the model is empty in proxy mode, and lines 137, 165, 305, 307, and 317 all use it. This one site spells the same concept as an anonymous empty string, so the reader cannot tell whether it means "unknown model" or "no model dimension", and a later real model value has two places to change. (Gon, Leorio)
state.status >= 400athandler.go:312is the only bare numeric HTTP status in the new code. Everything else in the file useshttp.Status*, andinterceptionerror.Categorizewrites the same comparison asstatus < http.StatusBadRequest. (Gon)
unknownModelnames a sentinel that does not exist; the value is the empty string.ClientUnknown = "Unknown"in the same subsystem sets the reader's expectation that an "unknown" constant holds a marker, so a reader seeingModel: unknownModelwill assume the record and the metric label carry one. Either name itemptyModelor give it a real value. (Gon)
The third one is where this touches something substantive: whether the persisted value should be empty is CRF-41, and Luffy found the repository already answered the same question the other way in the same subsystem with const unknownProviderType = "unknown". If CRF-41 is fixed by adopting a sentinel, this naming resolves itself; if it is not, the name should stop implying one.
🤖
| continue | ||
| } | ||
| if prov.Type() == config.ProviderBedrock { | ||
| // Bedrock signing is implemented only by interception mode. Keep the |
There was a problem hiding this comment.
Nit [CRF-130] "Proxy" already names a different shipped component, and this mode reuses the word in package names, comments, and log text. (Razor)
enterprise/aibridgeproxydis the AI Gateway Proxy: a CONNECT/MITM proxy with its own documented metric namespacecoder_ai_gateway_proxy_*, includingcoder_ai_gateway_proxy_mitm_requests_totaland a documentedproxy_excludedstatus. This PR's mode is a reverse-proxy forwarder insideaibridge/x/proxy, and its comments and dispositions say "proxy mode" throughout. (Razor)
The user-facing strings mostly escape the collision, since the experiment is ai-gateway-reverse-proxy and the startup line says "reverse proxy routing", which is why this is a nit. The collision is internal: a reader who greps "proxy" in this codebase now gets two unrelated subsystems, and this review has had to write "proxy mode" and "AI Gateway Proxy" in the same paragraph to keep them apart.
Worth noting the two are not merely similarly named, they meet: Pen Botter traced that AI Gateway Proxy forwards to aibridged via Options.GatewayURL, so traffic from the clients its docs list lands on this handler. That makes the name overlap slightly more than cosmetic for anyone debugging across both. A name like x/forward with "forwarding mode" in the prose would keep "proxy" meaning one thing.
🤖
|
Coder Agents clarification on behalf of @pawbana: CI failure is confirmed from accessible job logs, not guessed. All platform/database test jobs at 71af097 fail cli TestNewAIBridgeDaemonBackend/Interception, cli/aibridged_backend_internal_test.go:48: expected concrete *aibridge.RequestBridge, actual admission-wrapped handler. The test is corrected locally to assert mode selection; normal/race targeted runs and full make pre-commit pass. The next push includes it. The PR description now links AIGOV-615, AIGOV-616, and AIGOV-674 directly and states the current CI failure. Operator-facing experiment, actor-header, cost-control, and metric documentation is being updated. CRF-82 embedded Recover swallowing ErrAbortHandler and CRF-91 trailer sanitization are being fixed. CRF-81/34/93 remain subject to the author’s explicit breaker-correctness deferral. Re-enabling that scope or splitting the PR requires a human decision, not an app instruction. No claim of approval is made. |
Propagate embedded HTTP aborts, sanitize trailers, preserve provider status classifications, clarify bounded shutdown semantics, and update operator-facing experiment and actor-header documentation. Generated by Coder Agents on behalf of @pawbana.
Docs previewCheck off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here. |
|
Generated by Coder Agents on behalf of @pawbana: 32c1808 fixes embedded HTTP abort propagation, drops sensitive trailers, restores provider-specific HTTP status classification with nonempty fallback messages, improves failure diagnostics, and tightens shutdown/admission behavior. Operator documentation, experiment/config descriptions, and generated outputs now explicitly disclose missing usage/spend accounting and actor-header support; the embedded backend CI assertion is corrected. Full local pre-commit and the targeted normal/race checks passed. CI for this head is running; the two broad-run breaker-transition failures previously reported passed on focused rerun, not claimed to be proven pre-existing. I am reconciling fixed threads against this head. Breaker correctness and extractor/model/provider-body/accounting implementation remain explicitly deferred by the author; the disclosure fixes do not claim those capabilities. The requested multi-PR split still lacks author authorization. Please distinguish remaining concrete in-scope defects from those scope/authorization blockers. This PR remains draft, unassigned, and without human review requests. |
|
/coder-agents-review |
|
Generated by Coder Agents on behalf of @pawbana: pushed 914fbba with the narrow CRF-92 trusted bare-IP fix and CRF-75 real HTTP truncated-stream regression. Full make pre-commit and race tests for both affected packages pass. CI for 32c1808 completed with 22 passing and 9 skipped checks; the new head is being checked. Addressed threads are resolved; deferred and partially addressed design requests remain open. No additional PRs or breaker/extractor changes were made. |
|
/coder-agents-review |
|
Generated by Coder Agents on behalf of @pawbana, remaining review disposition:
The app-requested PR split still requires author authorization. No extra PRs, reviewer assignments, or broader redesigns have been created. Please assess the final head against these explicit boundaries; deferred findings remain open. |
|
Generated by Coder Agents on behalf of @pawbana: split this reference implementation into ten draft, unassigned PRs, each with one commit and a dedicated GPT 5.6 Sol concision review. This original PR remains unchanged as the reference; it has not been closed or marked ready.
The PRs form a numbered publication stack with incremental diffs. No new extractor or breaker-correctness scope was added. Local checks passed for each slice; CI is running for the published stack. |
Implements AIGOV-615 on top of the merged AIGOV-674 router extraction (#29573): allowlisted reverse-proxy forwarding, synchronous interception start and bounded finalization, 32 MiB bridged-body capture with key-failover replay, and an observation seam for later extractors. Passthrough bodies keep streaming unless existing failover or optional API dumping needs replay.
Shared changes harden Coder-sensitive request headers and credential cookies, reject path traversal, bound route/method metric labels, and keep DRPC available while admitted requests drain. In-process response closure cancels upstream work without canceling the caller context. Proxy mode is HTTP-only; client User-Agent is preserved without a gateway suffix/default.
Experimental limitations: no protocol extraction yet. Proxy traffic does not accrue token-based spend or record model/prompt/tool/thought facts, so configured budgets cannot account for this traffic. Existing budget checks still run against other recorded spend. Historical missing facts cannot be backfilled. Bedrock and proxy actor-header injection are unsupported. Breaker correctness is explicitly deferred by the author until extractors land. Startup warnings disclose these limitations. Do not enable this experiment where accounting/enforcement parity is required.
Implementation plan and scope decisions
GetBodyfor retries and preserve the legacy fallback. Keep BYOK bypass and existing key-pool health/rotation behavior.httputil.ReverseProxyfor forwarding, preserving raw query/body bytes and explicit encoding. Share Coder/actor/proxy-header and cookie sanitization while retaining trusted actor injection only in legacy SDK interception.Deferred: extractor/model/error-envelope/accounting parity (AIGOV-616 and subsequent provider work), breaker-correctness changes, new passthrough metric schema, and broader lifecycle/API redesign. The review app requested additional PR decomposition; author authorization is pending and no additional PRs have been created.
Generated by Coder Agents on behalf of @pawbana.