Report unsupported external auth types - #6242
Conversation
|
Hi @jhrozek — no rush at all, just flagging that this is ready for review whenever you have time. Happy to adjust anything about the approach. |
jhrozek
left a comment
There was a problem hiding this comment.
Thanks for digging into #5930 — the compatibility matrix itself is right. I checked every cell against the code rather than against the docs: the VirtualMCPServer and MCPServerEntry rows match the converter registry in pkg/vmcp/auth/converters/interface.go exactly, and the MCPServer/MCPRemoteProxy rows line up with the dispatcher switch in controllerutil/tokenexchange.go, including the non-obvious awsSts-on-MCPServer case, which fails on the empty-RemoteURL check in pkg/runner/middleware.go rather than silently no-opping. I also traced the reconcile flow to confirm it returns before the Deployment block, so this does close the unauthenticated-deploy footgun rather than just annotating it.
My main concern is scope. The issue asked for two things: reconcile-time validation surfaced as status conditions, and the supported consumer kinds documented in the field godoc. Both are here. But a lot came along with them, and some of it changes behaviour nobody asked to change.
The one I'd push back on hardest is the outgoing-auth precedence change in pkg/vmcp/aggregator/discoverer.go. In discovered mode an explicit per-backend override now wins over auth discovered from the backend resource, where previously discovered auth won. That decides which credential a live backend authenticates with. It may well be the right call, but it needs its own issue and its own discussion — it shouldn't be settled as a side effect of a PR about reporting unsupported types. Same goes for ExplicitBackends and its nil-versus-empty compatibility semantics, which exist only to support that change.
ExcludedBackends and DefaultAuthFailed are closer to the mark but still past the ask. The issue's vMCP line is "surface the skipped backend in VirtualMCPServer status rather than log-and-drop", which is a status condition. What's here is a fail-closed routing rework threaded through the discoverer, the watcher and the static path. I'd keep the condition and leave routing semantics for a separate change.
The examples fix (snake_case to camelCase under examples/operator/external-auth/) is a real bug and a good catch, but it's unrelated — worth its own PR so it can land quickly on its own merits.
Two things I'd treat as blocking regardless of what happens to the scope:
ensureVmcpConfigConfigMap now deep-copies the VirtualMCPServer, sets Spec.OutgoingAuth to nil, runs converter.Convert against that, then patches Source back on afterwards. That line is the only caller of Convert in the tree, so Converter.convertOutgoingAuth and convertBackendAuthConfig are now unreachable in production — they survive only in their own unit tests, which will keep passing while covering code that no longer runs. The controller already had a near-duplicate in convertBackendAuthConfigToVMCP, and this PR updates that copy to match the converter's behaviour. Better to pick one and delete the other than to disable the original by feeding it a modified input.
ExternalAuthConfigValidated on MCPServer has no path back to True. Both False branches are there (the new UnsupportedAuthType one and the pre-existing MultiUpstream one), plus RemoveStatusCondition when the ref goes away, but there's no success setter — MCPRemoteProxy has the matching block at the end of its handleExternalAuthConfig and MCPServer doesn't. Point a server at a two-upstream config, then edit that config down to one, and the condition stays False permanently. The MultiUpstream half predates this PR, but the new branch reproduces it in a function you're already editing.
Smaller things, several of which follow conventions already present in the files being changed:
excludedBackendsandexplicitBackendsneed+listType=atomic. That's what's failing the CRD Schema Compatibility check, and there's an existing+listType=atomicabout 200 lines up inpkg/vmcp/config/config.go. The other red checks aren't yours: the E2E lifecycle jobs all died on a Docker Hub 502 pullingpython:3.9-slim, and the unit-test failure is a flaky transparent-proxy test unrelated to this change.pkg/vmcp/backendregistry/registry.go:130still callsNewBackendWatcherwithout the auth config, and the comment directly above it asks that it stay in sync withcli/serve.go. It's only reachable frominternal/exampleembeddertoday, butNewKubernetesBackendRegistryis the exported constructor for embedders and now has no way to pass anOutgoingAuthConfigat all.- The "failed default means deny" rule is implemented three times:
ResolveForBackend,dependsOnFailedDefaultin the discoverer, and again inline inbackend_reconciler.go. They agree today, nothing keeps them agreeing, and drift reopens the hole this is meant to close. Same foroutgoingAuthExclusionSet, copy-pasted verbatim into two packages — it'd be better as a method onOutgoingAuthConfig. support_test.gore-declaressupportMatrixas a literal and assertsSupports()matches it, so it can only fail on a typo made differently twice. The test that would earn its place asserts the vMCP rows equal the key set ofconverters.DefaultRegistry()and the MCPServer/MCPRemoteProxy rows equal the non-no-op arms of the dispatcher switch. That one breaks when someone adds a converter and forgets the table, which is the failure this package exists to prevent.- Three new
...WithAuthConfigconstructors for two production callers of the old ones. Changing the signatures is smaller. - MCPRemoteProxy sets
Ready=FalsewithReason: DeploymentNotReadyfor an external-auth configuration error.
One structural thought worth weighing before the next revision. The silent no-ops in AddExternalAuthConfigOptions are the actual subject of #5930 and they're still there — headerInjection, upstreamInject and xaa all still return nil. The support matrix is a second, hand-maintained source of truth sitting beside them, with nothing enforcing agreement. Making those arms return the UnsupportedTypeError directly would put the guard at the choke point every consumer already routes through, and the MCPServer/MCPRemoteProxy rows could then derive from the switch instead of restating it.
MCPExternalAuthConfig has nine types, but each consumer kind implements only a subset, and the unsupported combinations failed silently: the resource read as valid, the workload ran, and the credential was never injected (stacklok#5930). Validate the consumer/type combination at reconcile time and surface failures as ExternalAuthConfigValidated=False with reason UnsupportedAuthType on MCPServer, MCPRemoteProxy, and MCPServerEntry, and as per-backend conditions on VirtualMCPServer. Terminal configuration errors return without requeueing; the watches on the spec and the referenced config trigger reconciliation when either changes. Document each consumer's supported types in the field godoc so the CRD schemas and reference docs carry them. The support matrix lives in one place and cannot drift from the code: the vMCP-backed rows derive from the converter registry, and the MCPServer/MCPRemoteProxy rows are enforced at the run-config dispatch in AddExternalAuthConfigOptions, which now takes the consumer and returns a typed UnsupportedTypeError instead of keeping silent no-op arms for headerInjection, upstreamInject, and xaa. A test drives that dispatch with every type for both consumers and requires agreement with the matrix in both directions. The generic vMCP converter no longer converts outgoing auth. The reconciler's status-aware path (processOutgoingAuth) was already the authority on it and is now the only implementation; the converter's unreachable copy is deleted rather than bypassed. This also turns the inline-mode hard error for a single broken backend into a per-backend condition with the remaining backends still served. handleExternalAuthConfig on MCPServer now sets the condition back to True once a previously rejected configuration is fixed, matching MCPRemoteProxy, and MCPRemoteProxy reports Ready=False with reason AuthInvalid rather than DeploymentNotReady for an auth configuration failure. Closes stacklok#5930. Signed-off-by: Enes Deniz <142517728+3nesdeniz@users.noreply.github.com>
8fda367 to
57eb59b
Compare
|
Thanks for the thorough review, @jhrozek — checking the matrix cell-by-cell against the code is exactly what this needed, and you were right about the scope. Reworked accordingly; the diff against main went from ~+3,300 to +1,620/−380, and every runtime behaviour change is gone. Scope
The two blockers
Structural suggestion — taken. Smaller items. Validation: build and |
Summary
MCPExternalAuthConfig has nine
spec.typevalues, but each consumer kind implements only a subset, and the unsupported combinations fail silently: the resource reads as valid, the workload runs, and the credential is never injected. This PR delivers the two things #5930 asks for:ExternalAuthConfigValidated=Falsewith reasonUnsupportedAuthTypeon MCPServer, MCPRemoteProxy, and MCPServerEntry, and a per-backend condition on VirtualMCPServer (including the discovered-mode case where abearerToken-backed MCPRemoteProxy used to vanish from aggregation with only a log line). Terminal configuration errors return without requeueing; the existing watches trigger reconciliation when the spec or the referenced config changes.MCPExternalAuthConfigSpec.Typeand on every consumer'sexternalAuthConfigRef, flowing into the generated CRD schemas and reference docs.The support matrix lives in
cmd/thv-operator/pkg/externalauthsupportand cannot drift from the code: the VirtualMCPServer/MCPServerEntry rows derive from the converter registry (converters.DefaultRegistry().RegisteredTypes()), and the MCPServer/MCPRemoteProxy rows are enforced at the run-config dispatch —AddExternalAuthConfigOptionsnow takes the consumer, validates against the matrix, and the silent no-op arms forheaderInjection/upstreamInject/xaaare gone.TestAddExternalAuthConfigOptions_EnforcesSupportMatrixdrives the real dispatch with all nine types for both consumers and requires agreement with the matrix in both directions.Two consolidations that fell out of review:
processOutgoingAuth) was already the authority and is now the only implementation; the converter's copy (convertOutgoingAuthWithDefaults/convertOutgoingAuth/convertBackendAuthConfig) is deleted rather than bypassed with a modified input. Discovered mode persists exactly what the spec declares (default + inline entries), as before. The one behaviour change is intentional and in scope: a single broken backend in inline mode now degrades to a per-backend condition instead of failing the whole conversion.handleExternalAuthConfigon MCPServer sets the condition back to True once a previously rejected config is fixed (it previously had no success setter, so the condition stuck at False forever), and MCPRemoteProxy reportsReady=Falsewith reasonAuthInvalidinstead ofDeploymentNotReadyfor an auth configuration failure.Out of scope, split out per review
pkg/vmcpruntime (discoverer, watcher, backend reconciler, workloads, CLI) is untouched by this PR. The precedence question is now Discovered-mode outgoing auth: should an explicit per-backend override win over discovered auth? #6329.Validation
go build ./...andgolangci-lint runon the touched trees: cleango test ./cmd/thv-operator/... ./pkg/vmcp/...: all unit suites passtest-integrationsuite under envtest (all 11 packages, including mcp-remote-proxy and virtualmcp): passmain