fix(mcp): skip the platform auth tests where their preconditions do not hold - #2231
Conversation
…ot hold These tests assert that a credential is *rejected*, which only means something if the endpoint exists and something is there to reject it. Neither holds on Unstract Cloud, and both went unnoticed because a missing endpoint fails in the direction that looks like success. Cloud main went red the moment #2207 landed. `copy_cloud_deps` overwrites the OSS `settings/test.py` with a redirect to `test_cloud`, which derives from `settings/cloud` — so `MCP_PLATFORM_SERVER_ENABLED = True` never reaches the cloud suite, `urls_v2` leaves the route unmounted, and five of these tests 404. The sixth is the reason this is a skip and not a deselect: `test_bad_credentials_are_rejected` *passed* on cloud. A 404 satisfies "this request is refused" exactly as well as the 401 it was written to assert, so it reported green for auth that was never reached. Ignoring the file would have kept that hidden; a skip names it. Enabling the flag on cloud would not have fixed it either. The same `test_cloud.py` drops `CUSTOM_AUTH_MIDDLEWARE` from the default test `MIDDLEWARE`, and this view carries `permission_classes = []` and deliberately does not re-authenticate — so the endpoint would have gone from unmounted to unauthenticated, trading five failures for a different five. So both preconditions are checked, and checked directly rather than by reading `MCP_PLATFORM_SERVER_ENABLED`: the flag is what causes the mount in the OSS URLconf, but it is not what these tests need. Asking for the route and the middleware stays correct for any tree that mounts the server another way, and the skip re-arms on its own if cloud ever runs this endpoint with auth. The route probe resolves a path with no organization segment, unlike the URL the tests request — tenant middleware strips the org from `path_info` before resolution, so resolving the request URL as written 404s even on OSS. Verified against real merged trees: OSS reports no unmet precondition and the suite runs unchanged; the cloud tree reports the route reason with the flag off and the middleware reason with it on. 198 unit tests still pass. This does not change what ships. The deployment-scoped MCP server — the one required for release — is mounted unconditionally in `api_v2/execution_urls.py` and is unaffected.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
Summary by CodeRabbit
WalkthroughThe platform authentication tests now verify MCP server wiring before setup. They skip when the platform server is disabled. Existing JSON-RPC payload and invalid-credentials assertions retain their behavior. ChangesPlatform authentication test suite
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
| Filename | Overview |
|---|---|
| backend/mcp_server/tests/test_platform_auth.py | Adds setup-time feature, routing, and middleware precondition checks to prevent misleading authentication-test results in configurations where the organization-scoped MCP server is disabled or misconfigured. |
Reviews (2): Last reviewed commit: "fix(mcp): gate on the flag, and assert t..." | Re-trigger Greptile
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@backend/mcp_server/tests/test_platform_auth.py`:
- Around line 83-84: Update the route check around resolve() in the affected
test to require the resolved match’s url_name to equal "platform_mcp_server" (or
verify the callable is PlatformMCPServerView) before skipping; do not treat any
URL match for the tenant MCP path as sufficient.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: fa185701-8bcd-4652-9c0e-76946780a894
📒 Files selected for processing (1)
backend/mcp_server/tests/test_platform_auth.py
Addresses review: skipping on a 404 can hide a genuine regression. That is right, and it is the direction that matters. Where the org-scoped server is meant to be enabled, an unmounted route is a bug — and the previous version would have gone quiet exactly when it should have shouted, which is the same failure this suite exists to catch, one level up. So the flag now decides whether the server is expected here, and the route and middleware become assertions rather than skip conditions: MCP_PLATFORM_SERVER_ENABLED off -> skip; the server is deliberately absent (Unstract Cloud ships it that way) and there is nothing to authenticate. MCP_PLATFORM_SERVER_ENABLED on -> the route must resolve and CUSTOM_AUTH_MIDDLEWARE must be in MIDDLEWARE, or the suite fails and says which one is missing. The middleware assertion is kept because the flag alone does not make these tests meaningful: cloud's `test_cloud.py` drops CUSTOM_AUTH_MIDDLEWARE from the default test MIDDLEWARE, and this view carries `permission_classes = []` and does not re-authenticate. Enabling the flag there without the middleware would move the endpoint from unmounted to unauthenticated — so if someone turns the server on for the cloud suite, this fails with the reason rather than passing against an open endpoint. Verified in all four states: enabled and wired up runs; disabled skips; enabled with the middleware dropped fails naming the middleware; enabled against a URLconf without the mount fails naming the path. 198 unit tests still pass.
|
You're right, and it's the more important half of the objection — taking it.
My reasoning for the 404 check was that the flag causes the mount but isn't what the tests need. That held for the case in front of me and missed yours: where the server is meant to be on, an unmounted route is a regression, and my version would have gone quiet exactly when it should have shouted — the same failure this suite exists to catch, one level up. I kept the middleware assertion, because the flag alone doesn't make these tests meaningful. Cloud's Four states, all exercised:
On the broader class — agreed that's the real problem, and I'd split it in two: Settings divergence. Floating OSS ref. Cloud CI resolves Happy to take either if useful — the settings-drift guard is small and would have caught this one before it merged. |
|
Unstract test resultsPer-group results
Critical paths
|



Unblocks Unstract Cloud
main, red since #2207 merged (cloud run 31159495978 and every run since).What is broken
mcp_server/tests/test_platform_auth.pydrives requests through the real URL stack soCustomAuthMiddlewareruns. On cloud neither thing it depends on is true:copy_cloud_deps.pyoverwrites the OSSsettings/test.pywith a redirect totest_cloud.py, which derives fromsettings/cloud.py→base.py. SoMCP_PLATFORM_SERVER_ENABLED = Truenever reaches the cloud suite, andurls_v2skips the mount. Five tests 404.test_cloud.pydropsCUSTOM_AUTH_MIDDLEWAREfrom the default testMIDDLEWARE(it is kept only inMIDDLEWARE_WITH_AUTH). This view haspermission_classes = []and deliberately does not re-authenticate.Why it passed pre-merge: cloud
ci-test.yamlchecks out OSS at a floatingref: main. #1712 was green at 07:42 against an OSS main wheremcp_server/did not exist; #2207 landed at 07:54:05 and the next cloud run was red. Nothing re-validates a merged cloud PR against a newer OSS main — worth addressing separately.The sixth test is why this is a skip, not a deselect
test_bad_credentials_are_rejectedpassed on cloud. A 404 satisfies "this request is refused" exactly as well as the 401 it was written to assert, so it reported green for auth that was never reached.--ignore-ing the file keeps that hidden; a skip names it.Why not just enable the flag on cloud
It mounts the route but does not go green — with the middleware still absent the endpoint goes from unmounted to unauthenticated, trading five failures for a different five. I verified this on a merged tree before discarding the approach.
Why not override the group in
groups.cloud.yamlpytest_extra: ["--ignore=..."]onintegration-backendcannot work. The rig rejects redefinition:_merge_manifestraises on name collision by design ("a name collision is an error rather than a silent override"), and it fails at theValidate rig manifestsstep before any test runs. There is no ignore/exclude/disable key in the rig either. So the fix has to live here.What this does
Checks the two preconditions directly and skips with a named reason otherwise — deliberately not by reading
MCP_PLATFORM_SERVER_ENABLED. That flag is what causes the mount in the OSS URLconf, but it is not what these tests need; asking for the route and the middleware stays correct for any tree that mounts the server another way, and re-arms on its own if cloud ever runs this endpoint with auth.The route probe resolves a path with no organization segment, unlike the URL the tests request: tenant middleware strips the org from
path_infobefore resolution, so resolving the request URL as written 404s even on OSS where these tests pass.Can this PR break any existing features? If yes, please list possible items. If no, please explain why.
No.
api_v2/execution_urls.pyand is unaffected by any of this.Notes on Testing
Verified against real
copy_cloud_deps-merged trees rather than by reasoning:mainBoth guard branches therefore exercised.
198 passedin the OSS unit tier.Related Issues or PRs