Skip to content

fix(mcp): skip the platform auth tests where their preconditions do not hold - #2231

Merged
chandrasekharan-zipstack merged 2 commits into
mainfrom
fix/skip-platform-mcp-tests-when-route-unmounted
Aug 7, 2026
Merged

fix(mcp): skip the platform auth tests where their preconditions do not hold#2231
chandrasekharan-zipstack merged 2 commits into
mainfrom
fix/skip-platform-mcp-tests-when-route-unmounted

Conversation

@Deepak-Kesavan

Copy link
Copy Markdown
Contributor

Unblocks Unstract Cloud main, red since #2207 merged (cloud run 31159495978 and every run since).

What is broken

mcp_server/tests/test_platform_auth.py drives requests through the real URL stack so CustomAuthMiddleware runs. On cloud neither thing it depends on is true:

  • The route is not mounted. copy_cloud_deps.py overwrites the OSS settings/test.py with a redirect to test_cloud.py, which derives from settings/cloud.pybase.py. So MCP_PLATFORM_SERVER_ENABLED = True never reaches the cloud suite, and urls_v2 skips the mount. Five tests 404.
  • The auth middleware is absent. The same test_cloud.py drops CUSTOM_AUTH_MIDDLEWARE from the default test MIDDLEWARE (it is kept only in MIDDLEWARE_WITH_AUTH). This view has permission_classes = [] and deliberately does not re-authenticate.

Why it passed pre-merge: cloud ci-test.yaml checks out OSS at a floating ref: main. #1712 was green at 07:42 against an OSS main where mcp_server/ did not exist; #2207 landed at 07:54:05 and the next cloud run was red. Nothing re-validates a merged cloud PR against a newer OSS main — worth addressing separately.

The sixth test is why this is a skip, not a deselect

test_bad_credentials_are_rejected passed on cloud. A 404 satisfies "this request is refused" exactly as well as the 401 it was written to assert, so it reported green for auth that was never reached. --ignore-ing the file keeps that hidden; a skip names it.

Why not just enable the flag on cloud

It mounts the route but does not go green — with the middleware still absent the endpoint goes from unmounted to unauthenticated, trading five failures for a different five. I verified this on a merged tree before discarding the approach.

Why not override the group in groups.cloud.yaml

pytest_extra: ["--ignore=..."] on integration-backend cannot work. The rig rejects redefinition:

ERROR: tests/groups.cloud.yaml: group 'integration-backend' already defined in a prior manifest

_merge_manifest raises on name collision by design ("a name collision is an error rather than a silent override"), and it fails at the Validate rig manifests step before any test runs. There is no ignore/exclude/disable key in the rig either. So the fix has to live here.

What this does

Checks the two preconditions directly and skips with a named reason otherwise — deliberately not by reading MCP_PLATFORM_SERVER_ENABLED. That flag is what causes the mount in the OSS URLconf, but it is not what these tests need; asking for the route and the middleware stays correct for any tree that mounts the server another way, and re-arms on its own if cloud ever runs this endpoint with auth.

The route probe resolves a path with no organization segment, unlike the URL the tests request: tenant middleware strips the org from path_info before resolution, so resolving the request URL as written 404s even on OSS where these tests pass.

Can this PR break any existing features? If yes, please list possible items. If no, please explain why.

No.

  • Test-only change; no production code is touched.
  • On OSS both preconditions hold, so the suite runs exactly as before — verified, no unmet precondition reported.
  • The deployment-scoped MCP server, the one required for release, is mounted unconditionally in api_v2/execution_urls.py and is unaffected by any of this.
  • Nothing enables the organization-scoped server anywhere; it stays off in staging and production, as intended.

Notes on Testing

Verified against real copy_cloud_deps-merged trees rather than by reasoning:

tree verdict
OSS main no unmet precondition → suite runs unchanged
cloud merged, flag off (today) skip, reporting the route reason
cloud merged, flag on skip, reporting the middleware reason

Both guard branches therefore exercised. 198 passed in the OSS unit tier.

Related Issues or PRs

…ot hold

These tests assert that a credential is *rejected*, which only means something
if the endpoint exists and something is there to reject it. Neither holds on
Unstract Cloud, and both went unnoticed because a missing endpoint fails in the
direction that looks like success.

Cloud main went red the moment #2207 landed. `copy_cloud_deps` overwrites the
OSS `settings/test.py` with a redirect to `test_cloud`, which derives from
`settings/cloud` — so `MCP_PLATFORM_SERVER_ENABLED = True` never reaches the
cloud suite, `urls_v2` leaves the route unmounted, and five of these tests 404.

The sixth is the reason this is a skip and not a deselect:
`test_bad_credentials_are_rejected` *passed* on cloud. A 404 satisfies "this
request is refused" exactly as well as the 401 it was written to assert, so it
reported green for auth that was never reached. Ignoring the file would have
kept that hidden; a skip names it.

Enabling the flag on cloud would not have fixed it either. The same
`test_cloud.py` drops `CUSTOM_AUTH_MIDDLEWARE` from the default test
`MIDDLEWARE`, and this view carries `permission_classes = []` and deliberately
does not re-authenticate — so the endpoint would have gone from unmounted to
unauthenticated, trading five failures for a different five.

So both preconditions are checked, and checked directly rather than by reading
`MCP_PLATFORM_SERVER_ENABLED`: the flag is what causes the mount in the OSS
URLconf, but it is not what these tests need. Asking for the route and the
middleware stays correct for any tree that mounts the server another way, and
the skip re-arms on its own if cloud ever runs this endpoint with auth.

The route probe resolves a path with no organization segment, unlike the URL
the tests request — tenant middleware strips the org from `path_info` before
resolution, so resolving the request URL as written 404s even on OSS.

Verified against real merged trees: OSS reports no unmet precondition and the
suite runs unchanged; the cloud tree reports the route reason with the flag off
and the middleware reason with it on. 198 unit tests still pass.

This does not change what ships. The deployment-scoped MCP server — the one
required for release — is mounted unconditionally in `api_v2/execution_urls.py`
and is unaffected.
@coderabbitai

coderabbitai Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: f139abe2-0300-436a-877b-6b2638e9d8df

📥 Commits

Reviewing files that changed from the base of the PR and between d6d6f91 and 1acd357.

📒 Files selected for processing (1)
  • backend/mcp_server/tests/test_platform_auth.py

Summary by CodeRabbit

  • Tests
    • Improved integration test setup to detect unavailable platform routes or missing authentication configuration.
    • Tests now clearly report why they are skipped when required conditions are not met.
    • Preserved existing invalid-credentials validation behavior.
    • Added validation to confirm platform authentication is correctly connected before test data is created.

Walkthrough

The platform authentication tests now verify MCP server wiring before setup. They skip when the platform server is disabled. Existing JSON-RPC payload and invalid-credentials assertions retain their behavior.

Changes

Platform authentication test suite

Layer / File(s) Summary
Server wiring and setup gating
backend/mcp_server/tests/test_platform_auth.py
The suite resolves the tenant MCP route and checks custom authentication middleware before creating test data. It skips setup when the platform server is disabled.
Request and assertion formatting
backend/mcp_server/tests/test_platform_auth.py
The JSON-RPC payload and invalid-credentials assertion formatting changed without behavioral changes.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: hari-kuriakose

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the MCP platform authentication test change and the precondition-based skip behavior.
Description check ✅ Passed The description explains the problem, implementation, impact, related issues, and testing results, and it includes the required feature-impact section.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/skip-platform-mcp-tests-when-route-unmounted

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@greptile-apps

greptile-apps Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This test-only change gates the organization-scoped MCP platform authentication suite on its feature flag and asserts that the enabled route and authentication middleware are correctly wired before creating test data.

  • Skips the suite when the organization-scoped MCP server is intentionally disabled.
  • Fails explicitly when an enabled server lacks its route or required authentication middleware.
  • Reformats two existing assertions without changing their behavior.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
backend/mcp_server/tests/test_platform_auth.py Adds setup-time feature, routing, and middleware precondition checks to prevent misleading authentication-test results in configurations where the organization-scoped MCP server is disabled or misconfigured.

Reviews (2): Last reviewed commit: "fix(mcp): gate on the flag, and assert t..." | Re-trigger Greptile

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@backend/mcp_server/tests/test_platform_auth.py`:
- Around line 83-84: Update the route check around resolve() in the affected
test to require the resolved match’s url_name to equal "platform_mcp_server" (or
verify the callable is PlatformMCPServerView) before skipping; do not treat any
URL match for the tenant MCP path as sufficient.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: fa185701-8bcd-4652-9c0e-76946780a894

📥 Commits

Reviewing files that changed from the base of the PR and between aac5ede and d6d6f91.

📒 Files selected for processing (1)
  • backend/mcp_server/tests/test_platform_auth.py

Comment thread backend/mcp_server/tests/test_platform_auth.py Outdated
Addresses review: skipping on a 404 can hide a genuine regression.

That is right, and it is the direction that matters. Where the org-scoped
server is meant to be enabled, an unmounted route is a bug — and the previous
version would have gone quiet exactly when it should have shouted, which is the
same failure this suite exists to catch, one level up.

So the flag now decides whether the server is expected here, and the route and
middleware become assertions rather than skip conditions:

  MCP_PLATFORM_SERVER_ENABLED off -> skip; the server is deliberately absent
  (Unstract Cloud ships it that way) and there is nothing to authenticate.

  MCP_PLATFORM_SERVER_ENABLED on  -> the route must resolve and
  CUSTOM_AUTH_MIDDLEWARE must be in MIDDLEWARE, or the suite fails and says
  which one is missing.

The middleware assertion is kept because the flag alone does not make these
tests meaningful: cloud's `test_cloud.py` drops CUSTOM_AUTH_MIDDLEWARE from the
default test MIDDLEWARE, and this view carries `permission_classes = []` and
does not re-authenticate. Enabling the flag there without the middleware would
move the endpoint from unmounted to unauthenticated — so if someone turns the
server on for the cloud suite, this fails with the reason rather than passing
against an open endpoint.

Verified in all four states: enabled and wired up runs; disabled skips; enabled
with the middleware dropped fails naming the middleware; enabled against a
URLconf without the mount fails naming the path. 198 unit tests still pass.
@Deepak-Kesavan

Copy link
Copy Markdown
Contributor Author

You're right, and it's the more important half of the objection — taking it.

1acd357a inverts the gate. The flag now decides whether the server is expected, and the route and middleware become assertions rather than skip conditions:

MCP_PLATFORM_SERVER_ENABLED off -> skip; deliberately absent, nothing to authenticate
MCP_PLATFORM_SERVER_ENABLED on  -> route must resolve AND CUSTOM_AUTH_MIDDLEWARE
                                   must be in MIDDLEWARE, or fail naming which

My reasoning for the 404 check was that the flag causes the mount but isn't what the tests need. That held for the case in front of me and missed yours: where the server is meant to be on, an unmounted route is a regression, and my version would have gone quiet exactly when it should have shouted — the same failure this suite exists to catch, one level up.

I kept the middleware assertion, because the flag alone doesn't make these tests meaningful. Cloud's test_cloud.py drops CUSTOM_AUTH_MIDDLEWARE from the default test MIDDLEWARE, and this view has permission_classes = [] and doesn't re-authenticate — so enabling the flag there without the middleware moves the endpoint from unmounted to unauthenticated, and the rejection tests would pass against an open endpoint. With the assertion it fails and says so. I verified that specific case rather than assuming it.

Four states, all exercised:

state outcome
enabled, wired up (OSS) runs
disabled (cloud today) skips
enabled, middleware dropped fails, naming the middleware
enabled, URLconf without the mount fails, naming the path

198 passed in the unit tier.


On the broader class — agreed that's the real problem, and I'd split it in two:

Settings divergence. copy_cloud_deps overwrites OSS settings/test.py with a redirect to test_cloud.py, which derives from settings/cloud.py. So every key OSS sets for tests is silently dropped downstream. I diffed the two: MCP_PLATFORM_SERVER_ENABLED is currently the only divergence, so we caught this at one. A test asserting the two files agree on a declared set of keys would turn the next one into a named failure instead of a mystery 404 in an unrelated suite.

Floating OSS ref. Cloud CI resolves Zipstack/unstract@main at run time, so a cloud PR's green tick is only valid against whatever OSS main was at that instant — which is exactly how #1712 merged clean and cloud went red 9 seconds later. Nothing re-validates a merged cloud PR against a newer OSS main. Worth its own issue; options range from pinning the ref to a scheduled cloud-main run against latest OSS.

Happy to take either if useful — the settings-drift guard is small and would have caught this one before it merged.

@sonarqubecloud

sonarqubecloud Bot commented Aug 7, 2026

Copy link
Copy Markdown

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Unstract test results

Per-group results

Status Group Tier Passed Failed Errors Skipped Duration (s)
e2e-api-deployment e2e 3 0 0 0 16.4
e2e-coowners e2e 1 0 0 0 1.2
e2e-etl e2e 1 0 0 0 9.6
e2e-login e2e 2 0 0 0 0.9
e2e-prompt-studio e2e 1 0 0 0 4.3
e2e-smoke e2e 2 0 0 0 1.1
e2e-workflow e2e 1 0 0 0 14.1
integration-backend integration 267 0 0 26 39.9
integration-connectors integration 1 0 0 7 9.2
integration-workers integration 140 0 0 1 47.6
unit-backend unit 998 0 0 1 38.7
unit-connectors unit 63 0 0 0 9.7
unit-core unit 33 0 0 0 1.3
unit-platform-service unit 15 0 0 0 2.5
unit-rig unit 109 0 0 0 5.2
unit-sdk1 unit 480 0 0 0 23.0
unit-workers unit 1335 0 0 1 97.9
TOTAL 3452 0 0 36 322.7

Critical paths

⚠️ Critical paths not yet covered

  • workflow-execution-fan-out — Multi-file workflow execution fans out to file-processing workers and rejoins. (declared coverage: no groups declared)
✅ Covered critical paths
  • auth-login — covered by e2e-login
  • adapter-register-llm — covered by integration-backend
  • workflow-author — covered by integration-backend
  • co-owner-manage — covered by integration-backend, e2e-coowners
  • workflow-create-execute — covered by e2e-workflow
  • api-deployment-provision — covered by integration-backend
  • api-deployment-auth — covered by integration-backend
  • api-deployment-run — covered by e2e-api-deployment
  • mcp-server-auth — covered by integration-backend
  • mcp-platform-auth — covered by integration-backend
  • prompt-studio-author — covered by integration-backend
  • prompt-studio-fetch-response — covered by e2e-prompt-studio
  • connector-register-test — covered by integration-backend
  • pipeline-etl-execute — covered by e2e-etl
  • usage-aggregate-read — covered by integration-backend
  • usage-token-tracking — covered by e2e-api-deployment
  • callback-result-delivery — covered by e2e-api-deployment

@chandrasekharan-zipstack
chandrasekharan-zipstack merged commit f74c39e into main Aug 7, 2026
11 checks passed
@chandrasekharan-zipstack
chandrasekharan-zipstack deleted the fix/skip-platform-mcp-tests-when-route-unmounted branch August 7, 2026 11:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants