feat(completion): forward completion/complete to owning upstream (SDK 2.0) - #6760

…_client

Adds CompletionService._acquire_upstream_session(), mirroring
PromptService._fetch_gateway_prompt_result()'s existing pattern: reuse the
upstream session pinned to the current downstream Mcp-Session-Id via
UpstreamSessionRegistry when one is in scope, otherwise fall back to a
short-lived session via mcp_proxy_client(). Neither branch hand-rolls a raw
ClientSession/transport — both go through Client(transport, mode=...), so
the fallback path negotiates the modern protocol the same way the registry
path does instead of being pinned to the legacy handshake.

Refs #6629

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

Adds CompletionService._error_from_upstream() (maps an upstream MCPError's
own JSON-RPC code to the matching CompletionError subclass, per R3/R4) and
_forward_completion_upstream() (acquires an upstream session, checks the
completions capability, calls session.complete(), and translates the
result).

Three SDK-2.0-driven fixes land here: session.server_capabilities is read
as a property (not called as get_server_capabilities()); non-MCPError
transport failures are categorized via the existing, tested
_categorize_upstream_error() instead of a hand-rolled unwrap +
sanitize_exception_message call; and Completion.has_more is read via its
real snake_case Python attribute rather than hasMore, which silently
returns None on a real mcp_types.Completion instance (hasMore is a
JSON-serialization alias only, not a readable attribute).

Refs #6629

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

_complete_prompt_argument() now forwards to _forward_completion_upstream()
for a prompt with a gateway_id (via new _is_federated() helper), using
original_name so the upstream sees its own prompt name. If the upstream
does not advertise the completions capability, falls back to the
locally-synced argument_schema (enum / custom completions) instead of
raising, matching R3's local-schema fallback. Non-federated prompts are
unaffected.

Also switches this method's own local-validation raises (missing prompt
name / not found / argument not found) from the bare CompletionError to
the new CompletionInvalidParamsError subclass so callers can map them to
-32602 — CompletionInvalidParamsError still satisfies existing
`pytest.raises(CompletionError)` assertions since it subclasses it.

Refs #6629

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

…tream

_complete_resource_uri() now looks up the DbResource row owning a
uri_template match; if that row is federated (has a gateway_id), the
request is forwarded to its upstream via _forward_completion_upstream().
Plain, non-template resources are never forwarding candidates — the owner
lookup filters on uri_template IS NOT NULL, so a matching plain resource
falls straight through to the existing local-listing behavior.

Switches the local "missing URI template" raise to
CompletionInvalidParamsError (subclasses CompletionError, so existing
assertions still hold) and adds arg_name/context passthrough parameters
mirroring _complete_prompt_argument()'s Task 4 shape.

Refs #6629

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

…letion

handle_completion() previously caught every exception and re-raised a bare
CompletionError(str(e)), discarding the specific subclass Tasks 1-5 now
raise (CompletionNotSupportedError/CompletionInvalidParamsError/
CompletionInternalError). Call sites need the specific subclass to map to
the correct JSON-RPC code via completion_error_code(), so a CompletionError
is now re-raised as-is; only a non-CompletionError exception is wrapped,
and as CompletionInternalError rather than the generic base class. Also
threads the request's optional completion `context` through to both
dispatch helpers, and switches the two local validation raises here to
CompletionInvalidParamsError.

Refs #6629

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

…e on REST and /rpc

Replaces the REST endpoint's unconditional 400 and the /rpc handler's
unconditional -32602 with taxonomy-aware mapping. REST now returns 400 for
CompletionInvalidParamsError/CompletionNotSupportedError (client-error
reads) and 500 for CompletionInternalError or an unclassified
CompletionError (previously always 400 — an upstream transport failure now
surfaces as 500 instead of masquerading as a bad request); /rpc now uses
completion_error_code(exc) to reproduce the upstream's own JSON-RPC code
(-32601/-32602/-32603) instead of collapsing every completion failure to
-32602.

Updates the two pre-#6629 test_main.py assertions that pinned the old
uniform codes (bare CompletionError -> 400 REST / -32602 rpc) to the new,
intentional per-subclass behavior, and adds coverage for each subclass at
both call sites.

Refs #6629

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

…te and streamable transport

Neither call site had CompletionError-specific handling before this
(spec §8.5): the internal Rust route fell through to a generic 500 with
str(exc) echoed as data, and streamablehttp_transport.complete() swallowed
every exception (any kind) into a *successful* empty CompleteResult. Both
are new except branches, not edits to existing mappings.

The internal Rust route now returns a JSON-RPC error body carrying
completion_error_code(exc) for a CompletionError, ahead of the existing
JSONRPCError/generic-Exception branches. complete() now re-raises a
CompletionError as an MCPError with the same taxonomy code instead of
returning empty results — a deliberate, visible behavior change: a
federated-forwarding failure was previously invisible to the client
(fail-open); it now surfaces as a proper JSON-RPC error.

The PermissionError raise at streamablehttp_transport.py:2999 is
deliberately left untouched — its dual-era divergence (spec §8.6) is
cross-cutting (9 handlers in this file) and pre-dates #6629, tracked as
its own issue rather than folded in here. types.Completion(...,
hasMore=False) construction sites are also left untouched: SDK 2.0's
Completion model accepts both aliases on construction
(validate_by_alias/validate_by_name), so only *reads* need has_more.

Refs #6629

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

…asks 1-8

Coverage: adds targeted tests for branches Tasks 1-8's happy/sad-path tests
didn't individually exercise (query-param auth resolution and its decode
failure, RegistryNotInitializedError fallback, the missing-gateway/
malformed-context/no-completions-capability/no-completion-payload guards
in _forward_completion_upstream, the MCPError and generic-exception
mapping branches, nested BaseExceptionGroup unwrapping, CancelledError
passthrough, and handle_completion's non-CompletionError wrap). Raises
completion_service.py coverage from 87% to 99% (the remainder is
pre-existing initialize()/shutdown(), out of scope for this diff).

Lint: fixes two real pylint findings introduced by Tasks 7/2 (verified via
an isolated pylint install after this sandbox's uv tool cache produced
corrupted wheels for pylint/astroid) --
  - main.py: a local `status` variable in the REST completion handler
    shadowed the module-level `fastapi.status` import; renamed to
    `status_code`.
  - completion_service.py: restructured _acquire_upstream_session()'s
    registry-or-mcp_proxy_client selection from an early-return `if` into
    an `if/else`, which resolves pylint's contextmanager-generator-
    missing-cleanup false positive on the second `async with ... yield`
    branch (both branches already clean up correctly via the standard
    async-context-manager protocol); the remaining false positive on the
    surviving branch is a scoped, justified `# pylint: disable=`.
completion_service.py and main.py now both score 10.00/10 under this
repo's pinned pylint config. streamablehttp_transport.py's 5 pre-existing
E1101 "ContextForgeMCPServer has no request_context member" findings are
unrelated to this diff (confirmed present, unchanged, on this branch's
pre-#6629 tip) and left untouched.

Refs #6629

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

Builds mcp-servers/python/completion_test_server fresh against this
branch's SDK (mcp>=2.0.0, mcp-types>=2.0.0) per spec §8.1 -- the sibling
worktree-agent-a763924ede37c48ff fixture targets the superseded main-based
line and pins mcp<2 (mcp.server.fastmcp.FastMCP, gone on this SDK
generation), so it is not reused. Verified against this SDK's actual
mcp.server.mcpserver.MCPServer.completion() implementation (handler
signature (ref, argument, context) -> Completion, same shape as 1.x) rather
than assumed. A COMPLETIONS_ENABLED=false toggle lets the same image run
without registering a completion handler at all, simulating an upstream
that lacks the capability.

Wires both variants into docker-compose.yml's `testing` profile
(completion_test_server, completion_test_server_no_completions on ports
9102/9103) alongside the existing fixtures. Registration and prompt
federation happen inside the test itself (tests/live_gateway/mcp/
test_completion_federation.py), not via a separate one-shot registration
container.

test_completion_federation.py proves R1-R6 end-to-end against a real
running gateway + fixture server: a federated prompt's completion/complete
is answered by the live upstream call (asserting a value only the upstream
knows, since the synced argument_schema has no enum for `style` at all --
this could not pass via a stale-schema fallback), and a federated prompt
owned by an upstream without the `completions` capability maps to -32601
over /rpc. R7 is not implemented on this branch (plan Task 9, withdrawn
per spec §5.4) so there is no gate for this test to exercise.

Verified locally: `make setup` + `docker compose --profile testing up -d
gateway nginx completion_test_server completion_test_server_no_completions`
(reusing an existing local mcpgateway/mcpgateway:latest image rather than
make docker-nuke docker-prod-rust, which is destructive to the whole
Docker daemon and not scoped to this project) brought up a healthy stack,
and both tests in this file passed against it.

Note for whoever brings up the full `testing-up` pre-merge gate: the
`migration` one-shot container in docker-compose.yml does not pass
DEFAULT_USER_PASSWORD through, so mcpgateway.bootstrap_db falls back to
the code-level "changeme" placeholder, which this branch's
SecurityConfigurationError now rejects outright -- migration exits 1
before the gateway ever starts. This reproduces on an unmodified
docker-compose.yml and is unrelated to #6629; left unfixed here as
out-of-scope for this PR.

Refs #6629

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

make detect-secrets-scan flagged 5 unaudited "Secret Keyword" findings on
the current tree: 3 false positives in fake test credentials this PR's own
test_completion_service.py added (query-param auth test fixtures using
literal "api_key"/"secret-value" placeholders), plus 2 pre-existing false
positives in the carried-forward spec/research docs (prose mentioning
auth-related identifiers, e.g. "AUTH_ENCRYPTION_SECRET",
"streamablehttp_client" near an "auth=" kwarg) that had never been audited.

Suppresses the three in test_completion_service.py inline with
`# pragma: allowlist secret` per CLAUDE.md's guidance for Python files, and
audits the two doc findings directly in .secrets.baseline (is_secret:
false) since the pragma convention doesn't apply to prose documentation.
The rest of this diff is line-number drift for already-audited entries in
files this PR touches (main.py, docker-compose.yml, test_main.py) --
confirmed via a before/after diff of (file, hash, is_secret) tuples: 4
entries added, all audited false, 0 removed.

Refs #6629

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

… test

Ruff flagged `lambda: _FakeRegistry()` as an unnecessary lambda when
patching get_upstream_session_registry -- the fake class itself is already
a zero-arg callable returning an instance, so the wrapper added nothing.
Only new ruff finding on this branch's set of touched/pre-existing files
after this PR (62 baseline -> 63, the remainder being this file's own
pre-existing `import mcp_types as mcp_types` house style, left as-is).

Refs #6629

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>

… docs

The docs/superpowers/ spec, plan, and research files this PR carried
forward have been removed from git history (kept locally, untracked, per
request -- they aren't meant to ship as part of this feature's tracked
source). Their two .secrets.baseline entries were left behind after the
rebase that dropped the commit adding those files; both referenced
already-audited false positives (is_secret: false) with nothing left in
the tree for them to describe, so they're removed rather than left
dangling.

Refs #6629

Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>