What should a stdio MCP validator check beyond initialize and tools/list? #2733

1Utkarsh1 · 2026-05-16T13:46:11Z

1Utkarsh1
May 16, 2026

I built a small CLI called mcp-stdio-guard to catch MCP stdio server failures before users wire a server into Claude/Cursor/etc.

Repo: https://github.com/1Utkarsh1/mcp-stdio-guard
Pilot notes: https://gist.github.com/1Utkarsh1/90d7a0f2ea85635e3a209eb1be4970f9

The current guard does a real initialize handshake, can send a post-initialize request like tools/list, fails on stdout pollution, invalid JSON-RPC frames, crashes, and missing responses, while allowing stderr diagnostics.

The JSON output now has schemaVersion: 1 and badge-friendly checks.* classes for:

initialize
stdout
jsonRpc
operation
process
pythonBuffering
staticScan
repeat

I also ran a small registry-style pilot against 30 real stdio MCP entries from public MCP indexes and upstream package metadata/docs. 21/30 passed initialize + tools/list twice. The 9 failures were all pre-initialize process exits from yanked/superseded packages, not stdout pollution or JSON-RPC framing issues.

I would love feedback from MCP server authors and client/registry maintainers:

What other stdio/server-health checks would be useful before listing or recommending a server?
Should install/runtime failures be separated more explicitly from protocol failures in the JSON schema?
Are there edge cases around Python buffering, stderr logging, or repeated cold/warm runs that this should handle differently?

My goal is not to shame server authors; it is to make failures visible and actionable before users lose time debugging a broken stdio setup.

yudin-s · 2026-05-16T19:39:18Z

yudin-s
May 16, 2026

I would separate the result model into three buckets:

install/runtime health
stdio transport hygiene
MCP protocol conformance

That distinction is worth keeping in the JSON schema. A package that cannot start is different from a server that starts but violates JSON-RPC/MCP. Registries and clients will want to display those differently.

Checks I would add beyond initialize and tools/list:

Lifecycle
- initialize response has protocolVersion, capabilities, and serverInfo
- client sends initialized notification after initialize
- server does not require a request before initialize except valid JSON-RPC errors
- repeated cold starts produce equivalent capabilities/tool lists

JSON-RPC framing
- stdout contains only complete JSON-RPC messages
- response id matches request id
- notifications do not receive responses
- invalid method returns a JSON-RPC error, not a process crash
- malformed JSON is handled predictably

Tool schema quality
- every tool has a stable name and description
- inputSchema is valid JSON Schema
- required properties are valid for the schema
- calling a tool with invalid arguments returns a structured error
- calling a harmless/sample tool, if configured, does not corrupt stdio framing

Capability honesty
- if resources/prompts are not advertised, related list calls should not pretend to work
- if resources/prompts are advertised, resources/list or prompts/list should pass

Operational behavior
- timeout classification: startup timeout vs request timeout vs clean exit
- stderr diagnostics allowed, stdout diagnostics fail
- exit code and signal captured
- optional second run to detect one-time init side effects

For Python buffering, I would report it as a runtime advisory rather than a protocol failure unless it causes an actual timeout or missing response. Something like:

{
  "class": "runtime.pythonBuffering",
  "severity": "warning",
  "evidence": "server produced no stdout before timeout; retry with PYTHONUNBUFFERED=1 changed behavior"
}

The other useful registry-facing check is a reproducibility fingerprint: command, args, cwd behavior, env vars used/redacted, package version, node/python version, and elapsed startup time. Without that, a pass/fail result is hard to debug when the same server works on the author's machine but fails in a client registry.

0 replies

PengSpirit · 2026-05-27T01:54:52Z

PengSpirit
May 27, 2026

The three-bucket separation yudin-s sketched (install/runtime, transport hygiene, protocol conformance) is the right top-level cut. The bucket that keeps getting under-served in practice is a fourth one: schema quality. It's not a runtime failure and it's not a protocol violation — the server happily passes initialize + tools/list — but it's the reason "agent connects but won't call my tool" reports outnumber actual protocol bugs by a wide margin.

Checks that belong in that fourth bucket, in rough order of how often they bite:

Tool schema quality (post-protocol-conformance)
- every tool has a non-empty description
- description is not just a noun phrase ("Search files") — has a verb + scope
- description includes anti-purpose (what the tool is NOT for) — the single biggest disambiguator for wrong-tool-selection
- every parameter has a description (the most common cause of "model guesses values")
- string parameters with finite value sets have enum constraints
- description ≤ 1024 chars (OpenAI hard cap; tightest of major providers)
- annotations populated (readOnlyHint, destructiveHint, idempotentHint, openWorldHint)
- tool count under client trim thresholds (some clients silently truncate after N tools)

The "annotations honesty" angle yudin-s mentioned for resources/prompts has a direct parallel for tools: if destructiveHint: false is set on a tool whose name starts with delete_ or drop_, that's a contradiction worth flagging — careful clients use those hints to gate consent prompts.

On reproducibility fingerprint — strong yes, and worth including the protocol version the server actually negotiated (server can downgrade), not just the version it claims to support. A server that advertises 2026-03-26 in package metadata but negotiates 2025-11-25 at runtime is the most common cause of "spec-says-this-should-work-but-doesn't" reports.

For separation in the JSON schema (Q2): yes, and the wire format gain is that registry consumers can filter on "passes protocol but has schema warnings" — that's the cohort that needs the most help and currently gets bucketed with "fully broken" because the failure mode is invisible without a schema-level check.

On Python buffering (Q3): treating it as a runtime advisory unless it causes a timeout is the right call. The trickier case is servers that emit partial JSON-RPC frames because of buffering interacting with print() calls in the wrong order — that one looks like a JSON-RPC framing failure but the root cause is buffering, and a guard that classifies it as "framing" will get a fix that doesn't address the cause. If mcp-stdio-guard can detect "stdout produced incomplete frames AND PYTHONUNBUFFERED=1 changes behavior," that's the high-signal advisory.

One more from the repeated-run side: schema fingerprint stability across runs. Re-running tools/list should return the same tool count, same names, same parameter shapes. Servers that lazy-register tools post-handshake (or whose schemas vary by env) break clients that cache the tool list. Flagging "tool count differs between cold runs" catches a class of bug that protocol-only checks miss.

For folks running adjacent tooling — mcp-stdio-guard is the strongest framing-and-process-health check I've seen; npx @incultnitollc/mcp-probe test "<launch command>" covers the schema-quality bucket above (descriptions, params, anti-purpose, length caps) and outputs a scorecard you can fail a build on; Anthropic's MCP Inspector is still the right tool for interactive click-through. Three different stages, three different failure surfaces. The more of them are run pre-publish, the fewer "Inspector says fine but my agent ignores it" reports get filed downstream.

0 replies

Zawwarsami16 · 2026-05-27T21:12:40Z

Zawwarsami16
May 27, 2026

Strong cuts upthread from @yudin-s on the three-bucket separation and @PengSpirit on schema quality as the fourth bucket. The gap I'd flag — and the one we kept tripping over running stdio MCP servers in production for the last few months — is a fifth bucket: live-call behavior. Everything above passes against initialize + tools/list + (maybe) one call, but the bugs that actually take down agent integrations show up only when the server is being driven, not introspected.

Concrete checks that belong in this bucket, in rough order of severity:

Live-call behavior (post-handshake, under realistic call patterns)

Error envelope conformance
- Invalid arguments produce JSON-RPC error with isError:true OR a structured tool result
  with content[0].type="text" and the error reason — not a process crash, not a hung response
- Server-side exceptions (5xx-equivalent) are caught and surfaced; "no response" is the worst
  failure mode for clients because they cannot distinguish a slow tool from a dead server
- Error responses include enough context to debug (which tool, which argument, what was expected)

Input validation honesty
- Calling a tool with arguments that violate the declared inputSchema should fail explicitly,
  not silently accept-and-proceed with garbage. The most common failure here is "the schema says
  the field is required but the handler treats it as optional" and vice versa.
- enum constraints are enforced at the server side, not just declared in the schema

Cancellation behavior
- $/cancelRequest on an in-flight tool call actually stops the work or returns promptly
- The server does not respond to the cancelled request after the cancel notification
- Concurrent in-flight requests are not all cancelled when one is cancelled

Progress reporting (for tools that declare long-running work)
- If notifications/progress is advertised in capabilities, long tool calls actually emit them
- Progress tokens from the client are echoed back correctly (so client can correlate)
- Final response still arrives even if no progress notifications were sent

Concurrency
- Two concurrent tool calls do not serialize behind each other unless explicitly required
- Two concurrent calls with different request ids return correctly-matched responses
  (the most common bug: a server with a single I/O queue mixing up which response goes to which id)
- Side effects do not leak between concurrent calls (shared mutable state caught here)

Idempotency-claim verification
- If a tool declares `idempotentHint: true`, calling it twice with identical arguments
  produces equivalent results (state, return value)
- This catches the common case of a tool author setting the hint aspirationally
  without checking it's actually true under the implementation

Timeout classification (refining @yudin-s's point)
- Distinguish: startup timeout, request-level timeout, idle-after-progress timeout,
  cancel-acknowledgment timeout. These have different root causes and different fixes.

Observability hooks (the audit-context angle)
- If the server emits any logging/tracing, calls produced under the validator should be
  identifiable in those logs (correlation id, structured trace). This is the check that
  catches "the server says it logs everything but the validator's call doesn't show up,"
  which is the audit-trail bug that bites later in an incident review.

On the JSON schema separation question (Q2): yes, and beyond install/runtime/transport/protocol/schema-quality, I'd recommend a liveCall class with sub-checks because consumers will want to filter "passes handshake but breaks under load" separately from schema-quality warnings. A registry can ship a server that has perfect tool descriptions but crashes on the second concurrent call; those failures need different remediation.

One protocol-version note worth folding in: when the server downgrades its negotiated protocolVersion from what its package metadata advertises, liveCall checks should be run against the negotiated version, not the advertised one. A tool that's optional at the older version but required at the newer one will pass the static-schema check and fail the live-call check, and the version-downgrade is the explanation.

The reproducibility fingerprint @yudin-s mentioned is the right anchor — extending it with the negotiated protocol version, server-side concurrency limit (if discoverable), and a hash over the tool list (so re-runs flag schema drift between cold starts) covers most of the "works on my machine" reports we saw.

Happy to PR any of this into mcp-stdio-guard if it's useful — the live-call checks reuse a lot of the existing transport-hygiene infrastructure you already have, mostly extending the JSON-RPC frame parser to track in-flight request ids and timing.

1 reply

PengSpirit May 28, 2026

Zawwarsami16's liveCall bucket is the right fifth cut. A few threads worth pulling on.

The schema ↔ liveCall pairing
A lot of bucket-4 items are claims the server makes about itself; the corresponding bucket-5 check is whether the server honors that claim under a real call. Pairing them up tightens the audit chain:

If inputSchema.required: ["x"] is declared, a call without x should produce an explicit error, not a silent default.
If a parameter has enum: [...], a call with an off-enum value should be rejected, not coerced.
If annotations.idempotentHint: true is set, calling twice with the same args should produce equivalent observable state.
If annotations.readOnlyHint: true is set, the call should mutate no resource the server exposes.
If annotations.destructiveHint: false is set, the tool's name shouldn't imply destruction and no side effects should be observed at runtime.
If a tool description claims a purpose, the result should align with that stated purpose (drift advisory, not strict check).

The last row won't survive a strict formal check — description→behavior is fuzzy — but a "tool was invoked with args matching its stated purpose and returned content unrelated to that purpose" advisory would catch a real class of drift bug (descriptions that haven't kept up with handler edits).

Error envelope: the spec ambiguity is the problem, not either pattern

The spec permits both JSON-RPC error response and tool result with isError:true / content[0].type:"text" as failure signals. Clients handle these differently. A validator probably shouldn't pick a winner — but it should flag inconsistency:

a server that returns JSON-RPC errors for invalid args but content-as-error for runtime failures is harder for a client to wrap than one that picks one pattern and sticks with it
- a server that returns content-as-error without isError:true is the worst case — looks like success to any non-MCP-aware wrapper in the chain
  "Error-signaling consistency across N tool calls" as a bucket-5 check costs little and catches a lot.

Concurrency: the request-id correlation bug is underdiagnosed

The "single I/O queue mixes up which response goes to which id" bug is one of those failures whose only in-the-wild signal is "agent occasionally returns the wrong answer," and almost nobody bisects it to the server. A check that fires N concurrent calls with distinct ids and verifies (a) all return, (b) each response's id matches a request, (c) each response's content matches what that specific request asked for — would catch it cheaply. The third condition is what distinguishes wrong-response-routing from races on shared state. Both fail the test but for different reasons, and the remediation diverges.

Negotiated-version anchor

This is the sharpest single point in the thread. Static check against package.json's advertised protocol version + live check against the negotiated version + a comparison flag is the closest thing to "did the server actually do what it claimed it could." Most "spec says this should work but doesn't" reports I've seen trace to that gap.

Scope reality

Candidly on current tooling: bucket 4 (schema quality) is what mcp-probe covers end-to-end today. Bucket 5 (live-call behavior) is mostly uncovered by anything I've seen in the OSS ecosystem, probe included. Zawwarsami16 PR'ing live-call checks into mcp-stdio-guard is the right home for that work — the JSON-RPC frame parser and process-lifecycle infrastructure there is the natural foundation, and keeping framing+process+live-call in one binary avoids the "three tools each find a third of the bugs" problem.

If mcp-stdio-guard lands a liveCall scorecard, wiring probe to surface its findings inline is straightforward on our side — so registry consumers and CI pipelines can get one composite pass/fail across all five buckets without orchestrating three runners.

The reproducibility fingerprint extended with negotiated protocol version, tool-list hash, and (where discoverable) server-side concurrency limit is the right anchor. If the spec repo wants to ship a reference fingerprint format, that's the artifact registries and CI configs will actually consume.

gustavo-sec · 2026-06-01T19:16:06Z

gustavo-sec
Jun 1, 2026

Hi, we at Gated have a catalog of 200+ checks across 5 families. Some of them are http specific, but most of them might be able to help you.
Our catalog is open: https://gated.cc/docs/catalog/security

0 replies

This comment was marked as spam.

Sign in to view

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

What should a stdio MCP validator check beyond initialize and tools/list? #2733

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 5 comments 1 reply

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

This comment was marked as spam.

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

What should a stdio MCP validator check beyond initialize and tools/list? #2733

Uh oh!

Uh oh!

1Utkarsh1 May 16, 2026

Replies: 5 comments · 1 reply

Uh oh!

yudin-s May 16, 2026

Uh oh!

PengSpirit May 27, 2026

Uh oh!

Zawwarsami16 May 27, 2026

Uh oh!

PengSpirit May 28, 2026

This comment was marked as spam.

Uh oh!

gustavo-sec Jun 1, 2026

1Utkarsh1
May 16, 2026

Replies: 5 comments 1 reply

yudin-s
May 16, 2026

PengSpirit
May 27, 2026

Zawwarsami16
May 27, 2026

gustavo-sec
Jun 1, 2026