SEP-2028: Automatic _meta to HTTP header forwarding for distributed tracing#2028
SEP-2028: Automatic _meta to HTTP header forwarding for distributed tracing#2028monahk wants to merge 28 commits into
Conversation
2e44019 to
049dfd8
Compare
049dfd8 to
3f21fdc
Compare
ea7a2a2 to
3d94ec6
Compare
|
One topic area I note that is missing from the proposal: expected error handling behavior in the event that there are multiple contexts present (both in headers and from |
|
Great catch, Michael! This is an important edge case that the proposal should address explicitly. Here's my thinking on the conflict resolution behavior: Recommended Priority Order (highest to lowest):
Rationale: The For example:
Proposed Addition to Spec: ### 6. Conflict Resolution
When `_meta` contains trace context fields and the outbound HTTP request
already has corresponding headers set (e.g., via auto-instrumentation):
1. `_meta` values MUST take precedence and override existing headers
2. SDKs SHOULD log a debug-level message when overriding existing headers
3. No error should be raised—this is expected behavior when integrating
with existing observability infrastructureAlternative Considered: Raising an error or warning seems too disruptive—many servers will have OpenTelemetry or similar instrumentation already running, and we want this to "just work" by propagating the client's context. Does this approach make sense? Happy to add this section to the SEP. |
Addresses @michaelsafyan's question about handling conflicts when trace context exists in both _meta and pre-existing HTTP headers. Specifies that _meta values take precedence to ensure trace continuity. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
|
|
||
| When `_meta` contains trace context fields and the outbound HTTP request already has corresponding headers set (e.g., via auto-instrumentation): | ||
|
|
||
| 1. `_meta` values MUST take precedence and override existing headers |
There was a problem hiding this comment.
Thanks for this update!
Would suggest one other clarification / exposition here: the situation where not all headers are in both but only some of them are, such as where traceparent is not in _meta but tracestate is in both headers and in _meta or where traceparent is in _meta but baggage is only in headers.
Recommended behavior:
- If
traceparentis both present and valid inparams._meta:- Discard
traceparent,tracestate,baggagefrom headers - Use only the subset of valid values (
traceparent,tracestate,baggage) fromparams._meta
- Discard
- Otherwise:
- Use
traceparent,tracestate,baggagefrom headers
- Use
Agreed, generally, that _meta should override. But if there is a mix, there can be connections between them (e.g. tracestate may assume a particular trace ID implicitly from traceparent), and so I think it should be an all-or-nothing replacement.
There was a problem hiding this comment.
Great refinement, @michaelsafyan! The "all-or-nothing" approach makes a lot of sense.
You're right that traceparent, tracestate, and baggage are semantically linked—mixing traceparent from _meta with tracestate from auto-instrumentation would create an inconsistent/invalid trace context.
I've updated Section 6 to adopt this approach:
### 6. Conflict Resolution
When `_meta` contains W3C Trace Context fields and the outbound HTTP request already has corresponding headers set (e.g., via auto-instrumentation):
1. If `traceparent` is present and valid in `_meta`, SDKs MUST use an **all-or-nothing** approach:
- Discard ALL existing W3C Trace Context headers (`traceparent`, `tracestate`, `baggage`) from the outbound request
- Forward ONLY the W3C fields present in `_meta`
2. SDKs SHOULD log a debug-level message when overriding existing headers
3. No error should be raised—this is expected behavior when integrating with existing observability infrastructure
**Rationale**: W3C Trace Context fields are semantically linked—`tracestate` contains vendor-specific data keyed to the `traceparent`, and `baggage` may contain context relevant to that specific trace. Mixing fields from different sources would create inconsistent trace context. The all-or-nothing approach ensures trace coherence across the MCP boundary.This ensures that when _meta provides trace context, it's used as a complete, coherent unit rather than being merged with potentially incompatible auto-instrumented headers.
Pushing the update now.
Per @michaelsafyan's feedback: when traceparent is present in _meta, discard ALL existing W3C headers and use only _meta values. This prevents semantic inconsistencies between linked trace context fields. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
|
@monahk this SEP does not have an assigned sponsor. Are you looking to continue working on it and find sponsorship? |
@localden Thanks for checking in! Yes, I'm actively working on this and would love to find a sponsor. Would you be interested in sponsoring this SEP, or could you recommend someone who might be a good fit given the distributed tracing/observability focus? |
@localden any thoughts on who might be a good sponsor for this SEP? |
305dadf to
e794a44
Compare
Major redesign based on community feedback: - Replace hardcoded W3C header handling with extensible group-based system - Separate trace-context and baggage into independent groups (per @lmolkova) - Single 'policy' setting per group: clear-and-use-meta, merge-with-meta, ignore-meta - Group integrity: SDK never mixes headers from different sources within a group - Support for user-defined custom groups (Datadog, Jaeger, internal headers) - Aligns with OpenTelemetry MCP context propagation guidance Addresses feedback from @michaelsafyan and @lmolkova on PR modelcontextprotocol#2028.
| | `clear-and-use-meta` | Yes (any in group) | Yes | Clear ALL existing in group, forward ALL from `_meta` | | ||
| | `clear-and-use-meta` | Yes (any in group) | No | Forward from `_meta` | | ||
| | `clear-and-use-meta` | No | Yes | Keep existing | | ||
| | `merge-with-meta` | Yes | Yes | Overwrite that specific header with `_meta` value | |
There was a problem hiding this comment.
"Overwrite that specific header" is a surprising behavior given the name "merge".
I would expect "key: value1" and "key: value2" to merge into "key: value1,value2" following standard HTTP behavior for repeating a header line with different values:
https://httpwg.org/http-core/draft-ietf-httpbis-semantics-latest.html#field.lines
There was a problem hiding this comment.
Thanks for the thorough review @michaelsafyan! Addressing your points:
1. Validation Order
You're right that validation must precede conflict resolution. Section 3 states "SDKs MUST validate _meta fields before forwarding" — I'll make the ordering more explicit in the processing flow.
2. tracestate without traceparent
Great catch! The current wording has a gap. Rather than hardcoding this for trace-context only, I'll add a generic required attribute to header group configuration:
headerGroups: {
// Predefined group - traceparent is required, tracestate is optional
"trace-context": {
headers: ["traceparent", "tracestate"],
policy: "clear-and-use-meta",
required: ["traceparent"] // Group skipped if traceparent missing
},
// Custom group example - can specify their own required headers
"datadog": {
headers: ["x-datadog-trace-id", "x-datadog-parent-id", "x-datadog-sampling-priority"],
policy: "clear-and-use-meta",
required: ["x-datadog-trace-id"]
}
}Behavior with required: ["traceparent"]:
_meta contains |
Result |
|---|---|
traceparent + tracestate |
Forward both |
traceparent only |
Forward traceparent only; discard any existing tracestate |
tracestate only (no traceparent) |
Skip group entirely; preserve existing headers |
| Neither | Skip group; preserve existing headers |
This correctly models W3C Trace Context semantics (tracestate is optional but requires traceparent) while allowing custom groups to define their own requirements.
3. "merge" Terminology
Agreed that "merge" is misleading given HTTP semantics. I'll rename to prefer-meta — clearer that _meta takes precedence when present, without implying value concatenation.
Updated policies:
clear-and-use-meta— All-or-nothing replacement for coupled headersprefer-meta(néemerge-with-meta) — Per-header:_metawins if present, else keep existingignore-meta— Always keep existing, never forward from_meta
4. Vendor Implementations
This is spec-first — no production implementations yet. The reference implementation will be in the TypeScript SDK after acceptance. However, the design aligns with:
- OpenTelemetry's MCP context propagation guidance
- Standard W3C Trace Context propagation patterns
Happy to coordinate with other SDK maintainers once the spec direction is agreed.
I'll update the SEP with these changes. Let me know if you have any concerns with the required array approach!
There was a problem hiding this comment.
I guess I'm confused regarding "clear-and-use-meta" vs "prefer-meta" ... couldn't one be implemented in terms of the other based on listing a single header at a time (groups of 1 header) vs multiple headers?
The "required" policy makes sense. A "validator" option may also be necessary.
In terms of vendor implementations, that is assuming that all clients are open source. What I'm saying is that I know of at least one vendor implementation that does what was proposed in my earlier comment . In particular, an all-or-nothing keyed on traceparent for traceparent, tracestate, and baggage. (Yes, baggage is a separate spec and can be carried independent of W3C tracing context; however, this vendor solution considers baggage as logically similar to trace context propagation and often confused by users interchangeable with tracestate as a carrier of extra data piggybacking over tracing instrumentation). That doesn't mean that it has to land that way in OSS, but it may be worth some ground truth research to inform the spec (and to determine where divergences from private vendor solutions may hinder adoptability or create friction points).
Thanks so much for listening and for driving this work forward!
There was a problem hiding this comment.
Good points @michaelsafyan!
On policy distinction: The difference is meaningful for multi-header groups:
clear-and-use-meta: If_metahastraceparentbut nottracestate, existingtracestateis cleared (prevents orphaned state)prefer-meta: Existingtracestatewould be preserved (could cause trace coherence issues)
For single-header groups like baggage, they behave identically.
On validator: I like this idea. The required array handles presence checks, but a validator function could handle format validation:
"trace-context": {
headers: ["traceparent", "tracestate"],
policy: "clear-and-use-meta",
required: ["traceparent"],
validator: (headers) => isValidTraceparent(headers.traceparent)
}Should I add this to the spec, or keep it as an SDK implementation detail?
On vendor research: This is a great point. You seem to have good visibility into existing implementations — would you be interested in contributing this research? Specifically:
- Documenting how existing OSS propagators (OpenTelemetry, Jaeger, Zipkin) handle W3C Trace Context
- Identifying any proprietary implementations that treat
traceparent/tracestate/baggageas a coupled unit - Flagging potential friction points between this spec and existing approaches
If you'd like to contribute, I'd be happy to add you as a co-author on the SEP. Otherwise, I can do the research — just wanted to offer since you raised the concern and may already have context here.
|
|
||
| 4. **Simple mental model**: Single `policy` setting per group, 3 clear policies, no ambiguity. | ||
|
|
||
| ### Why Default-On for W3C Trace Context? |
There was a problem hiding this comment.
+1 to making it default for tracing.
michaelsafyan
left a comment
There was a problem hiding this comment.
Thanks for the changes!
In terms of vendor review, I only have insight into the one. Left one more comment which I think would at least make that behavior implementable in terms of this proposal even if not exactly aligned on the default configuration.
Other than that, LGTM.
Thank you so much for driving this forward!
| "trace-context": { | ||
| policy: "clear-and-use-meta", | ||
| required: ["traceparent"], | ||
| validator: (headers) => isValidTraceparent(headers.traceparent), |
There was a problem hiding this comment.
Suggestion: allow separate validators per header as in:
validators: {
# Will drop/scrub/ignore if invalid
"traceparent": isValidTraceparent,
# Will drop/scrub/ignore if invalid
"tracestate": isValidTracestate,
}
# Presence evaluated after validation
required: ["traceparent"],
There was a problem hiding this comment.
Thanks for the review and the suggestion!
Both changes incorporated:
- Per-header validators - replaced the group-level
validatorwithvalidators(per-header), so individual headers can be validated/dropped independently:
"trace-context": {
policy: "clear-and-use-meta",
validators: {
traceparent: isValidTraceparent,
tracestate: isValidTracestate,
},
required: ["traceparent"],
},-
Processing order updated - per-header validation now runs before required presence checks:
- Validation (field-level: string type, ASCII, size limits)
- Per-header validation (drop individual headers that fail)
- Required check (if a required header was dropped by validation, group is skipped)
- Policy application
- Changed `validator` to `validators` for per-header validation - Reordered processing: per-header validation before required check - Added rationale for validation ordering
|
Looking for a sponsor to move this SEP forward. Tagging a couple of folks whose work this directly touches: @felixweinberger - This defines new SDK behavior for the TypeScript and Python SDKs (automatic @jonathanhefner - Given your work on the Transports Interest Group and Reference Servers, this sits right at the transport boundary — how Would either of you be interested in sponsoring? |
Resolve conflicts from upstream moving docs/community/seps/ to docs/seps/ and add SEP-2028 to new navigation and index.
35f287f to
e6c9d7d
Compare
|
@localden any suggestions on how i can find a sponsor for this PR? |
…r-forwarding # Conflicts: # docs/seps/index.mdx
|
Posted a status update in Discussion #2029 covering the current design (the group-based Also worth flagging: VS Code / Copilot ran into this exact gap in microsoft/vscode#302301 (shipped a The one thing still blocking this SEP from advancing is an assigned sponsor. If you're a maintainer working on transports, SDK behavior, or observability and this is interesting, I'd love to talk. |
…r-forwarding # Conflicts: # docs/docs.json # docs/seps/index.mdx
Summary
This SEP proposes automatic forwarding of W3C Trace Context fields (
traceparent,tracestate,baggage) from_metato downstream HTTP headers, enabling distributed tracing across the MCP boundary without requiring MCP server modifications.Motivation and Context
Currently,
_metareaches the MCP server but is not forwarded to downstream HTTP APIs, breaking end-to-end observability. This was verified by intercepting HTTP traffic from an unmodified MCP server (@timlukahorstmann/mcp-weather) - no trace headers were received.This SEP provides the protocol-level foundation that was explicitly requested when PR #666 (typescript-sdk) was rejected with the rationale: "This PR introduces transport-specific concerns (HTTP headers) into the MCP SDK without protocol-level support."
Related:
_metaconventionHow Has This Been Tested?
Proof of concept verified the gap exists:
_metawithtraceparentandcorrelation_id@timlukahorstmann/mcp-weatherMCP serverBreaking Changes
This introduces a behavioral change: W3C Trace Context headers will be forwarded by default. Servers can opt-out via
forwardTraceContext: false. Downstream APIs will ignore unknown headers per HTTP spec.Types of changes
Checklist
Additional context
Key design decisions:
traceparent,tracestate,baggage): Forwarded by default (opt-out) - matches OpenTelemetry auto-instrumentation behavior