Skip to content

SEP-2028: Automatic _meta to HTTP header forwarding for distributed tracing#2028

Open
monahk wants to merge 28 commits into
modelcontextprotocol:mainfrom
monahk:sep-meta-http-header-forwarding
Open

SEP-2028: Automatic _meta to HTTP header forwarding for distributed tracing#2028
monahk wants to merge 28 commits into
modelcontextprotocol:mainfrom
monahk:sep-meta-http-header-forwarding

Conversation

@monahk

@monahk monahk commented Dec 29, 2025

Copy link
Copy Markdown

Summary

This SEP proposes automatic forwarding of W3C Trace Context fields (traceparent, tracestate, baggage) from _meta to downstream HTTP headers, enabling distributed tracing across the MCP boundary without requiring MCP server modifications.

Motivation and Context

Currently, _meta reaches the MCP server but is not forwarded to downstream HTTP APIs, breaking end-to-end observability. This was verified by intercepting HTTP traffic from an unmodified MCP server (@timlukahorstmann/mcp-weather) - no trace headers were received.

This SEP provides the protocol-level foundation that was explicitly requested when PR #666 (typescript-sdk) was rejected with the rationale: "This PR introduces transport-specific concerns (HTTP headers) into the MCP SDK without protocol-level support."

Related:

How Has This Been Tested?

Proof of concept verified the gap exists:

  • MCP client sending _meta with traceparent and correlation_id
  • Unmodified @timlukahorstmann/mcp-weather MCP server
  • Mock HTTPS server intercepting AccuWeather API calls
  • Result: No trace headers received at mock server

Breaking Changes

This introduces a behavioral change: W3C Trace Context headers will be forwarded by default. Servers can opt-out via forwardTraceContext: false. Downstream APIs will ignore unknown headers per HTTP spec.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update

Checklist

  • I have read the MCP Documentation
  • My code follows the repository's style guidelines
  • New and existing tests pass locally
  • I have added appropriate error handling
  • I have added or updated documentation as needed

Additional context

Key design decisions:

  • W3C Trace Context (traceparent, tracestate, baggage): Forwarded by default (opt-out) - matches OpenTelemetry auto-instrumentation behavior
  • Custom fields: Require explicit opt-in with whitelist - prevents accidental data leakage
  • Includes security mitigations for header injection (validation, size limits)

@monahk monahk force-pushed the sep-meta-http-header-forwarding branch from 2e44019 to 049dfd8 Compare December 29, 2025 21:35
@monahk monahk force-pushed the sep-meta-http-header-forwarding branch from 049dfd8 to 3f21fdc Compare December 29, 2025 21:44
@monahk monahk force-pushed the sep-meta-http-header-forwarding branch from ea7a2a2 to 3d94ec6 Compare December 29, 2025 21:47
@jonathanhefner jonathanhefner changed the title SEP: Automatic _meta to HTTP header forwarding for distributed tracing SEP-2028: Automatic _meta to HTTP header forwarding for distributed tracing Dec 31, 2025
@michaelsafyan

Copy link
Copy Markdown

One topic area I note that is missing from the proposal: expected error handling behavior in the event that there are multiple contexts present (both in headers and from params._meta)... any thoughts on what the desired or correct error handling behavior should be in such a case?

@ghost

ghost commented Jan 12, 2026

Copy link
Copy Markdown

Great catch, Michael! This is an important edge case that the proposal should address explicitly.

Here's my thinking on the conflict resolution behavior:

Recommended Priority Order (highest to lowest):

  1. params._meta (MCP-level context) takes precedence
  2. Existing HTTP headers on the outbound request are overwritten

Rationale:

The _meta context represents the current trace context as understood by the MCP client/host, which has visibility into the full distributed trace. If an MCP server's HTTP client library (or auto-instrumentation) has already set trace headers, those represent the server's local context, not the end-to-end trace the client is trying to propagate.

For example:

  • Client sends traceparent in _meta representing the user's request trace
  • Server has OpenTelemetry auto-instrumentation that creates a new trace for its HTTP calls
  • Without override, the client's trace context is lost and traces break at the MCP boundary (the exact problem we're solving)

Proposed Addition to Spec:

### 6. Conflict Resolution

When `_meta` contains trace context fields and the outbound HTTP request
already has corresponding headers set (e.g., via auto-instrumentation):

1. `_meta` values MUST take precedence and override existing headers
2. SDKs SHOULD log a debug-level message when overriding existing headers
3. No error should be raised—this is expected behavior when integrating
   with existing observability infrastructure

Alternative Considered: Raising an error or warning seems too disruptive—many servers will have OpenTelemetry or similar instrumentation already running, and we want this to "just work" by propagating the client's context.

Does this approach make sense? Happy to add this section to the SEP.

Addresses @michaelsafyan's question about handling conflicts when trace
context exists in both _meta and pre-existing HTTP headers. Specifies
that _meta values take precedence to ensure trace continuity.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

When `_meta` contains trace context fields and the outbound HTTP request already has corresponding headers set (e.g., via auto-instrumentation):

1. `_meta` values MUST take precedence and override existing headers

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this update!

Would suggest one other clarification / exposition here: the situation where not all headers are in both but only some of them are, such as where traceparent is not in _meta but tracestate is in both headers and in _meta or where traceparent is in _meta but baggage is only in headers.

Recommended behavior:

  • If traceparent is both present and valid in params._meta:
    • Discard traceparent, tracestate, baggage from headers
    • Use only the subset of valid values (traceparent, tracestate, baggage) from params._meta
  • Otherwise:
    • Use traceparent, tracestate, baggage from headers

Agreed, generally, that _meta should override. But if there is a mix, there can be connections between them (e.g. tracestate may assume a particular trace ID implicitly from traceparent), and so I think it should be an all-or-nothing replacement.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great refinement, @michaelsafyan! The "all-or-nothing" approach makes a lot of sense.

You're right that traceparent, tracestate, and baggage are semantically linked—mixing traceparent from _meta with tracestate from auto-instrumentation would create an inconsistent/invalid trace context.

I've updated Section 6 to adopt this approach:

### 6. Conflict Resolution

When `_meta` contains W3C Trace Context fields and the outbound HTTP request already has corresponding headers set (e.g., via auto-instrumentation):

1. If `traceparent` is present and valid in `_meta`, SDKs MUST use an **all-or-nothing** approach:
   - Discard ALL existing W3C Trace Context headers (`traceparent`, `tracestate`, `baggage`) from the outbound request
   - Forward ONLY the W3C fields present in `_meta`
2. SDKs SHOULD log a debug-level message when overriding existing headers
3. No error should be raised—this is expected behavior when integrating with existing observability infrastructure

**Rationale**: W3C Trace Context fields are semantically linked—`tracestate` contains vendor-specific data keyed to the `traceparent`, and `baggage` may contain context relevant to that specific trace. Mixing fields from different sources would create inconsistent trace context. The all-or-nothing approach ensures trace coherence across the MCP boundary.

This ensures that when _meta provides trace context, it's used as a complete, coherent unit rather than being merged with potentially incompatible auto-instrumented headers.

Pushing the update now.

monahk and others added 2 commits January 12, 2026 17:46
Per @michaelsafyan's feedback: when traceparent is present in _meta,
discard ALL existing W3C headers and use only _meta values. This
prevents semantic inconsistencies between linked trace context fields.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@dsp-ant dsp-ant added SEP proposal SEP proposal without a sponsor. labels Jan 14, 2026
Comment thread seps/0000-meta-http-header-forwarding.md Outdated
Comment thread seps/0000-meta-http-header-forwarding.md Outdated
@localden

Copy link
Copy Markdown
Contributor

@monahk this SEP does not have an assigned sponsor. Are you looking to continue working on it and find sponsorship?

@monahk

monahk commented Jan 22, 2026

Copy link
Copy Markdown
Author

@monahk this SEP does not have an assigned sponsor. Are you looking to continue working on it and find sponsorship?

@localden Thanks for checking in! Yes, I'm actively working on this and would love to find a sponsor. Would you be interested in sponsoring this SEP, or could you recommend someone who might be a good fit given the distributed tracing/observability focus?

@monahk

monahk commented Jan 30, 2026

Copy link
Copy Markdown
Author

@monahk this SEP does not have an assigned sponsor. Are you looking to continue working on it and find sponsorship?

@localden Thanks for checking in! Yes, I'm actively working on this and would love to find a sponsor. Would you be interested in sponsoring this SEP, or could you recommend someone who might be a good fit given the distributed tracing/observability focus?

@localden any thoughts on who might be a good sponsor for this SEP?

@monahk monahk force-pushed the sep-meta-http-header-forwarding branch from 305dadf to e794a44 Compare January 30, 2026 23:45
@monahk monahk requested a review from a team as a code owner January 30, 2026 23:45
Major redesign based on community feedback:
- Replace hardcoded W3C header handling with extensible group-based system
- Separate trace-context and baggage into independent groups (per @lmolkova)
- Single 'policy' setting per group: clear-and-use-meta, merge-with-meta, ignore-meta
- Group integrity: SDK never mixes headers from different sources within a group
- Support for user-defined custom groups (Datadog, Jaeger, internal headers)
- Aligns with OpenTelemetry MCP context propagation guidance

Addresses feedback from @michaelsafyan and @lmolkova on PR modelcontextprotocol#2028.
| `clear-and-use-meta` | Yes (any in group) | Yes | Clear ALL existing in group, forward ALL from `_meta` |
| `clear-and-use-meta` | Yes (any in group) | No | Forward from `_meta` |
| `clear-and-use-meta` | No | Yes | Keep existing |
| `merge-with-meta` | Yes | Yes | Overwrite that specific header with `_meta` value |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Overwrite that specific header" is a surprising behavior given the name "merge".

I would expect "key: value1" and "key: value2" to merge into "key: value1,value2" following standard HTTP behavior for repeating a header line with different values:

https://httpwg.org/http-core/draft-ietf-httpbis-semantics-latest.html#field.lines

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the thorough review @michaelsafyan! Addressing your points:

1. Validation Order

You're right that validation must precede conflict resolution. Section 3 states "SDKs MUST validate _meta fields before forwarding" — I'll make the ordering more explicit in the processing flow.

2. tracestate without traceparent

Great catch! The current wording has a gap. Rather than hardcoding this for trace-context only, I'll add a generic required attribute to header group configuration:

headerGroups: {
  // Predefined group - traceparent is required, tracestate is optional
  "trace-context": {
    headers: ["traceparent", "tracestate"],
    policy: "clear-and-use-meta",
    required: ["traceparent"]  // Group skipped if traceparent missing
  },

  // Custom group example - can specify their own required headers
  "datadog": {
    headers: ["x-datadog-trace-id", "x-datadog-parent-id", "x-datadog-sampling-priority"],
    policy: "clear-and-use-meta",
    required: ["x-datadog-trace-id"]
  }
}

Behavior with required: ["traceparent"]:

_meta contains Result
traceparent + tracestate Forward both
traceparent only Forward traceparent only; discard any existing tracestate
tracestate only (no traceparent) Skip group entirely; preserve existing headers
Neither Skip group; preserve existing headers

This correctly models W3C Trace Context semantics (tracestate is optional but requires traceparent) while allowing custom groups to define their own requirements.

3. "merge" Terminology

Agreed that "merge" is misleading given HTTP semantics. I'll rename to prefer-meta — clearer that _meta takes precedence when present, without implying value concatenation.

Updated policies:

  • clear-and-use-meta — All-or-nothing replacement for coupled headers
  • prefer-meta (née merge-with-meta) — Per-header: _meta wins if present, else keep existing
  • ignore-meta — Always keep existing, never forward from _meta

4. Vendor Implementations

This is spec-first — no production implementations yet. The reference implementation will be in the TypeScript SDK after acceptance. However, the design aligns with:

Happy to coordinate with other SDK maintainers once the spec direction is agreed.


I'll update the SEP with these changes. Let me know if you have any concerns with the required array approach!

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess I'm confused regarding "clear-and-use-meta" vs "prefer-meta" ... couldn't one be implemented in terms of the other based on listing a single header at a time (groups of 1 header) vs multiple headers?

The "required" policy makes sense. A "validator" option may also be necessary.

In terms of vendor implementations, that is assuming that all clients are open source. What I'm saying is that I know of at least one vendor implementation that does what was proposed in my earlier comment . In particular, an all-or-nothing keyed on traceparent for traceparent, tracestate, and baggage. (Yes, baggage is a separate spec and can be carried independent of W3C tracing context; however, this vendor solution considers baggage as logically similar to trace context propagation and often confused by users interchangeable with tracestate as a carrier of extra data piggybacking over tracing instrumentation). That doesn't mean that it has to land that way in OSS, but it may be worth some ground truth research to inform the spec (and to determine where divergences from private vendor solutions may hinder adoptability or create friction points).

Thanks so much for listening and for driving this work forward!

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good points @michaelsafyan!

On policy distinction: The difference is meaningful for multi-header groups:

  • clear-and-use-meta: If _meta has traceparent but not tracestate, existing tracestate is cleared (prevents orphaned state)
  • prefer-meta: Existing tracestate would be preserved (could cause trace coherence issues)

For single-header groups like baggage, they behave identically.

On validator: I like this idea. The required array handles presence checks, but a validator function could handle format validation:

"trace-context": {
  headers: ["traceparent", "tracestate"],
  policy: "clear-and-use-meta",
  required: ["traceparent"],
  validator: (headers) => isValidTraceparent(headers.traceparent)
}

Should I add this to the spec, or keep it as an SDK implementation detail?

On vendor research: This is a great point. You seem to have good visibility into existing implementations — would you be interested in contributing this research? Specifically:

  1. Documenting how existing OSS propagators (OpenTelemetry, Jaeger, Zipkin) handle W3C Trace Context
  2. Identifying any proprietary implementations that treat traceparent/tracestate/baggage as a coupled unit
  3. Flagging potential friction points between this spec and existing approaches

If you'd like to contribute, I'd be happy to add you as a co-author on the SEP. Otherwise, I can do the research — just wanted to offer since you raised the concern and may already have context here.


4. **Simple mental model**: Single `policy` setting per group, 3 clear policies, no ambiguity.

### Why Default-On for W3C Trace Context?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to making it default for tracing.

@michaelsafyan michaelsafyan left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes!

In terms of vendor review, I only have insight into the one. Left one more comment which I think would at least make that behavior implementable in terms of this proposal even if not exactly aligned on the default configuration.

Other than that, LGTM.

Thank you so much for driving this forward!

"trace-context": {
policy: "clear-and-use-meta",
required: ["traceparent"],
validator: (headers) => isValidTraceparent(headers.traceparent),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: allow separate validators per header as in:

  validators: {
     # Will drop/scrub/ignore if invalid
     "traceparent": isValidTraceparent,
      # Will drop/scrub/ignore if invalid
     "tracestate": isValidTracestate,
   }
   # Presence evaluated after validation
   required: ["traceparent"],

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review and the suggestion!

Both changes incorporated:

  1. Per-header validators - replaced the group-level validator with validators (per-header), so individual headers can be validated/dropped independently:
"trace-context": {
  policy: "clear-and-use-meta",
  validators: {
    traceparent: isValidTraceparent,
    tracestate: isValidTracestate,
  },
  required: ["traceparent"],
},
  1. Processing order updated - per-header validation now runs before required presence checks:

    1. Validation (field-level: string type, ASCII, size limits)
    2. Per-header validation (drop individual headers that fail)
    3. Required check (if a required header was dropped by validation, group is skipped)
    4. Policy application

- Changed `validator` to `validators` for per-header validation
- Reordered processing: per-header validation before required check
- Added rationale for validation ordering
@monahk

monahk commented Feb 12, 2026

Copy link
Copy Markdown
Author

Looking for a sponsor to move this SEP forward. Tagging a couple of folks whose work this directly touches:

@felixweinberger - This defines new SDK behavior for the TypeScript and Python SDKs (automatic _meta to HTTP header forwarding, headerGroups configuration API, per-header validators). Would love your input on the SDK implementation surface.

@jonathanhefner - Given your work on the Transports Interest Group and Reference Servers, this sits right at the transport boundary — how _meta context gets carried across to outbound HTTP calls. Reference servers would also be the natural place to demonstrate this.

Would either of you be interested in sponsoring?

@monahk monahk requested a review from a team as a code owner March 24, 2026 23:09
Resolve conflicts from upstream moving docs/community/seps/ to docs/seps/
and add SEP-2028 to new navigation and index.
@monahk monahk force-pushed the sep-meta-http-header-forwarding branch from 35f287f to e6c9d7d Compare March 24, 2026 23:10
@monahk

monahk commented Mar 24, 2026

Copy link
Copy Markdown
Author

@localden any suggestions on how i can find a sponsor for this PR?

…r-forwarding

# Conflicts:
#	docs/seps/index.mdx
@monahk

monahk commented Jun 1, 2026

Copy link
Copy Markdown
Author

Posted a status update in Discussion #2029 covering the current design (the group-based policy model), the conflict-resolution behavior now in §6 of the SEP, and where things stand:
#2029

Also worth flagging: VS Code / Copilot ran into this exact gap in microsoft/vscode#302301 (shipped a _meta.traceparent fix in 1.119.0) and referenced this SEP as the piece that would close the remaining HTTP-header gap.

The one thing still blocking this SEP from advancing is an assigned sponsor. If you're a maintainer working on transports, SDK behavior, or observability and this is interesting, I'd love to talk.

…r-forwarding

# Conflicts:
#	docs/docs.json
#	docs/seps/index.mdx
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

proposal SEP proposal without a sponsor. SEP

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

5 participants