A couple of comments on this as I prepare to undraft #223 for RFC:

• I've updated RFC: Client / Server Content capabilities #223 to indicate that Servers that support structured output should advertise generates: ["application/json"]
• Is there any consideration for the Server returning a TextResourceContents with a mimeType of application/json. I think this would be a more deliberate action by the Server in this scenario.

[update]
The specific proposal would be to return a CallToolResult as follows:

{
  "jsonrpc": "2.0",
  "id": "abc123",
  "result": {
    "content": [
      {
        "type": "resource",
        "resource": {
          "uri": "file:///example/data.json",
          "mimeType": "application/json",
          "text": "{\"name\":\"John Doe\",..... and so on }}"
        }
      }
    ],
    "isError": false
  }
}

With guidance that Servers returning a Structured Response MUST return a CallToolResult containing one EmbeddedResource of type application/json.

Having outputSchema restrict you to to returning a single text content entry who's text validates to the schema feels oddly restrictive. That approach makes annotations largely pointless, and I can think of plenty of cases where one would want to have multiple content entries that would be possible with #356:

• Document processing:
  • Have an outputSchema that is treated as definitions for multiple document types
  • Return content entries for:
    • ImageContent - Generated thumbnail for the document
    • TextContent - Plain text contents of the document
    • DataContent - Structured data, with schema referencing one of the definitions in outputSchema
• Multi-entity type search
  • Have an outputSchema that is treated as definitions for multiple entity types
  • Return multiple content entries with schema refs and annotations for relevance/importance

Another drawback I see is the lack of ability to dynamically define the structure/schema for response content. There are certainly cases where the output schema may not be known ahead of time, but would still be useful for the client or LLM consuming the content; it also enriches the capability of sampling and prompt messages.

Lastly, going with this approach, extending its functionality in the future would likely represent a breaking chance since it largely goes against the implied design pattern of CallToolResult having an arbitrary number of content entries.

Having outputSchema restrict you to to returning a single text content entry who's text validates to the schema feels oddly restrictive. That approach makes annotations largely pointless, and I can think of plenty of cases where one would want to have multiple content entries that would be possible with #356:

• Document processing:

  • Have an outputSchema that is treated as definitions for multiple document types

  • Return content entries for:

    • ImageContent - Generated thumbnail for the document
    • TextContent - Plain text contents of the document
    • DataContent - Structured data, with schema referencing one of the definitions in outputSchema

• Multi-entity type search

  • Have an outputSchema that is treated as definitions for multiple entity types
  • Return multiple content entries with schema refs and annotations for relevance/importance

Another drawback I see is the lack of ability to dynamically define the structure/schema for response content. There are certainly cases where the output schema may not be known ahead of time, but would still be useful for the client or LLM consuming the content; it also enriches the capability of sampling and prompt messages.

Lastly, going with this approach, extending its functionality in the future would likely represent a breaking chance since it largely goes against the implied design pattern of CallToolResult having an arbitrary number of content entries.

@lukaswelinder first of all, my apologies for putting up this PR without first participating in the discussion on #356 - I only saw it as I was writing the PR comments for this, but hadn't had a chance to look properly yet. Definitely didn't mean to step on your ongoing work.

I'll respond to your comment a bit later (not at keyboard right now) and also make comments on #356. Meanwhile I'll move this to draft, pending further discussion.

A couple of comments on this as I prepare to undraft #223 for RFC:

• I've updated RFC: Client / Server Content capabilities #223 to indicate that Servers that support structured output should advertise generates: ["application/json"]
• Is there any consideration for the Server returning a TextResourceContents with a mimeType of application/json. I think this would be a more deliberate action by the Server in this scenario.

[update] The specific proposal would be to return a CallToolResult as follows:

{
  "jsonrpc": "2.0",
  "id": "abc123",
  "result": {
    "content": [
      {
        "type": "resource",
        "resource": {
          "uri": "file:///example/data.json",
          "mimeType": "application/json",
          "text": "{\"name\":\"John Doe\",..... and so on }}"
        }
      }
    ],
    "isError": false
  }
}

With guidance that Servers returning a Structured Response MUST return a CallToolResult containing one EmbeddedResource of type application/json.

@evalstate thanks for the heads up - will come back to this after we see what comes out of the discussion on #356 , per previous comment

@lukaswelinder first of all, my apologies for putting up this PR without first participating in the discussion on #356 - I only saw it as I was writing the PR comments for this, but hadn't had a chance to look properly yet. Definitely didn't mean to step on your ongoing work.

I'll respond to your comment a bit later (not at keyboard right now) and also make comments on #356. Meanwhile I'll move this to draft, pending further discussion.

@bhosmer-ant No offense taken, just glad to see there is motivation here. Input and feedback on #356 would be great.