[Enhancement]: Improve AI discovery, crawler indexing, and browser-agent integration (WebMCP, ChatGPT, Claude, Gemini) #4408

Describe the solution you'd like

Building on our initial llms.txt setup (#3534) and WebMCP (document.modelContext) support (#3844, #4301), we should improve AI discovery, crawler indexing, structured metadata, and browser-agent integrations across WordPress Playground Web (playground.wordpress.net, my.wordpress.net) and the Documentation site (wordpress.github.io/wordpress-playground).

Below is a summary of findings across Google Chrome / Gemini (WebMCP), OpenAI (ChatGPT Browser & Codex), Anthropic (Claude Browser Use), and AI Search / Web Crawlers, along with a proposed implementation roadmap.

1. Audit of Current Gaps

2. Platform-by-Platform Findings

A. Google Chrome & Gemini — WebMCP (document.modelContext)

References:

• WebMCP: Build effective tools
• WebMCP: Best practices
• WebMCP: Imperative API & Declarative API
• WebMCP: Secure tools & Evals

Key improvements needed:

1. WebMCP Annotation Hints (tool.annotations):
   • consequentialHint: boolean: Signals high-stakes or state-mutating actions so the browser/agent prompts for user confirmation. Currently tool-definitions.ts sets MCP's destructiveHint, whereas WebMCP checks consequentialHint.
   • untrustedContentHint: boolean: Critical security signal for tools returning site/user-generated content (playground_request, playground_read_file, playground_list_emails) to defend against indirect prompt injection.
   • debugging: boolean: Marks developer/diagnostic tools (playground_execute_php, playground_request, etc.).
   • readOnlyHint: boolean: Already present on read-only tools; should also be preserved across webmcp-frame-bridge.ts along with consequentialHint, untrustedContentHint, and debugging.
2. Cancellation via AbortSignal: Support the second parameter execute(input, { signal }) in webmcp.ts and webmcp-site-tools.ts.
3. Positive Framing & Character Budgets:
   • Replace negative constraints (e.g., playground_request: "Do not use this to navigate the user's browser — use playground_navigate instead") with positive, action-oriented guidance ("Fetches an HTTP response in the background without changing the visible browser page. To navigate the visible preview iframe, call playground_navigate.").
   • Keep descriptions within recommended WebMCP budgets (< 500 chars per tool description, < 150 chars per parameter description, < 30 chars per name) and return actionable recovery hints on errors.

B. OpenAI ChatGPT & Codex — Site Tools (WebMCP) & Annotations Extensibility

References:

• ChatGPT Site tools (WebMCP)
• ChatGPT Annotations Extensibility
• ChatGPT Built-in Browser
• ChatGPT Configuration

Key improvements needed:

1. ChatGPT WebMCP ("Site Tools") Compatibility:
   • ChatGPT's built-in browser discovers top-level document.modelContext.registerTool tools under Site tools in the address bar.
   • Iframe & Declarative Limitations: ChatGPT does not support the Declarative HTML Form API and does not discover tools inside iframes. Playground's webmcp-site-tools.ts top-level proxy already bridges inner WordPress iframe tools to the top-level document, solving the iframe limitation!
   • Strict Input Schema: Ensure zodToJsonSchema in tool-definitions.ts outputs additionalProperties: false on object schemas, matching OpenAI's WebMCP schema convention.
2. ChatGPT Browser Annotations (oai-annotatable & document.oai.annotation):
   • Add oai-annotation-container, oai-annotatable="<region-name>", and oai-annotation-metadata='<flat JSON>' to Playground's top-level UI regions (Viewport, Dock toolbar, Site Settings, Blueprint editor, Logs, File browser).
   • Attach live active-site metadata (siteSlug, wpVersion, phpVersion, currentUrl, activeDockTool) via oai-annotation-metadata so when a developer annotates a visual bug in Playground using ChatGPT's built-in browser, Codex automatically receives the active WordPress/PHP version and route context.

C. Anthropic Claude — Browser Use Tool (browser_toolset_20260801)

References:

• Claude Browser use tool
• Claude Use case guides

Key improvements needed:

1. Accessibility Tree (read_page & find) Optimization:
   • Claude's browser tool navigates primarily via read_page (accessibility tree [ref_N] handles) and find (accessible names and roles), and struggles with elements buried inside nested iframes without top-level controls.
   • Ensure Playground's top-level shell, URL bar, and Dock panels expose clear ARIA landmarks and accessible labels.
2. Agent Documentation Directives:
   • Both learn.chatgpt.com and platform.claude.com expose a machine-readable directive linking to /llms.txt and .md page twins so browsing agents ingest clean Markdown instead of scraping UI chrome.

3. Proposed Implementation Tracks

Track 1: llms.txt, .well-known JSON & Documentation AI Index

• Enrich packages/playground/website/public/llms.txt (and mirror to personal-wp) with:
  • Query API parameters (?php=, ?wp=, ?plugin=, ?theme=, ?url=, ?blueprint-url=, ?networking=yes) and inline # Blueprint JSON URLs so LLMs can construct working Playground links in chat responses.
  • Built-in WebMCP (document.modelContext) tools reference and @wp-playground/mcp / @wp-playground/cli usage.
• Add /llms-full.txt and /.well-known/llms.txt / /.well-known/mcp.json discovery manifests.
• Extend packages/docs/site/plugins/generate-handbook-manifest.js (or add a companion Docusaurus plugin) to auto-generate /llms.txt and /llms-full.txt linking directly to raw .md docs and materialized TypeDoc Markdown (static/handbook/*.md).

Track 2: AI Web Crawlers (robots.txt), Static HTML Fallback & Schema.org JSON-LD

• Fix the raw HTML scraper trap in packages/playground/website/index.html and packages/playground/personal-wp/index.html by providing semantic static fallback content (<h1>, <p>, feature list, and doc links) inside <main id="root"> (automatically replaced when React mounts) and/or <noscript>.
• Add packages/playground/website/public/robots.txt explicitly allowing AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, Google-Extended, PerplexityBot, Applebot-Extended) and referencing /llms.txt.
• Add Schema.org JSON-LD (<script type="application/ld+json">) with WebApplication / SoftwareApplication, Organization (WordPress.org), featureList, offers (free), and sameAs entity links (GitHub, WordPress.org, npm, Docs).
• Enrich packages/playground/website/public/manifest.json with a complete description, categories, and shortcuts.

Track 3: WebMCP (document.modelContext) Spec & Security Alignment

• Update packages/playground/mcp/src/webmcp.ts, webmcp-site-tools.ts, tool-definitions.ts, and packages/playground/remote/src/lib/webmcp-frame-bridge.ts to support Chrome 156+ WebMCP annotations (consequentialHint, untrustedContentHint, debugging, readOnlyHint).
• Add additionalProperties: false to generated JSON schemas in zodToJsonSchema for OpenAI WebMCP compatibility.
• Support AbortSignal (execute(input, { signal })) and actionable error recovery responses in webmcp.ts.
• Refactor negative phrasing in tool descriptions in tool-definitions.ts to follow Chrome WebMCP best practices.

Track 4: ChatGPT Browser Annotations & Claude Browser Accessibility Polish

• Add oai-annotation-container, oai-annotatable, and live oai-annotation-metadata (active site slug, PHP/WP versions, current URL, active Dock pane) to Playground's top-level layout and Dock components.
• Audit top-level ARIA landmarks and interactive control labels for Claude Browser Use (read_page / find).

Describe alternatives you've considered

• Relying solely on client-side rendering and existing llms.txt: Leaves non-JS AI crawlers indexing only the #wkwebview-notice / #load-error-fallback text in index.html, and misses new WebMCP security hints (untrustedContentHint, consequentialHint) and ChatGPT browser annotation hooks.
• Using Declarative WebMCP (<form toolname="...">) alone: Not supported by ChatGPT's built-in browser; keeping our imperative document.modelContext.registerTool implementation as the primary surface (with optional declarative progressive enhancement) ensures cross-platform support across Chrome/Gemini and ChatGPT.

Additional context

• Related PRs: [Website] Add AI discoverability: llms.txt, meta tags, docs section #3534, [MCP] Update WebMCP registration API #3844, Proxy WebMCP tools registered inside the WordPress iframe #4301