Skip to content

AI Request Logging only captures providers that use the SDK HTTP transporter; sidecar/custom-transport providers are invisible #732

Closed

Description

@henryperkins

Summary

The AI Request Logging experiment captures requests by decorating the SDK's HTTP transporter. Any provider that performs its own HTTP (rather than routing generation through AiClient's transporter) is therefore never logged — even though it's a first-class provider reachable via wp_ai_client_prompt() and shown in the Connectors UI. The Request Log silently under-reports, which is surprising given it presents as a log of every AI request.

Root cause

Logging_Integration::wrap_transporter() installs the tap by swapping in a decorator:

// includes/Logging/Logging_Integration.php
$registry = AiClient::defaultRegistry();
$registry->setHttpTransporter( new Logging_Http_Transporter( $registry->getHttpTransporter(), $log_manager ) );

Logging then happens entirely inside Logging_Http_Transporter::send(). A request is logged iff it flows through HttpTransporter::send(). Provider attribution compounds this — Log_Data_Extractor::detect_provider() derives the provider from the request host, so a provider talking to 127.0.0.1 couldn't be attributed even if its request were seen.

Reproduction

  1. Enable the AI Request Logging experiment.
  2. Use a provider that doesn't route generation through the SDK transporter (e.g. one that proxies to a localhost sidecar or otherwise calls wp_remote_request() itself).
  3. Run a generation via wp_ai_client_prompt(). It succeeds, but no row appears in Tools → AI Request Log. The same site logs OpenAI/Anthropic generations correctly.

Impact

  • Any provider with a custom transport is omitted from the log — observability, cost/token accounting, and debugging all under-count.
  • Concrete case: a third-party codex provider that brokers requests through a localhost sidecar for ChatGPT-managed auth. It registers normally and serves generations, but is entirely absent from the Request Log. We've worked around it provider-side, but that means every such provider must reimplement logging and couple to AI_Request_Log_Manager internals — which is exactly what this issue asks to avoid.

Proposed fix

Add a provider-agnostic fallback alongside the transporter decorator: have the experiment also listen to the core generation events, which fire for every provider regardless of transport — wp_ai_client_before_generate_result / wp_ai_client_after_generate_result (bridged from the SDK's BeforeGenerateResultEvent / AfterGenerateResultEvent, with the dispatcher registered in wp-settings.php).

The after-event exposes everything a row needs:

  • getModel()->providerMetadata()->getId() — provider
  • getModel()->metadata()->getId() — model
  • getResult()->getTokenUsage() — tokens
  • duration = before → after

To avoid double-logging transporter-based providers (which fire the event and pass through the decorator), the listener should only write when the transporter did not already log the current generation — e.g. a flag reset in the before-event and set by Logging_Http_Transporter::send().

Known limitation: the SDK's after-event fires on success only (there is no error event), so this fallback captures successful non-transporter generations but not failed ones. That still closes the success case for all providers; failed custom-transport generations remain a separate gap that would need an SDK-level error event.

Alternatives considered

  • Provider-side logging (our current stopgap): works, but pushes coupling to AI_Request_Log_Manager into every provider and is easy to get wrong.
  • Hooking the WP HTTP API (pre_http_request): too broad, noisy, and can't reliably attribute the provider.

Prior art

#680 fixed a different "the logger misses data" case (thinking-token undercount) — same spirit of completeness.

Activity

  1. moved this from Triage to Backlog in WordPress AI Roadmapon Jun 18, 2026
  2. added this to the Future Release milestone on Jun 18, 2026
  3. added theissue type on Jun 18, 2026
  4. moved this from Backlog to In progress in WordPress AI Roadmapon Jun 22, 2026
  5. modified the milestones: Future Release, 1.2.0 on Jun 26, 2026
  6. modified the milestones: 1.2.0, 1.3.0 on Jul 13, 2026
  7. moved this from In progress to Needs review in WordPress AI Roadmapon Aug 10, 2026
  8. modified the milestones: 1.3.0, 1.4.0 on Aug 11, 2026
  9. modified the milestones: 1.4.0, 1.6.0 on Sep 17, 2026
  10. moved this from Needs review to In progress in WordPress AI Roadmapon Sep 17, 2026
  11. DevMahbubHussain commented on Oct 8, 2026

    @DevMahbubHussain
    Contributor

    Hi, I'd like to work on this one.

    I checked the proposed approach against the current code and it holds up:

    • wp_ai_client_before_generate_result and wp_ai_client_after_generate_result are dispatched from PromptBuilder::generateResult() for every provider, regardless of transport.
    • The after-event provides the provider ID, model ID, token usage and capability. Duration can be measured between the two events.

    My plan:

    1. Add a small listener class in includes/Logging/, registered from Logging_Integration::init().
    2. The before-event resets a per-generation flag and starts a timer.
    3. Logging_Http_Transporter::send() sets the flag when it logs a request.
    4. The after-event writes a row only if the flag is unset, so transporter-based providers are not double-logged.
    5. Add PHPUnit coverage in tests/Integration/Includes/Logging/ for both the custom-transport case and the no-double-logging case.

    Two things I'd like maintainers' input on before I start:

    • Fallback rows have no HTTP URL, so they would carry less detail than transporter rows (provider, model, tokens, duration, status, and capability as the operation). Is that acceptable, or should they also include an input preview built from the messages?
    • As noted in the issue, the after-event fires on success only. I'd treat failed custom-transport generations as out of scope here and leave them for a separate issue that needs an SDK error event. Does that work?

    I won't open a PR until the direction is agreed.

  12. dkotter commented on Oct 8, 2026

    @dkotter
    Contributor

    @DevMahbubHussain As you can see, there's already an open PR linked to this Issue so there's not currently a need for a new PR. If you want to review and test that PR, that would be helpful

  13. DevMahbubHussain commented on Oct 8, 2026

    @DevMahbubHussain
    Contributor

    Thanks @dkotter, I reviewed and tested #757 and left my notes there: #757 (comment)

  14. modified the milestones: 1.6.0, 1.5.0 on Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions