Skip to content

Latest commit

 

History

History
152 lines (121 loc) · 10.6 KB

ai-request-logging.md

File metadata and controls

152 lines (121 loc) · 10.6 KB

AI Request Logging

Summary

Provides an opt-in observability surface that records every AI request (provider, model, duration, token counts, status, and request source) and exposes them through a React-powered dashboard under Tools → AI Request Logs. When enabled, the SDK's HTTP transporter is wrapped with a logging decorator using the public setHttpTransporter() API, and the AI Client generation events are used to log providers that send requests through their own transport.

Key Hooks & Entry Points

  • WordPress\AI\Experiments\AI_Request_Logging\AI_Request_Logging::register():
    • Calls AI_Request_Log_Manager::init() to ensure the log table schema is synchronized and the cleanup cron exists.
    • Calls Logging_Integration::init() so the SDK transporter is wrapped only while the feature is enabled.
    • rest_api_init → registers WordPress\AI\Logging\REST\AI_Request_Log_Controller, which exposes /ai/v1/logs, /summary, /filters, and per-log endpoints.
    • admin_menu → registers WordPress\AI\Logging\AI_Request_Log_Page, which adds the Tools submenu and enqueues assets.
  • Database + cleanup are handled inside AI_Request_Log_Manager::init() (schema-version-gated table sync, cron scheduling, option storage).

Architecture

The logging system uses the decorator pattern to wrap the SDK's HTTP transporter:

  1. Logging_Integration::init() is called from the feature registration flow when the experiment is enabled.
  2. On wp_loaded or admin_init, Logging_Integration::wrap_transporter():
    • Gets the current transporter from AiClient::defaultRegistry()
    • Creates a Logging_Http_Transporter decorator around it
    • Uses registry->setHttpTransporter() to swap in the logging version
  3. All subsequent AI requests go through the logging transporter, which records metrics before delegating to the underlying transporter.
  4. Logging_Http_Transporter also inspects the PHP call stack to record whether the request was initiated by a plugin, mu-plugin, theme, or WordPress core file when that can be determined.
  5. Data extraction is handled by Log_Data_Extractor, which parses request/response payloads and extracts provider, model, tokens, and previews.

Providers that send requests through their own transport (for example one that proxies to a localhost sidecar) never reach the transporter, so Logging_Event_Listener covers them using the core wp_ai_client_{before,after}_generate_result and wp_ai_client_{before,after}_generate_embedding actions, which fire for every provider:

  1. The before-event starts tracking the generation (provider, capability, start time). Generations can nest, so they are tracked as a stack.
  2. Whenever a row is written (wpai_request_logged) for the provider of the innermost generation, that generation is marked as logged. This covers the transporter and any provider that writes its own row via log_ai_request(). Rows for other providers, and writes that fail, do not count.
  3. The after-event writes a row (with context.logged_via set to generation_event) only if nothing was logged for that generation.

Core only fires the after-events on success, so failed generations from custom-transport providers are not logged.

Transporter rows label operation as {provider}:{endpoint} (for example openai:responses). Rows written by Logging_Event_Listener have no request URL to derive an endpoint from, so they use {provider}:{capability} instead (for example codex:text_generation).

This approach uses the SDK's public API rather than reflection or internal hacks, making it resilient to SDK updates.

Public API

Consumers that surface abilities themselves — an MCP server, or code invoking an ability directly — can record their own requests instead of shipping a parallel log.

WordPress\AI\log_ai_request()

Writes an entry to the same log the experiment populates. Returns the log identifier on success, or false when the experiment is disabled or the write failed, so it is safe to call unconditionally.

\WordPress\AI\log_ai_request( array(
    'type'        => 'mcp_tool',       // Required. One of AI_Request_Log_Manager::get_types().
    'operation'   => 'example/tool',   // Required. Non-empty string.
    'status'      => 'success',        // Required. Non-empty string.
    'provider'    => 'openai',
    'model'       => 'gpt-5-nano',
    'duration_ms' => 120,
    'context'     => array( 'input_preview' => '…' ),
) );

type, operation, and status are validated: anything else triggers _doing_it_wrong() and no row is written. type must be one of the values AI_Request_Log_Manager::get_types() returns — ai_client (generations made through the AI Client SDK), mcp_tool, or ability. A row carrying an unlisted type could never be filtered through the REST API, and operation and status are stored in columns that cannot be null.

The REST type filter enum is derived from get_types(), so the read and write sides cannot drift apart.

Action Hooks

wpai_request_logged

Fires after an entry is written, with the log identifier and the stored row. Lets consumers react to logged requests without polling the REST endpoint.

add_action( 'wpai_request_logged', function( $log_id, $data ) {
    if ( 'error' === $data['status'] ) {
        // Alert on failed AI requests.
    }
}, 10, 2 );

Filter Hooks

The logging system exposes several filter hooks for extensibility:

wpai_request_log_providers

Filters the provider detection patterns. The default map is derived from the connectors registry — every ai_provider connector contributes a host-substring pattern equal to its slug (with google matching googleapis since its API runs on a different domain). This filter is the extension point for non-connector providers or for connectors whose slug doesn't appear in their API host (for example, a connector whose API is served from an unrelated domain).

add_filter( 'wpai_request_log_providers', function( $patterns ) {
    $patterns['my_provider'] = array( 'my-api.com', 'api.myprovider.io' );
    return $patterns;
} );

wpai_request_log_context

Filters the log context data before it's stored. Allows adding custom context or removing sensitive data.

add_filter( 'wpai_request_log_context', function( $context, $decoded, $log_data ) {
    $context['custom_field'] = 'custom_value';
    unset( $context['sensitive_field'] );
    return $context;
}, 10, 3 );

wpai_request_log_tokens

Filters the extracted token usage. Allows custom providers to supply their own token extraction logic.

add_filter( 'wpai_request_log_tokens', function( $tokens, $response ) {
    if ( isset( $response['my_token_field'] ) ) {
        $tokens['input'] = $response['my_token_field']['in'];
        $tokens['output'] = $response['my_token_field']['out'];
    }
    return $tokens;
}, 10, 2 );

wpai_request_log_kind

Filters the detected request kind (text, image, embeddings, audio, metadata).

add_filter( 'wpai_request_log_kind', function( $kind, $provider, $path, $payload ) {
    if ( false !== strpos( $path, '/my-custom-endpoint' ) ) {
        return 'custom';
    }
    return $kind;
}, 10, 4 );

wpai_request_log_retention_days

Filters the time-based retention period in days. Logs are retained indefinitely by default (the filter returns 0). Return a positive integer to delete entries older than that many days during the daily cleanup.

add_filter( 'wpai_request_log_retention_days', function() {
    return 30;
} );

Assets & Data Flow

  1. When AI Request Logs is visited, Asset_Loader enqueues admin/ai-request-logs (src/admin/ai-request-logs/index.tsx) plus its stylesheet. The localized payload (window.AiRequestLogsSettings) includes REST routes, a nonce, and initial state (summary and filter metadata).
  2. The React app:
    • Configures @wordpress/api-fetch with the nonce/root.
    • Fetches logs (GET /ai/v1/logs with search/filter params) and displays them in a table with pagination.
    • Uses a persisted operations multi-select that excludes *:models discovery calls by default so capability lookups do not drown out actual request traffic unless the user opts in.
    • Fetches summaries (GET /ai/v1/logs/summary) for the KPI cards, including minute, hour, day, week, month, and all periods, and filter metadata (GET /ai/v1/logs/filters).
    • Sends DELETE /ai/v1/logs from the Manage Logs panel to delete logs. The optional before_days integer argument (minimum 0, default 0) deletes only entries older than that many days; omitting it or passing 0 deletes every log. The response is { success, deleted, message }, where deleted is the number of entries removed. This is a one-off action and does not change the wpai_request_log_retention_days retention period.
  3. On the backend, every AI HTTP request sent through the SDK transporter flows through Logging_Http_Transporter, which records metrics via AI_Request_Log_Manager::log() before returning the response to callers. Successful generations that bypass the transporter are recorded by Logging_Event_Listener instead. Logs are stored in the wp_ai_request_logs table alongside JSON-encoded context for later inspection.

Testing

  1. Toggle AI Request Logging on, and ensure valid AI credentials exist.
  2. Trigger an AI-powered feature (e.g., Type Ahead or Title Generation) so the system issues at least one completion request.
  3. Navigate to Tools → AI Request Logs. Confirm the chart and table populate, and that *:models discovery calls only appear after you explicitly include them in the operations filter.
  4. In the Manage Logs panel, check that the delete dropdown defaults to "Older than 1 year". Choose "Older than 30 days", click "Delete", and confirm the prompt reads "Delete logs older than 30 days?". Click "Cancel" and check that nothing is deleted and focus returns to the "Delete" button.
  5. Click "Delete" again and then "Yes, Delete". Confirm the success notice reports how many entries were deleted, and that logs from the last 30 days remain in the table (switch the time period to "All Time" to see older entries).
  6. Choose "All logs", click "Delete", confirm the prompt reads "Delete all logs?", click "Yes, Delete", and check the table empties.
  7. Disable the experiment and reload a front-end AI feature; no new rows should appear, and the logging integration should remain inactive.

Notes

  • Logs are retained indefinitely by default. Add a wpai_request_log_retention_days filter returning a positive integer if you want time-based cleanup. Site owners can also delete older logs by hand from the Manage Logs panel; that action is one-off and is not saved as a setting.
  • REST endpoints require manage_options.