Skip to content

Long-running agent sessions exhaust event storage and appear cancelled while CLI remains active #4467

Open

Description

@rchiodo

Description

Long-running Copilot CLI project sessions that spawn many subagents can exhaust the remote session event store. Once this happens, session status and handoffs become unreliable: sessions appear inactive or cancelled even though their CLI processes remain alive and may still be executing subagents.

Environment

  • Copilot CLI: 1.0.79-5
  • OS: Windows
  • Workload: multiple long-running project/worktree sessions using custom agents with planner, debugger, impact-analysis, coverage, and review subagents

Reproduction

  1. Start a project session that performs a multi-hour workflow with frequent tool calls, planner checkpoints, and nested subagents.

  2. Allow the session to accumulate a large event history.

  3. Observe repeated exporter failures such as:

    Failed to submit events to Mission Control session …: 409
    {"code":"session_event_storage_exhausted","message":"session event storage exhausted"}
    remote session batch flush failed … "event_count":500
    remote session exporter circuit opened after batch flush failure
    
  4. Query the session through the app/runtime status APIs.

Actual behavior

  • The project session may lose its active_session_id or appear inactive.
  • Background-agent status may report Cancelled or no agents, without a cancellation reason.
  • Final handoffs may not arrive.
  • Restarting based on this status can create a second writer for the same dirty worktree.
  • The underlying CLI process can still be alive, consuming CPU, updating its log, and executing internal subagents after the UI/runtime reports no active work.
  • Logs can also report cleanup of many orphaned tool calls, but do not identify a timeout, quota, or other cancellation cause.

Expected behavior

  • Event storage should rotate, compact, or start a continuation stream before exhaustion.
  • Exporter failure should not make an otherwise-running session unsteerable or misreport its liveness.
  • Status should distinguish at least: CLI alive, agent running, agent idle, export disconnected, and task cancelled.
  • Cancellation should always expose a reason such as user request, timeout, execution budget, parent teardown, or storage failure.
  • The runtime should prevent or clearly warn about concurrent writers when recovery is attempted for a worktree whose original process is still active.

Impact

This makes long-running autonomous work unsafe to recover. Users cannot tell whether work has stopped, and a reasonable restart attempt can cause concurrent edits in the same worktree. Dirty changes were preserved in this case, but reliable coordination required inspecting local process IDs and logs rather than the product status APIs.

Suggested mitigation

Automatically rotate or compact session event history near the storage limit, preserve a local liveness/control channel when remote export fails, and surface a prominent event storage exhausted health state with recovery guidance.

Activity

  1. added
    area:agentsSub-agents, fleet, autopilot, plan mode, background agents, and custom agents
    area:sessionsSession management, resume, history, session picker, and session state
    and removed on Aug 13, 2026
  2. OpusVision commented on Nov 12, 2026

    @OpusVision

    Additional diagnostic gap on 1.0.109: the exporter now reports only:

    Mission Control rejected events with a permanent client error; disabling further event submission for this session {"status":"Some(409)"}
    

    On the affected resumed session, another launch reproduces the rejection while local conversation continues. The diagnostic does not include the backend reason code, so this observation cannot establish that the cause is session_event_storage_exhausted rather than another conflict.

    Please retain a sanitized backend error code in the diagnostic and provide a supported history-preserving recovery or continuation path. It should not require deleting history, manually changing session identifiers, or disabling sync to make the warning disappear.

    Separate isolated prompt-mode startup tests did not obtain a Mission Control binding, so their successful completion is not evidence that this existing-session export failure is repaired.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:agentsSub-agents, fleet, autopilot, plan mode, background agents, and custom agentsarea:sessionsSession management, resume, history, session picker, and session state

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions