feat(core): add a temporal durable-execution mode - #2

Phase 2: make each v2 session a durable Temporal workflow. SessionExecutionTemporal
implements the substitutable SessionExecution service (wake -> signalWithStart, resume
-> forced signalWithStart, interrupt -> cancel signal), with the local coordinator's
drain (SessionRunner.run for the whole turn) moved into a runContinuation activity that
runs against the durable event log. The Temporal client and an embedded worker are
co-hosted inside the server process (both run under bun). It is opt-in via
OPENCODE_SESSION_EXECUTION=temporal; the one-line swap is at routes.ts, and the loop,
tools, model, storage, and HTTP API are untouched.

Verified: with it enabled, prompting a v2 session drove a full turn to completion
(step.ended) and Temporal recorded a completed per-session workflow (session-exec-<id>).

Added the engine-level crash-recovery harness (kill the whole server mid-turn, restart,
the turn continues from the event log) and updated the README: Phase 2 is built and
verified, with the run recipe, the code layout, and the known limits (resume does not
yet return the typed RunError; active is process-local; startup needs Temporal reachable).

resume previously drove a forced run fire-and-forget and returned void, so a run
failure was swallowed. It now drives the run through a Temporal Update-with-Start and
awaits the result: a genuine run error is thrown non-retryable by the activity (so only
crashes/timeouts still retry), rejects the update, and the layer maps it to a RunError
(carried as ContextSnapshotDecodeError with the original text in details). The per-session
workflow is now long-lived with an idle timeout so update-with-start can always reach it.

Verified (scripts/resume-check.ts): resume resolves on a healthy session and rejects on a
failing one. Full typecheck stays green (30/30).

active listed a process-local set of sessions this process had started, so it was
empty after a restart. It now queries Temporal for the open per-session workflows
(WorkflowType 'sessionExecution', Running) and maps their ids back to session ids,
so it reflects durable state and survives a restart. The process-local set is gone.

resume previously surfaced run failures as a generic ContextSnapshotDecodeError carrier.
The activity now encodes the error through a Schema.Union of every RunError member
(run-error-codec.ts) into the non-retryable failure details, and the layer walks the
failure chain and decodes it back into the exact tagged instance (e.g. LLMError with its
reason), falling back to the carrier only if decoding fails.

Verified: a unit round-trip reconstructs an LLMError faithfully, and scripts/resume-check.ts
shows a failing resume rejects with the encoded _tag = LLM.Error reaching the caller.

The v2 engine event-sources to SQLite, so a session was resumable only on the host with
the file. Point every worker at one shared store and any worker resumes: a new libSQL
SqlClient backend (sqlite.libsql.ts, over @libsql/client) selected by OPENCODE_DB_URL
gives a networked SQLite (sqld/Turso) across hosts, and OPENCODE_DB already shares a file
same-host. Same SQLite dialect, so the schema and all migrations are unchanged; the
local-only PRAGMAs are skipped for the shared backend.

Verified: a full turn runs against a libSQL file: store (migrations + event log), and
scripts/shared-store-failover.sh shows turn 1 on worker A, A killed, then a fresh worker
B recalls turn 1's code word (two distinct worker identities) purely from the shared
store. Remote-URL transaction atomicity is documented as the remaining networked-writer
step. Full typecheck 30/30.

OPENCODE_SESSION_EXECUTION=temporal ran a whole turn as one activity. New temporal-turn
mode drives the turn one step at a time: SessionRunner gains runStep (one iteration of
run's loop, reusing runTurn), the sessionTurn workflow loops a runTurnStep activity, and
each step (one provider attempt + its tools) is its own activity with its own
retry/timeout/visibility. The step loop is workflow control flow; turn semantics are
unchanged. Selected by one env check in routes.ts.

Verified: a create-then-read-then-reply turn recorded three runTurnStep activities under
a sessionTurn workflow and completed. Finer per-model-call / per-tool granularity is a
larger rewrite left for later. Full typecheck 30/30.

The libSQL client runs each statement as its own auto-commit request, so a
multi-statement event append (the `event_sequence` upsert plus the `event`
insert) could tear on a crash against a remote store. The transaction
connection now drives a real interactive libSQL transaction, so those writes
commit all-or-nothing. Verified against a `file:` store; networked
crash-atomicity still needs a live `sqld`/Turso to test.

Remote libSQL writes now go through interactive transactions, so the caveat about torn multi-statement writes no longer applies; only the networked crash test stays pending.

In temporal-turn mode a Temporal step retry re-invokes `runStep` with
`first=false`, so `failInterruptedTools` never ran and the re-drive
re-streamed a request with a `tool_use` and no `tool_result`. The provider
rejects that, and with `maximumAttempts: 100` it becomes a poison loop. It now
runs before every turn; on a healthy step (whose prior tools already settled)
it is a no-op.

A Temporal step retry re-invokes `runStep` on the same durable log. If the
in-flight step already dispatched tools (`Tool.Called` is recorded before the
side effect runs), re-streaming would re-run those side effects and append a
duplicate assistant message. `runStep` now finalizes such a step from the log:
completed tools keep their results, still-unsettled ones are failed, and a
synthesized `Step.Ended` closes it without re-calling the model. A step with
no dispatched tools is still re-streamed, which is safe. Token metering is 0
for the resumed step; faithful metering would need a durable sealed marker.

Covers the every-entry tool-close fix and the log-resume of a crashed step, with the in-flight-tool honest limit.

The runner rebuilds context only from the shared DB, so the conversation resumes on any worker. The working tree is the one host-local correctness constraint; snapshots and retained tool-output files are viewing-only. Corrects the earlier overstated tool-output gap.

`OPENCODE_TEMPORAL_ROLE` (`both` default, `client`, `worker`) gates the
embedded activity worker and the workflow client, so serve can run client-only
and a worker fleet can scale independently. `packages/server/src/worker.ts`
builds the same application context serve uses (`createWorkerLayer`) without
the HTTP API, so a worker resumes a session purely from the shared store.
Verified by `scripts/standalone-worker-smoke.sh`: a worker comes up with no
serve process and registers a poller on the task queue.

The migration guard was a process-local semaphore, so N cold workers pointing
at one shared store raced: a TOCTOU table check let two processes both create
the schema, or insert the same migration id. `apply` now runs its
check-and-apply inside a single BEGIN IMMEDIATE transaction, so a concurrent
start on another process waits and then observes the migrations already
applied. A new test races five subprocesses against one fresh file.

A tool declares `idempotent` when it has no external side effect (the reads:
`read`, `glob`, `grep`). On a crash resume a tool caught running is ambiguous
(its result was never committed, so we cannot know if it ran), so a
side-effecting tool is still failed and left for the model to redo. A
declared-idempotent one is now re-settled for a real result, since re-running a
pure read is safe. Extracted a shared `emitToolResult` so the resume path and
the streaming publisher encode outcomes identically.

HTTP-level provider retries were already bounded in the RequestExecutor, but
the runner's step loop had no ceiling unless the agent configured one, and a
model repeating the exact same tool calls step after step looped forever. A
default step ceiling (200) and a stuck-loop detector (3 consecutive steps
whose entire tool-call signature set is identical) both route into the
existing last-step machinery: tools disabled, one final text-only wrap-up.
Detection reads only durable history, so it holds across re-drives. A step
that mixes in different work is iterating, not stuck, and never counts.

A pending approval was an in-memory deferred: invisible outside the asking
process, so an ask raised inside a standalone worker could never be answered,
and it vanished on restart. A pending ask is now also a `permission_request`
row in the shared store; the blocked `assert` races its local deferred against
a poll of the row, replies operate on the row (with the decline cascade and
always-rule retro-approval preserved), and reads list the rows. In addition, a
user decline inside a Temporal activity is now non-retryable, so the workflow
does not re-drive a turn the user stopped. The `question` tool still needs the
same treatment.

The remote atomicity story was integration-test-pending for lack of a server.
A new opt-in suite (OPENCODE_LIBSQL_TEST_URL, e.g. `turso dev`) proves over
HTTP that a multi-statement transaction commits all-or-nothing, a
mid-transaction failure rolls back everything, and the schema migrations apply
remotely through the single BEGIN IMMEDIATE path. Skips when no server is set.

Weighs per-worktree task queues, a shared filesystem, and snapshot reconstruction for the one remaining cross-host constraint; recommends affinity now, reconstruction long-term.

Promotion consumes the pending input row inside the turn's own transaction,
so a crashed wake-driven turn left nothing in the input tables and a retried
activity with force=false returned as a no-op: the turn was abandoned until
the next user input. Eligibility now also consults the log: a promoted prompt
with no assistant reply, or an in-flight assistant, is recoverable work in
both `run` and a first-step `runStep`. A settled history still no-ops, so a
retry of a completed drain never re-calls the model. The crash test now
requires the post-crash token (and aborts as invalid if the task finished
before the kill); its old gate was satisfied by pre-crash step.ended events.

…uch.

The wall-clock heartbeat only detects process death, so the 30-minute
startToClose was the sole bound on a running drain and hard-killed legitimate
long turns (many steps, long tools, a human considering a permission ask),
each kill opening a short two-writer window until the zombie attempt noticed.
The heartbeat stays the liveness bound; startToClose is now a 12-hour
backstop. In addition, a drain interrupted by Temporal cancellation rethrows
the cancellation reason, so the attempt records Cancelled instead of Failed.

A retried activity minted a fresh ask id per attempt, piling up pending rows
and re-asking for approvals the user had already given. A tool-originated ask
now derives its id from session + callID + action + resources: a re-drive
adopts the pending row (one visible ask, one reply), an approval that landed
while the asker was dead short-circuits the retry, a decline stays declined,
and a graceful-shutdown `expired` row is revived by the next attempt.

Two waiters parked on the drain condition could both observe it satisfied in
one activation and start two concurrent drains against one session log; the
wait now re-checks in a loop before claiming the drain. And the interrupt
signal cancels the workflow's root scope, so a cancellation could surface at
the idle wait outside any try/catch and record the workflow as Failed; the
main loop now treats that cancellation as a normal stop. Applies to both
workflows. Running workflows from before this change will not replay cleanly;
acceptable on this fork.

resume retries once when the update lands on a workflow that idle-completed
in the same instant, so the caller gets a fresh run instead of the race.
interrupt no longer swallows delivery failures silently (an already-completed
workflow stays a quiet no-op; anything else logs a warning). active queries
both workflow types, so sessions survive a temporal/temporal-turn mode
switch, and intersects with the session store so other deployments sharing
the namespace do not appear as ghost sessions.

The acquirer released its permit on handing out the connection, so an
auto-commit statement (a permission reply, a cross-service read) could race an
open pinned write transaction and die on SQLITE_BUSY, since the remote path
has no busy_timeout. Each statement now runs under the permit; in-transaction
statements use the transaction's own connection, so nothing self-deadlocks.
Documented the per-delta remote transaction cost as a known streaming limit.

The Phase 1 proxy gave createSession/abortTurn a heartbeat timeout they never
satisfy and the abort signal left a floating activity with unlimited retries;
they now use their own bounded proxy and the rejection is swallowed. A
crash-finalized step with only provider-executed tools now closes as "stop"
instead of "tool-calls". The README migration caveat predated the
cross-process BEGIN IMMEDIATE serialization and is updated.

Evaluates ways to run a Temporal-integrated agent without Temporal (language
shim, rust-core shim, plugin pattern, local dev server, harness option). This
branch is first-hand evidence for the plugin pattern, so the doc lives here.
Includes measured numbers: dev server 237 MB / ~780 ms / ~102 MB RSS, and the
marginal cost on this very app (297 MB local vs 494 MB + 123 MB dev server in
temporal-turn mode). Research sections marked pending.

The scenarios now live in runContract(label, makeExec), a suite
parameterized over the SessionExecution factory: wake drives a turn to
settlement then the idle coordinator retires, resume forces a healthy
turn to completion (new), resume surfaces the exact tagged RunError
through the shared codec, and interrupt cancels an in-flight turn.

Because the local coordinator and the Temporal workflow are now
separate loops sharing only the drain, this suite is what holds them to
one behavior. It runs against the local coordinator here and can be
pointed at a Temporal test-env factory to assert the same contract on
that side.

Updated the header comments in workflow-core.ts, temporal-workflow.ts,
drain.ts, temporal.ts, and the factory comment in routes.ts, plus the
temporal README (section renamed "Two modes, one supervisor" -> "Two
modes, one drain"), to reflect that local and Temporal are now separate
coordination loops sharing only the drain body. Parity is stated as an
enforced-by-contract-test property rather than a single shared loop.

Comment/doc only; no behavior change to the Temporal path.

Local mode reads the override at layer build; the Temporal workflow ran
on the hard 5-minute default because the sandbox cannot read env. The
client now forwards the override as a workflow argument, and a
continue-as-new run keeps it.

The suite was parameterized for exactly this. The Temporal run is
opt-in (OPENCODE_CONTRACT_TEMPORAL=1 against a dev server) with one
task queue per run, so a stale worker on a shared server cannot steal
activities; all four scenarios pass through real workflows.

The hand-rolled unit table duplicated Duration.fromInputUnsafe.

The AI-399 evaluation and the worktree design memo are internal
decision records, not part of the change this branch proposes. The
shipped worktree mechanism stays documented in the README and the code;
the one load-bearing note (warm-path alternatives) moved inline.

The claims stay documented and verified in the README; the runnable
reproductions and the tmux demo ride a stacked PR so the branch being
curated for upstreaming carries only the change itself.

Stood up an @temporalio/testing (time-skipping) harness that runs the real
sessionTurn workflow with a mock activity, in-process and deterministic
(test/temporal-harness-smoke.test.ts). It immediately caught a blocker:

On @temporalio/workflow 1.21, condition(fn, timeout) called when fn is
already true leaves the current CancellationScope cancelled. The supervisor
starts with pendingWake=true, so the first idle-wait condition returns true,
and the NEXT condition (in drainTurn) throws CancelledFailure -- which the
loop reads as an interrupt. Result: the workflow completes without ever
scheduling a runTurnStep activity. Temporal mode never drained a turn in
this SDK version. (This, not the HTTP "steer" delivery, was the real cause
of the "0 activities" seen end-to-end.)

Fix: temporal-workflow.ts's condition adapter short-circuits an already-true
predicate, keeping the timeout timer and its scope off that path. Verified
in the harness (activity now scheduled, one drain, clean idle completion)
and live against a dev server (a real gpt-5-mini turn completes: assistant
reply recorded, one runTurnStep activity completed).

The token was runId#attempt, but Temporal activity attempt numbers restart
at 1 for every step, so step 1 attempt 1 and step 2 attempt 1 both minted
`run#1`. A zombie attempt left over from an earlier step could therefore
re-match the current owner and append stale events past the fence. Include
the per-execution activity id so every step's tokens are disjoint; a retry
of the same step still differs by attempt, so it still fences its prior
attempt. Unit-tested in temporal-owner-token.test.ts.

The interrupt path treated any signal failure other than "already
completed"/"not found" as success (logged a warning, returned void), so a
real control-plane failure -- the user's stop never delivered -- read as a
successful stop. Classification is now a tested pure helper
(classifyInterruptError); a genuine failure is surfaced as a defect rather
than false success, while an already-closed idle workflow stays a no-op.

The bound only counted wake-loop drains, so a resume-heavy workflow never
continued-as-new and its history grew until it hit Temporal's limit. The
counter now increments inside drainTurn (every drain), and a `rolloverPending`
flag lets the main loop trigger continueAsNew -- from the workflow's main
method, never an update handler -- once the bound is crossed, even if the
crossing drain came from a resume. Verified with a fake-runtime unit test
(a resume drain crosses maxDrainsPerRun and rolls over).

Records what's fixed (condition blocker, owner-token collision, interrupt
failure reporting, continue-as-new counting) and the interlocking deep
items left as one coherent pass (resume/wake lost at interrupt, concurrent
resumes duplicating turns, fresh-resume spurious drain), each with a fix
sketch and a harness test to validate it.

Addresses the deep coordination findings from independent review, for
Temporal mode (local mode uses SessionRunCoordinator and is unaffected):

- resume JOINS the single in-flight drain instead of queueing a second
  forced one (concurrent resumes no longer duplicate provider turns / tool
  side effects), mirroring SessionRunCoordinator.run.
- interrupt stops the CURRENT turn, not the session: it cancels only the
  turn's child cancellation scope (runInDrainScope) and the long-lived
  workflow keeps serving, so a wake/resume that races the interrupt drives
  a fresh turn on the same workflow instead of being lost to a doomed one.
- A real workflow (root) cancellation is detected via the root scope and
  stops the supervisor -- never keeps serving or continue-as-news.
- Explicit start intent: resume-with-start passes startWithWake=false, so a
  fresh resume no longer does a spurious wake drain; carried across
  continue-as-new.
- continue-as-new counts every drain and gates on allHandlersFinished() so
  an in-flight update's result is never abandoned; a resume-driven rollover
  carries no spurious wake.

Also fixes a blocker uncovered while validating this: the SDK's
condition(fn, timeout) on @temporalio/workflow 1.21 leaks its timer-scope
cancellation into the root scope when it resolves, which poisoned the next
drain -- a session could serve only one turn. The timed wait now races a
no-timeout condition against a bare sleep and abandons the loser, cancelling
no scope, so nothing leaks (and root-cancellation detection stays reliable).

Fake-runtime unit tests (session-supervisor.test.ts): resume joins one
drain, concurrent resumes join, a wake that only joins a resume drain still
gets a follow-up, interrupt keeps the supervisor serving, a root
cancellation stops it, and a fresh resume-with-start does exactly one drain.
Rollover test asserts a resume-driven rollover carries startWithWake=false.

Real-Temporal harness (@temporalio/testing), each in its own file since two
native servers per bun process segfault:
- interrupt: a per-turn interrupt cancels the turn but a later wake runs a
  second turn on the same workflow.
- multiturn: turn 1 completes, the supervisor parks in the idle timed wait
  (asserted via a TimerStarted history event), then a wake drives turn 2 --
  the exact path the condition-leak broke.

…ot a quirk.

The 0-activities symptom was the condition-timeout scope leak, not the delivery
mode. A default (steer) prompt drives a turn to completion once the leak is fixed.

The supervisor redesign reverted the OPENCODE_SESSION_IDLE_TIMEOUT
forwarding and removed the file the contract lib typed against. The
override rides as a third workflow argument now (continue-as-new keeps
it), the lib types against the Temporal node, and the interrupt
scenario expects idle retirement since an interrupted supervisor keeps
serving.