Skip to content

refactor(coderd/x/chatd/chatdebug): share one finalize-once lifecycle between the stream wrappers - #30981

Merged
ibetitsmike merged 37 commits into
mainfrom
mike/chatdebug-stream-finalize-once
Oct 21, 2026
Merged

ibetitsmike merged 37 commits into
mainfrom
mike/chatdebug-stream-finalize-once

Conversation

@ibetitsmike

@ibetitsmike ibetitsmike commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

wrapStreamSeq and wrapObjectStreamSeq each carried a copy of the same finalize-once lifecycle. This PR moves it into a private streamStep (the once-only gate and the context safety net) and a streamOutcome (finish and error bookkeeping, and the early-stop and missing-finish status rules), and rewrites both wrappers on them. Each wrapper keeps its own part switch and payload type. It also deletes a stop != nil check that is never false and a seq != nil check that Stream and StreamObject make before they wrap a sequence.

Why it is safe: the PR changes no test. #30980 pins both wrappers' status rules, and the chatdebug package and the chatd debug-run tests pass before and after. All 39 mutants of the new lifecycle and of each wrapper's wiring fail the package's tests, and a copy of model.go with garbage in it fails to build. One more mutant drops the finalized check from the safety net, so a step can be written twice; #30980's race case kills it in 50 of 50 runs without -race and in 45 to 49 of 50 with -race. Two details keep their values: the response usage normalizes a zero fantasy.Usage when no finish part arrived, which gives the same zero value as before, and the summary's error count and last error are copied from streamOutcome when the step finalizes instead of being counted inline.

Review focus: the once-only gate stays per iterator (streamStep), not on stepHandle, because stepHandle.finish lets a retry overwrite an earlier result. finish builds the payload only when its call finalizes the step, under the same lock as before.

Moved onto the deletion stack: #30988 deletes the stream heartbeats, so the merge commit that moved this layer (no force push) also drops them from the new lifecycle: heartbeatDone, startHeartbeat, the channel close in finalizeLocked, both step.startHeartbeat() calls and launchHeartbeat, and with them the heartbeat blocks in debugModel.Generate and GenerateObject. The rest of the diff is unchanged.

Validation:

  • go test -race -count=1 -p 4 -parallel 4 ./coderd/x/chatd/chatdebug/... passes before and after.
  • go test -race -count=1 -p 4 -parallel 4 ./coderd/x/chatd/ -run '^(TestActiveServer_ChatTurnDebugRun.*|TestRunnerDebugTurn.*|TestProposeChatTitle_DebugRun|TestArchiveChatDeletesDebugRuns|TestEditMessageDebugCleanup.*|TestQuickgen.*|TestMaybeGenerateChatTitle.*|TestPrepareManualTitleDebugRunCreateFailure)$': the same 25 tests pass before and after.
  • golangci-lint run --fix=false ./coderd/x/chatd/chatdebug/... reports 0 issues, and the pre-commit hook passed.

Stack

This stack deletes the stale debug-run sweep, then simplifies the chatdebug debug-run, stream wrapper and step numbering code, drops the indexes the sweep used, and last deletes a runner debug field that lost its only reader with the heartbeats. The debug-run and stream wrapper refactors each sit on a test-only layer that pins the behavior they keep.

  1. refactor(coderd): delete the chat debug run and step heartbeats #30988 Delete the chat debug run and step heartbeats
  2. refactor: delete chatdebug FinalizeStale and the stale threshold #30989 Delete FinalizeStale and the stale threshold
  3. refactor: delete the FinalizeStaleChatDebugRows query #30995 Delete the FinalizeStaleChatDebugRows query
  4. test(coderd/x/chatd): pin the debug runs that turns, titles and status labels record #30933 Pin the debug runs that turns, titles and status labels record
  5. refactor(coderd/x/chatd): create and finalize debug runs through one helper #30934 Create and finalize debug runs through one helper
  6. test(coderd/x/chatd/chatdebug): table the stream wrapper tests and cover their status rules #30980 Table the stream wrapper tests and cover their status rules
  7. This PR: Share one finalize-once lifecycle between the stream wrappers
  8. refactor(coderd): number chat debug steps in the database #31007 Number chat debug steps in the database
  9. refactor(coderd/x/chatd/chatdebug): delete the process-global step counters #31008 Delete the process-global step counters
  10. chore(coderd/database): drop the chat debug stale-sweep indexes #31009 Drop the chat debug stale-sweep indexes
  11. refactor(coderd/x/chatd): drop the unused runnerCtx from the runner debug turn #31010 Drop the unused runnerCtx from the runner debug turn

Xum, an AI agent, opened this PR for @ibetitsmike and wrote this description.

Table the step heartbeat tests over the step and run launchers, check the exact touch parameters on each tick, and wait for the ticker to stop after done closes or ctx is canceled. The run launcher had no test before.
…uction

chatd sets the debug stale threshold once, inside debugSvcFactory, before
any caller can see the service. Replace Service.SetStaleAfter with a
WithStaleAfter option and delete the atomic, mutex and wakeup channel
that let running heartbeats pick up a runtime change, along with the
subtest that exercised that path. The step and run heartbeat launchers
now share one loop, and the test-only WithClock option is gone.
…s labels record

Four call sites create and finalize debug runs by hand. Before they share
one helper, pin what each records: every create parameter, the run context
the model call sees, the finalize status and seed summary, and what happens
when the insert fails. Title generation and status labels had no debug-run
test at all; the chat turn test checked only a few fields.
…helper

Chat turns, manual titles, automatic titles and turn status labels each
created a debug run on a detached, time-bounded context, rebuilt its run
context from the returned row and finalized it by hand. Add
chatdebug.Service.BeginRun and StartRun and route those call sites through
them. This deletes prepareQuickgenDebugCandidate, shortTextCandidate,
chatdebugRunContext, debugCreateRunTimeout and a literal 5 second insert
timeout.

Create and finalize failures now log through the chatdebug logger with one
message each. The compaction copy is unchanged.
prepareManualTitleDebugRun is a plain function on this branch, so the merged test calls it directly.
The heartbeats only refreshed updated_at on in-progress debug runs and
steps so that FinalizeStale would not sweep them. FinalizeStale has no
production caller, so the heartbeats kept nothing alive. Delete both
launchers, Service.TouchStep, Service.TouchRun, the threshold-change
wake-up channel, and the TouchChatDebugStepAndRun and
TouchChatDebugRunUpdatedAt queries with their authz wrappers.
FinalizeStale has no production caller. The stale threshold,
SetStaleAfter, WithClock, the finalize event and its broadcast channel
existed only for it and its tests. Delete them, the SetStaleAfter call
in chatd.go, and the terminal-status alignment test together with
codersdk.ChatDebugTerminalStatuses, which only that test used. Comments
that pointed at the stale sweep now describe dbpurge instead.
…he WithClock doc

This PR deletes the heartbeat tickers, so the WithClock doc no longer
names them.
…alize-stale

# Conflicts:
#	coderd/x/chatd/chatdebug/service.go
… stale sweep

The edit cleanup comment and the notes on removed archive and stop
tests named the stale-finalization sweep, which this PR deletes. The
edit cleanup test now scopes its "only DeleteAfterMessageID removes
the run" argument to the test, since the chat delete cascade and
dbpurge also remove runs.
…vice_StartRun

The finalize mock checked only the deadline, run ID, chat ID and status,
so a finalizer that dropped the seed summary still passed. Require the
finalized summary to carry the seeded first message.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

Xum, an AI agent, requested this review for @ibetitsmike.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: 5a4003f0ae

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike
ibetitsmike marked this pull request as ready for review October 19, 2026 12:14
@pr-queue-manager
pr-queue-manager Bot requested a review from jaaydenh October 19, 2026 12:16
@ibetitsmike
ibetitsmike requested review from jscottmiller and removed request for jaaydenh October 20, 2026 12:09
ibetitsmike added a commit that referenced this pull request Oct 20, 2026
Deletes the chat debug heartbeats. Model calls and streams started a
goroutine that touched the step and its run on a ticker, and runner
debug turns started a second one that touched the run. Both existed so
that `FinalizeStale` would not mark a live run as interrupted, but
nothing in production calls `FinalizeStale` since #26270. Mike chose to
delete the stale sweep as dead code. This PR is the first of three that
do it.

- `chatdebug/model.go`: `Generate`, `GenerateObject` and both stream
wrappers no longer start `launchHeartbeat`.
- `active_turn_debug.go`: the runner debug turn no longer starts
`LaunchRunHeartbeat`.
- `chatdebug/service.go`: deletes `TouchStep`, `TouchRun`,
`LaunchRunHeartbeat`, `heartbeatInterval`, and the channel that
`SetStaleAfter` closed to wake the heartbeats. The `WithClock` doc no
longer names the heartbeat tickers.
- Deletes the `TouchChatDebugStepAndRun` and
`TouchChatDebugRunUpdatedAt` queries with their dbauthz wrappers and
tests, and `TestLaunchHeartbeat`. `TestFinalizeStaleChatDebugRows` now
backdates its run with `UpdateChatDebugRun` instead of the deleted
test-only query.

**Why it is safe:** the heartbeats wrote only `updated_at` on
`chat_debug_runs` and `chat_debug_steps`. These read it:

| Reader | Why it does not need the heartbeat |
|---|---|
| `FinalizeStaleChatDebugRows` and the partial indexes
`idx_chat_debug_runs_stale` and `idx_chat_debug_steps_stale` | No
production caller. The next two PRs delete the query, and #31009 drops
the indexes in a migration. |
| `DeleteOldChatDebugRuns` (dbpurge) | See the next paragraph. |
| `DebugPanel.tsx` run sort | Sorts by `started_at \|\| updated_at`, and
`started_at` is `NOT NULL`, so it never falls back to `updated_at`. |
| Debug run and step API responses | Return `updated_at` as data.
Nothing uses it to decide whether a run is alive. |

dbpurge deletes runs whose `updated_at` is older than the debug
retention window, whatever their status, and their steps cascade. The
window is `agents_chat_debug_retention_days`: 30 days by default, and 0
turns the purge off, so dbpurge never deletes a run that was written in
the last day. A run's `updated_at` still moves when the run is created,
when each step is inserted (the `InsertChatDebugStep` CTE), and on every
run update and finalize. So dbpurge could only delete a live run if one
model call plus the tool calls after it ran for more than a day. The
runner finalizes its debug run when the runner context ends, a
`requires_action` wait times out after 5 minutes (the case `TouchRun`
covered), and chatloop aborts a stream that is silent for
`--chat-stream-silence-timeout`, 10 minutes by default (0 turns that
guard off).

Mike accepted the cost: a run or step orphaned by a crashed replica
stays `in_progress` until dbpurge deletes it. That is already true on
`main`, because nothing runs the sweep.

This does not touch the chat worker's `chat_heartbeats` loop that
ARCHITECTURE.md describes, and no ARCHITECTURE.md sentence about debug
runs becomes false.

**Review focus:** nothing uses `WithClock` after this PR, but deleting
it here would take this PR past 500 changed lines, so it goes with
`FinalizeStale` in the next PR.

Validation: `make gen/db`, `go build ./coderd/...`, and `golangci-lint
run` on the touched packages (0 issues). `go test -count=1` passes for
`./coderd/x/chatd/chatdebug/...`, `./coderd/database/dbauthz/ -run
TestMethodTestSuite`, `./coderd/database/ -run ChatDebug`,
`./coderd/x/chatd/ -run Debug` and `./coderd/database/dbpurge/`. The
pre-commit hook passed.

## Stack

This stack deletes the stale debug-run sweep, then simplifies the
chatdebug debug-run, stream wrapper and step numbering code, drops the
indexes the sweep used, and last deletes a runner debug field that lost
its only reader with the heartbeats. The debug-run and stream wrapper
refactors each sit on a test-only layer that pins the behavior they
keep.

1. This PR: Delete the chat debug run and step heartbeats
2. #30989 Delete `FinalizeStale` and the stale threshold
3. #30995 Delete the `FinalizeStaleChatDebugRows` query
4. #30933 Pin the debug runs that turns, titles and status labels record
5. #30934 Create and finalize debug runs through one helper
6. #30980 Table the stream wrapper tests and cover their status rules
7. #30981 Share one finalize-once lifecycle between the stream wrappers
8. #31007 Number chat debug steps in the database
9. #31008 Delete the process-global step counters
10. #31009 Drop the chat debug stale-sweep indexes
11. #31010 Drop the unused `runnerCtx` from the runner debug turn

> Xum, an AI agent, opened this PR for @ibetitsmike and wrote this
description.
ibetitsmike added a commit that referenced this pull request Oct 20, 2026
)

Deletes `chatdebug.Service.FinalizeStale` and everything that existed
only for it. Nothing in production calls `FinalizeStale` since #26270,
and #30988 removed the heartbeats that kept live runs out of its reach.
This is the second of three PRs that delete the stale debug-run sweep.

- `chatdebug/service.go`: deletes `FinalizeStale`, `SetStaleAfter`,
`DefaultStaleThreshold`, the `staleAfterNanos` field and `WithClock`.
The service keeps its real clock for the timestamps it writes.
- `chatd.go`: the debug service factory no longer calls
`SetStaleAfter(inFlightChatStaleAfter * 3)`. `inFlightChatStaleAfter`
stays, because the chat worker uses it.
- `chatdebug/types.go`: deletes `EventKindFinalize` and
`BroadcastPubsubChannel`. `FinalizeStale` was the only publisher without
a chat ID, so `PubsubChannel` loses its `uuid.Nil` branch. Nothing in
production subscribes to `chat_debug:*` channels.
- Deletes the four `FinalizeStale` service tests, and
`TestFinalizeStaleChatDebugRows_TerminalStatusAlignment` together with
`codersdk.ChatDebugTerminalStatuses`, which only that test used. The
test compared that list with the `NOT IN` lists in
`FinalizeStaleChatDebugRows` and nothing else: no other debug query or
constraint lists the statuses. The next PR deletes that query.
- Comments that named the stale sweep now say what happens instead:
`debugCleanupClockSkew`, `errRunFinalized`, `CreateStep`, `UpdateStep`,
the `InsertChatDebugStep` and `UpdateChatDebugStep` queries, the edit
cleanup in `chat_mutations.go`, the two notes on removed tests in
`chatd_internal_test.go`, and four comments in `chatd_test.go`.

**Why it is safe:** `FinalizeStale` and `SetStaleAfter` had no
production caller other than the factory line this PR removes, and
nothing calls `WithClock` after #30988. `make gen/db` changes only the
regenerated query comments. #30988 explains why no reader of
`updated_at` needs the sweep and why dbpurge cannot delete a live run.

**Review focus:** the `debugCleanupClockSkew` comment in
`chatd_debug.go`. The value stays `30 * time.Second`. Runs that start
inside that buffer survive the fast cleanup; the old comment said the
stale sweep would then finalize them, but dbpurge deletes them once they
age past the debug retention window.

Validation: `make gen/db`, `go build ./...`, and `go vet` on the touched
packages. `go test -count=1` passes for
`./coderd/x/chatd/chatdebug/...`, `./coderd/x/chatd/ -run
'Debug|EditMessage|ArchiveChat'`, `./coderd/database/ -run
'ChatDebug|TestIsAuthorizedQuery|TestInsertChatMessagesOrderContract|ConnectionLogs'`
and `./codersdk/`. The pre-commit hook passed.

## Stack

This stack deletes the stale debug-run sweep, then simplifies the
chatdebug debug-run, stream wrapper and step numbering code, drops the
indexes the sweep used, and last deletes a runner debug field that lost
its only reader with the heartbeats. The debug-run and stream wrapper
refactors each sit on a test-only layer that pins the behavior they
keep.

1. #30988 Delete the chat debug run and step heartbeats
2. This PR: Delete `FinalizeStale` and the stale threshold
3. #30995 Delete the `FinalizeStaleChatDebugRows` query
4. #30933 Pin the debug runs that turns, titles and status labels record
5. #30934 Create and finalize debug runs through one helper
6. #30980 Table the stream wrapper tests and cover their status rules
7. #30981 Share one finalize-once lifecycle between the stream wrappers
8. #31007 Number chat debug steps in the database
9. #31008 Delete the process-global step counters
10. #31009 Drop the chat debug stale-sweep indexes
11. #31010 Drop the unused `runnerCtx` from the runner debug turn

> Xum, an AI agent, opened this PR for @ibetitsmike and wrote this
description.
ibetitsmike added a commit that referenced this pull request Oct 20, 2026
Deletes the `FinalizeStaleChatDebugRows` query. #30989 deleted
`chatdebug.Service.FinalizeStale`, its only caller. This is the last of
three PRs that delete the stale debug-run sweep.

- Deletes the query from `chatdebug.sql` and regenerates the querier,
dbmock and dbmetrics code. `make gen/db` leaves the dbmetrics wrapper of
a deleted query behind, so I removed it by hand.
- Deletes the dbauthz wrapper and its `TestMethodTestSuite` case, and
`TestFinalizeStaleChatDebugRows` from `querier_test.go`.
- The `codersdk.ChatDebugStatus` comment no longer names the query.

**Why it is safe:** nothing calls the query after #30989. #30988
explains why no reader of the debug `updated_at` columns needs the sweep
and why dbpurge cannot delete a live run.

**Review focus:** the partial indexes `idx_chat_debug_runs_stale` and
`idx_chat_debug_steps_stale` from migration 000468 served only this
query. They stay in this PR; #31009, at the top of this stack, drops
them in a migration.

Validation: `make gen/db` and `go build ./...`. `go test -count=1`
passes for `./coderd/database/dbauthz/ -run TestMethodTestSuite`,
`./coderd/database/ -run ChatDebug`, `./codersdk/` and
`./coderd/database/dbpurge/`. The pre-commit hook passed.

## Stack

This stack deletes the stale debug-run sweep, then simplifies the
chatdebug debug-run, stream wrapper and step numbering code, drops the
indexes the sweep used, and last deletes a runner debug field that lost
its only reader with the heartbeats. The debug-run and stream wrapper
refactors each sit on a test-only layer that pins the behavior they
keep.

1. #30988 Delete the chat debug run and step heartbeats
2. #30989 Delete `FinalizeStale` and the stale threshold
3. This PR: Delete the `FinalizeStaleChatDebugRows` query
4. #30933 Pin the debug runs that turns, titles and status labels record
5. #30934 Create and finalize debug runs through one helper
6. #30980 Table the stream wrapper tests and cover their status rules
7. #30981 Share one finalize-once lifecycle between the stream wrappers
8. #31007 Number chat debug steps in the database
9. #31008 Delete the process-global step counters
10. #31009 Drop the chat debug stale-sweep indexes
11. #31010 Drop the unused `runnerCtx` from the runner debug turn

> Xum, an AI agent, opened this PR for @ibetitsmike and wrote this
description.
ibetitsmike added a commit that referenced this pull request Oct 20, 2026
…s labels record (#30933)

Adds tests that pin what four call sites record when they create and
finalize a debug run: chat turns, manual titles, automatic titles and
turn status labels. #30934 moves all four onto one helper. Automatic
titles and status labels had no debug-run test, and the chat turn and
manual title tests checked only some fields.

- `TestRunnerDebugTurnEnsureCreatesOnce` now checks every insert
parameter, the full `RunContext` the model call sees, and that the
insert runs on a context that ignores the caller's cancellation but has
a deadline.
- New `TestRunnerDebugTurnEnsureCreateFailureDisablesTurn`: a failed
insert disables debug for the turn, and the turn does not retry it.
- `TestProposeChatTitle_DebugRun` also checks that the run has no
trigger message, its model config and model, the seed label in its
summary, and that it recorded steps.
- New `TestQuickgenDebugRun` covers automatic titles and status labels
when the model succeeds, when it fails, and when the insert fails. It
checks the insert, the run context and the finalize call. Titles record
the configured provider and model; status labels record the resolved
ones.
- New `TestPrepareManualTitleDebugRunCreateFailure`: when the manual
title insert fails, the title call gets no run and finishing does
nothing.

**Why it is safe:** the PR changes only tests, and they pass on `main`'s
production code. To show they can fail, I ran 34 mutants of the four
call sites, and 31 fail the new tests. Two survivors change nothing
these tests can observe: the seed summary on the manual title insert
(finalize rewrites the summary) and the 10 second finalize timeout for
titles and labels. This layer does not pin the third: removing
`context.WithoutCancel` from the title and label insert in
`quickgen.go`. #30934 moves that detach into the shared `BeginRun`,
which the chat turn test pins.

Validation: `go test -race -count=1 -p 4 -parallel 4 ./coderd/x/chatd/
-run
'^(TestRunnerDebugTurn.*|TestProposeChatTitle_DebugRun|TestQuickgenDebugRun|TestPrepareManualTitleDebugRunCreateFailure)$'`
passes on this branch and with `main`'s production code, and the
pre-commit hook passed.

This layer now sits on #30995 instead of #30932. A merge commit moved it
there without a force push, and the layer's own diff is unchanged (same
patch-id). The same command and `go test -race
./coderd/x/chatd/chatdebug/...` pass at the new head.

## Stack

This stack deletes the stale debug-run sweep, then simplifies the
chatdebug debug-run, stream wrapper and step numbering code, drops the
indexes the sweep used, and last deletes a runner debug field that lost
its only reader with the heartbeats. The debug-run and stream wrapper
refactors each sit on a test-only layer that pins the behavior they
keep.

1. #30988 Delete the chat debug run and step heartbeats
2. #30989 Delete `FinalizeStale` and the stale threshold
3. #30995 Delete the `FinalizeStaleChatDebugRows` query
4. This PR: Pin the debug runs that turns, titles and status labels
record
5. #30934 Create and finalize debug runs through one helper
6. #30980 Table the stream wrapper tests and cover their status rules
7. #30981 Share one finalize-once lifecycle between the stream wrappers
8. #31007 Number chat debug steps in the database
9. #31008 Delete the process-global step counters
10. #31009 Drop the chat debug stale-sweep indexes
11. #31010 Drop the unused `runnerCtx` from the runner debug turn

> Xum, an AI agent, opened this PR for @ibetitsmike and wrote this
description.
}

// Stopping at the finish part is not an interruption.
func (o *streamOutcome) stoppedStatus() Status {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't from this PR (the old inline switch did the same, and the refactor keeps it as it was), but pulling the rules into one place makes it easy to see: stoppedStatus and endedStatus disagree on one input.

Take a stream that sends an error part carrying context.Canceled and then a finish part. observeError sets out.status to interrupted through streamErrorStatus, and observeFinish then sets finished.

  • If the consumer stops at the finish part, stoppedStatus falls through to case o.finished and stores completed, with an error payload and error_count: 1.
  • If the stream ends on its own, endedStatus returns o.status and stores interrupted.

So the same parts end up with a different status depending on whether the consumer breaks early. If interrupted is the intended result, stoppedStatus could return o.status whenever it isn't StatusCompleted. Then it matches endedStatus and the two could become one function. Either way, a TestStreamWrapperStatus case for each ending would pin the behaviour. Fine as a follow-up if you'd rather keep this PR a pure refactor.

Drafted with Coder Agents.


// Mark completion before yielding so cancellation while the consumer holds
// the finish part leaves finalization to the iterator.
func (o *streamOutcome) observeFinish(step *streamStep, reason fantasy.FinishReason, usage fantasy.Usage) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking readability suggestion: streamStep and streamOutcome each reach into the other. observeFinish takes the step and sets step.complete, and streamStep.finish reads out.finished, out.usage and out.err back out. As a result, the finish part is recorded both in out.finished and in step.complete, and you have to read both types to see they always change together.

Splitting it so streamStep owns the lifecycle and streamOutcome is plain data would keep the behaviour identical:

func (s *streamStep) markComplete() { s.complete.Store(true) }

func (o *streamOutcome) observeFinish(reason fantasy.FinishReason, usage fantasy.Usage) {
	o.finished = true
	o.finishReason = reason
	o.usage = usage
}

func (s *streamStep) finish(status Status, usage, errPayload any, payload func() (resp, metadata any))

In the part switch:

case fantasy.StreamPartTypeFinish:
	out.observeFinish(part.FinishReason, part.Usage)
	step.markComplete()

Each wrapper's finalize then passes out.err, plus &out.usage when out.finished (nil otherwise). A small usageArg() on streamOutcome would save repeating that in both wrappers. complete is still set before yield, and the lock order doesn't change.

Drafted with Coder Agents.

@jscottmiller jscottmiller left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved via Coder Agents on Scott's behalf.

ibetitsmike added a commit that referenced this pull request Oct 20, 2026
…helper (#30934)

Chat turns, manual titles, automatic titles and turn status labels each
created a debug run on a detached, time-bounded context, rebuilt the run
context from the row, and finalized the run by hand. This PR adds
`chatdebug.Service.BeginRun` and `StartRun` and routes all four through
them. It deletes `prepareQuickgenDebugCandidate`, `shortTextCandidate`,
`chatdebugRunContext`, `debugCreateRunTimeout` and a literal 5 second
insert timeout.

- `BeginRun` inserts the run on a context that ignores the caller's
cancellation, with a 5 second limit, and builds the `RunContext` from
its parameters. Chat turns call it and still finalize in
`runnerDebugTurn`.
- `StartRun` wraps `BeginRun` for one model call and returns a context
that carries the run plus a function that finalizes it from the call's
error. Titles and status labels use it.
- New `TestService_StartRun` covers a finalized run, a failed insert and
a failed finalize with a mock store, including the seed summary that
each finalize keeps and the warning that each failure logs.

**Why it is safe:** #30933 pins the insert parameters, the run context,
the finalize status and summary, and the failed insert for all four call
sites. Those tests and the other debug-run, title and quickgen tests
pass before and after. `CreateRun` stores its parameters unchanged, so a
`RunContext` built from the parameters equals one built from the row. A
repo-wide search finds no reference to the deleted symbols. Of 14
mutants of `BeginRun` and `StartRun`, 13 fail `TestService_StartRun`.
They drop either failure warning, report a failed insert as success, let
the insert inherit the caller's cancellation or lose its deadline, drop
the parent chat or the history tip from the run context, drop the seed
summary from the insert or the finalize, ignore the call error or the
finalize timeout, leave the run out of the returned context, or finalize
the wrong run. The survivor shortens the insert limit from 5 seconds to
1.

**Review focus:** log output. Create and finalize failures now log
through the chatdebug logger as `failed to create debug run` and `failed
to finalize debug run`. Before, each call site used its own logger and
message. Compaction still creates its debug run by hand, because #30766
changes that code; it can move to `StartRun` after that PR merges.

**Moved onto the deletion stack:** a merge commit moved this layer onto
the updated #30933 without a force push. It carries one forced change:
#30989 removed the `database/sql` import from
`chatdebug/service_test.go` with the last test that used it, and
`TestService_StartRun` here still uses `sql.NullString`, so the merge
adds the import back. The rest of the diff is unchanged.

Validation:
- `go test -race -count=1 -p 4 -parallel 4 ./coderd/x/chatd/ -run
'^(TestActiveServer_ChatTurnDebugRun.*|TestRunnerDebugTurn.*|TestProposeChatTitle_DebugRun|TestArchiveChatDeletesDebugRuns|TestEditMessageDebugCleanup.*|TestQuickgen.*|TestMaybeGenerateChatTitle.*|TestPrepareManualTitleDebugRunCreateFailure)$'`:
the same 25 tests pass before and after, and again after the move.
- `go test -race -count=1 -p 4 -parallel 4
./coderd/x/chatd/chatdebug/...` passes.
- `golangci-lint run --fix=false ./coderd/x/chatd/chatdebug/...
./coderd/x/chatd/` reports 0 issues, and the pre-commit hook passed.

## Stack

This stack deletes the stale debug-run sweep, then simplifies the
chatdebug debug-run, stream wrapper and step numbering code, drops the
indexes the sweep used, and last deletes a runner debug field that lost
its only reader with the heartbeats. The debug-run and stream wrapper
refactors each sit on a test-only layer that pins the behavior they
keep.

1. #30988 Delete the chat debug run and step heartbeats
2. #30989 Delete `FinalizeStale` and the stale threshold
3. #30995 Delete the `FinalizeStaleChatDebugRows` query
4. #30933 Pin the debug runs that turns, titles and status labels record
5. This PR: Create and finalize debug runs through one helper
6. #30980 Table the stream wrapper tests and cover their status rules
7. #30981 Share one finalize-once lifecycle between the stream wrappers
8. #31007 Number chat debug steps in the database
9. #31008 Delete the process-global step counters
10. #31009 Drop the chat debug stale-sweep indexes
11. #31010 Drop the unused `runnerCtx` from the runner debug turn

> Xum, an AI agent, opened this PR for @ibetitsmike and wrote this
description.
Base automatically changed from mike/chatdebug-stream-wrapper-tests to main October 21, 2026 05:04
ibetitsmike added a commit that referenced this pull request Oct 21, 2026
…ver their status rules (#30980)

Replaces five stream wrapper tests with two tables that run every case
against both `wrapStreamSeq` and `wrapObjectStreamSeq`. #30981 rewrites
both wrappers on one finalize-once lifecycle, and these tests pin what
that rewrite must keep. It deletes
`TestWrapStreamSeq_CompletedNotDowngradedByCtxCancel`,
`TestWrapObjectStreamSeq_CompletedNotDowngradedByCtxCancel`,
`TestWrapStreamSeq_DroppedStreamFinalizedOnCtxCancel` and
`TestDebugModel_StreamObjectRejectsNilSequence`.

- New `TestStreamWrapperStatus` runs 13 cases on each wrapper: a
finished stream, a consumer that stops before or after the finish part,
a stream that ends without one, an error part with and without an error,
an error on a text part, a consumer that stops after an error, a
canceled error part, cancellation before the finish part and while the
consumer holds it, a dropped stream, and a consumer that stops right
after cancellation, which races the context safety net. Each case writes
through a recording store and checks that the step is written exactly
once, plus its status, finish reason, stored usage (all six token
counts, or NULL), error count, last error, and whether the wrapper or
the safety net finalized it.
- `TestDebugModel_StreamRejectsNilSequence` is now one table over
`Stream` and `StreamObject`.
- `TestDebugModel_Stream` also checks the streamed text, not only that
the response has content.

**Why it is safe:** the PR changes only a test file, and the tests pass
on the production code of the layer below. To show they can fail, I ran
46 mutants of both wrappers and the nil-sequence checks. The whole
package kills 17 of them before this PR and all 46 after it. The 29 it
missed include the object wrapper's dropped-stream path, the rule that a
stream without a finish part is interrupted, both error-part rules,
cancellation while the consumer holds the finish part, text that is
never materialized, usage that keeps only the input tokens or is stored
without a finish part, and summary metadata on the safety net's write.
Two more mutants drop the safety net's `finalized` check, so a step can
be written twice. Across repeated batches of 50 runs, the race case
kills them in 49 to 50 of 50 without `-race`. Under `-race` the kill
rate depends on machine load: batches of 50 ranged from 24 to 46 kills.
The unchanged code passes 200 runs with and without `-race`. A copy of
`model.go` with garbage in it fails to build, which shows the mutants
replaced the real file.

**Review focus:** the PR adds 117 more lines than it deletes. It is the
test-first layer for #30981, which deletes 65 production lines.

**Moved onto the deletion stack:** a merge commit moved this layer onto
the updated #30934 without a force push; its only conflict was the test
file's import block (#30988 dropped `time` with the heartbeat tests). A
second commit adds the audit follow-ups: the recording store, the stored
usage check, the race case and the text check.

Validation: `go test -race -count=1 ./coderd/x/chatd/chatdebug/...`
passes, `golangci-lint run --fix=false ./coderd/x/chatd/chatdebug/...`
reports 0 issues, and the pre-commit hook passed.

## Stack

This stack deletes the stale debug-run sweep, then simplifies the
chatdebug debug-run, stream wrapper and step numbering code, drops the
indexes the sweep used, and last deletes a runner debug field that lost
its only reader with the heartbeats. The debug-run and stream wrapper
refactors each sit on a test-only layer that pins the behavior they
keep.

1. #30988 Delete the chat debug run and step heartbeats
2. #30989 Delete `FinalizeStale` and the stale threshold
3. #30995 Delete the `FinalizeStaleChatDebugRows` query
4. #30933 Pin the debug runs that turns, titles and status labels record
5. #30934 Create and finalize debug runs through one helper
6. This PR: Table the stream wrapper tests and cover their status rules
7. #30981 Share one finalize-once lifecycle between the stream wrappers
8. #31007 Number chat debug steps in the database
9. #31008 Delete the process-global step counters
10. #31009 Drop the chat debug stale-sweep indexes
11. #31010 Drop the unused `runnerCtx` from the runner debug turn

> Xum, an AI agent, opened this PR for @ibetitsmike and wrote this
description.
@ibetitsmike
ibetitsmike merged commit 95e6a73 into main Oct 21, 2026
32 of 35 checks passed
@ibetitsmike
ibetitsmike deleted the mike/chatdebug-stream-finalize-once branch October 21, 2026 07:45
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 21, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants