Skip to content

test(coderd/x/chatd): pin the debug runs that turns, titles and status labels record - #30933

Merged
ibetitsmike merged 20 commits into
mainfrom
mike/chatdebug-debug-run-tests
Oct 20, 2026
Merged

ibetitsmike merged 20 commits into
mainfrom
mike/chatdebug-debug-run-tests

Conversation

@ibetitsmike

@ibetitsmike ibetitsmike commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

Adds tests that pin what four call sites record when they create and finalize a debug run: chat turns, manual titles, automatic titles and turn status labels. #30934 moves all four onto one helper. Automatic titles and status labels had no debug-run test, and the chat turn and manual title tests checked only some fields.

  • TestRunnerDebugTurnEnsureCreatesOnce now checks every insert parameter, the full RunContext the model call sees, and that the insert runs on a context that ignores the caller's cancellation but has a deadline.
  • New TestRunnerDebugTurnEnsureCreateFailureDisablesTurn: a failed insert disables debug for the turn, and the turn does not retry it.
  • TestProposeChatTitle_DebugRun also checks that the run has no trigger message, its model config and model, the seed label in its summary, and that it recorded steps.
  • New TestQuickgenDebugRun covers automatic titles and status labels when the model succeeds, when it fails, and when the insert fails. It checks the insert, the run context and the finalize call. Titles record the configured provider and model; status labels record the resolved ones.
  • New TestPrepareManualTitleDebugRunCreateFailure: when the manual title insert fails, the title call gets no run and finishing does nothing.

Why it is safe: the PR changes only tests, and they pass on main's production code. To show they can fail, I ran 34 mutants of the four call sites, and 31 fail the new tests. Two survivors change nothing these tests can observe: the seed summary on the manual title insert (finalize rewrites the summary) and the 10 second finalize timeout for titles and labels. This layer does not pin the third: removing context.WithoutCancel from the title and label insert in quickgen.go. #30934 moves that detach into the shared BeginRun, which the chat turn test pins.

Validation: go test -race -count=1 -p 4 -parallel 4 ./coderd/x/chatd/ -run '^(TestRunnerDebugTurn.*|TestProposeChatTitle_DebugRun|TestQuickgenDebugRun|TestPrepareManualTitleDebugRunCreateFailure)$' passes on this branch and with main's production code, and the pre-commit hook passed.

This layer now sits on #30995 instead of #30932. A merge commit moved it there without a force push, and the layer's own diff is unchanged (same patch-id). The same command and go test -race ./coderd/x/chatd/chatdebug/... pass at the new head.

Stack

This stack deletes the stale debug-run sweep, then simplifies the chatdebug debug-run, stream wrapper and step numbering code, drops the indexes the sweep used, and last deletes a runner debug field that lost its only reader with the heartbeats. The debug-run and stream wrapper refactors each sit on a test-only layer that pins the behavior they keep.

  1. refactor(coderd): delete the chat debug run and step heartbeats #30988 Delete the chat debug run and step heartbeats
  2. refactor: delete chatdebug FinalizeStale and the stale threshold #30989 Delete FinalizeStale and the stale threshold
  3. refactor: delete the FinalizeStaleChatDebugRows query #30995 Delete the FinalizeStaleChatDebugRows query
  4. This PR: Pin the debug runs that turns, titles and status labels record
  5. refactor(coderd/x/chatd): create and finalize debug runs through one helper #30934 Create and finalize debug runs through one helper
  6. test(coderd/x/chatd/chatdebug): table the stream wrapper tests and cover their status rules #30980 Table the stream wrapper tests and cover their status rules
  7. refactor(coderd/x/chatd/chatdebug): share one finalize-once lifecycle between the stream wrappers #30981 Share one finalize-once lifecycle between the stream wrappers
  8. refactor(coderd): number chat debug steps in the database #31007 Number chat debug steps in the database
  9. refactor(coderd/x/chatd/chatdebug): delete the process-global step counters #31008 Delete the process-global step counters
  10. chore(coderd/database): drop the chat debug stale-sweep indexes #31009 Drop the chat debug stale-sweep indexes
  11. refactor(coderd/x/chatd): drop the unused runnerCtx from the runner debug turn #31010 Drop the unused runnerCtx from the runner debug turn

Xum, an AI agent, opened this PR for @ibetitsmike and wrote this description.

Table the step heartbeat tests over the step and run launchers, check the exact touch parameters on each tick, and wait for the ticker to stop after done closes or ctx is canceled. The run launcher had no test before.
…uction

chatd sets the debug stale threshold once, inside debugSvcFactory, before
any caller can see the service. Replace Service.SetStaleAfter with a
WithStaleAfter option and delete the atomic, mutex and wakeup channel
that let running heartbeats pick up a runtime change, along with the
subtest that exercised that path. The step and run heartbeat launchers
now share one loop, and the test-only WithClock option is gone.
…s labels record

Four call sites create and finalize debug runs by hand. Before they share
one helper, pin what each records: every create parameter, the run context
the model call sees, the finalize status and seed summary, and what happens
when the insert fails. Title generation and status labels had no debug-run
test at all; the chat turn test checked only a few fields.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-19T12:17:30.914464Z ac7e32e Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 584e4fb5d2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: 9c018c1b70

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

The heartbeats only refreshed updated_at on in-progress debug runs and
steps so that FinalizeStale would not sweep them. FinalizeStale has no
production caller, so the heartbeats kept nothing alive. Delete both
launchers, Service.TouchStep, Service.TouchRun, the threshold-change
wake-up channel, and the TouchChatDebugStepAndRun and
TouchChatDebugRunUpdatedAt queries with their authz wrappers.
FinalizeStale has no production caller. The stale threshold,
SetStaleAfter, WithClock, the finalize event and its broadcast channel
existed only for it and its tests. Delete them, the SetStaleAfter call
in chatd.go, and the terminal-status alignment test together with
codersdk.ChatDebugTerminalStatuses, which only that test used. Comments
that pointed at the stale sweep now describe dbpurge instead.
FinalizeStale was the only caller, and it is gone. Delete the query, its
dbauthz wrapper and test, and TestFinalizeStaleChatDebugRows. The two
partial indexes that served the query stay until a migration drops them.
…-tests

Moves this layer off #30932 and onto the stale-sweep deletion stack
(#30995) without rewriting history. The tree is the new parent plus this
layer's own diff, which is unchanged.
@ibetitsmike
ibetitsmike removed this pull request from stack #30935 October 18, 2026 18:34
@ibetitsmike
ibetitsmike changed the base branch from mike/chatdebug-stale-after-option to mike/chatdebug-delete-stale-query October 18, 2026 18:34
ibetitsmike added a commit that referenced this pull request Oct 18, 2026
Moves this layer onto the updated #30933, which now sits on the
stale-sweep deletion stack (#30995), without rewriting history. The
tree is the new parent plus this layer's own diff, with one forced
change: the deletion stack removed the database/sql import from
chatdebug/service_test.go because its last user was a deleted test,
and this layer's TestService_StartRun still uses sql.NullString, so
the merge re-adds the import.
@ibetitsmike
ibetitsmike added this pull request to stack #31004 October 18, 2026 18:59
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

Xum, an AI agent, requested this review for @ibetitsmike.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. What shall we delve into next?

Reviewed commit: faea8f416a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…he WithClock doc

This PR deletes the heartbeat tickers, so the WithClock doc no longer
names them.
…alize-stale

# Conflicts:
#	coderd/x/chatd/chatdebug/service.go
… stale sweep

The edit cleanup comment and the notes on removed archive and stop
tests named the stale-finalization sweep, which this PR deletes. The
edit cleanup test now scopes its "only DeleteAfterMessageID removes
the run" argument to the test, since the chat delete cascade and
dbpurge also remove runs.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

Xum, an AI agent, requested this review for @ibetitsmike.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: ac7e32e60a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike
ibetitsmike marked this pull request as ready for review October 19, 2026 12:13
ibetitsmike added a commit that referenced this pull request Oct 20, 2026
Deletes the chat debug heartbeats. Model calls and streams started a
goroutine that touched the step and its run on a ticker, and runner
debug turns started a second one that touched the run. Both existed so
that `FinalizeStale` would not mark a live run as interrupted, but
nothing in production calls `FinalizeStale` since #26270. Mike chose to
delete the stale sweep as dead code. This PR is the first of three that
do it.

- `chatdebug/model.go`: `Generate`, `GenerateObject` and both stream
wrappers no longer start `launchHeartbeat`.
- `active_turn_debug.go`: the runner debug turn no longer starts
`LaunchRunHeartbeat`.
- `chatdebug/service.go`: deletes `TouchStep`, `TouchRun`,
`LaunchRunHeartbeat`, `heartbeatInterval`, and the channel that
`SetStaleAfter` closed to wake the heartbeats. The `WithClock` doc no
longer names the heartbeat tickers.
- Deletes the `TouchChatDebugStepAndRun` and
`TouchChatDebugRunUpdatedAt` queries with their dbauthz wrappers and
tests, and `TestLaunchHeartbeat`. `TestFinalizeStaleChatDebugRows` now
backdates its run with `UpdateChatDebugRun` instead of the deleted
test-only query.

**Why it is safe:** the heartbeats wrote only `updated_at` on
`chat_debug_runs` and `chat_debug_steps`. These read it:

| Reader | Why it does not need the heartbeat |
|---|---|
| `FinalizeStaleChatDebugRows` and the partial indexes
`idx_chat_debug_runs_stale` and `idx_chat_debug_steps_stale` | No
production caller. The next two PRs delete the query, and #31009 drops
the indexes in a migration. |
| `DeleteOldChatDebugRuns` (dbpurge) | See the next paragraph. |
| `DebugPanel.tsx` run sort | Sorts by `started_at \|\| updated_at`, and
`started_at` is `NOT NULL`, so it never falls back to `updated_at`. |
| Debug run and step API responses | Return `updated_at` as data.
Nothing uses it to decide whether a run is alive. |

dbpurge deletes runs whose `updated_at` is older than the debug
retention window, whatever their status, and their steps cascade. The
window is `agents_chat_debug_retention_days`: 30 days by default, and 0
turns the purge off, so dbpurge never deletes a run that was written in
the last day. A run's `updated_at` still moves when the run is created,
when each step is inserted (the `InsertChatDebugStep` CTE), and on every
run update and finalize. So dbpurge could only delete a live run if one
model call plus the tool calls after it ran for more than a day. The
runner finalizes its debug run when the runner context ends, a
`requires_action` wait times out after 5 minutes (the case `TouchRun`
covered), and chatloop aborts a stream that is silent for
`--chat-stream-silence-timeout`, 10 minutes by default (0 turns that
guard off).

Mike accepted the cost: a run or step orphaned by a crashed replica
stays `in_progress` until dbpurge deletes it. That is already true on
`main`, because nothing runs the sweep.

This does not touch the chat worker's `chat_heartbeats` loop that
ARCHITECTURE.md describes, and no ARCHITECTURE.md sentence about debug
runs becomes false.

**Review focus:** nothing uses `WithClock` after this PR, but deleting
it here would take this PR past 500 changed lines, so it goes with
`FinalizeStale` in the next PR.

Validation: `make gen/db`, `go build ./coderd/...`, and `golangci-lint
run` on the touched packages (0 issues). `go test -count=1` passes for
`./coderd/x/chatd/chatdebug/...`, `./coderd/database/dbauthz/ -run
TestMethodTestSuite`, `./coderd/database/ -run ChatDebug`,
`./coderd/x/chatd/ -run Debug` and `./coderd/database/dbpurge/`. The
pre-commit hook passed.

## Stack

This stack deletes the stale debug-run sweep, then simplifies the
chatdebug debug-run, stream wrapper and step numbering code, drops the
indexes the sweep used, and last deletes a runner debug field that lost
its only reader with the heartbeats. The debug-run and stream wrapper
refactors each sit on a test-only layer that pins the behavior they
keep.

1. This PR: Delete the chat debug run and step heartbeats
2. #30989 Delete `FinalizeStale` and the stale threshold
3. #30995 Delete the `FinalizeStaleChatDebugRows` query
4. #30933 Pin the debug runs that turns, titles and status labels record
5. #30934 Create and finalize debug runs through one helper
6. #30980 Table the stream wrapper tests and cover their status rules
7. #30981 Share one finalize-once lifecycle between the stream wrappers
8. #31007 Number chat debug steps in the database
9. #31008 Delete the process-global step counters
10. #31009 Drop the chat debug stale-sweep indexes
11. #31010 Drop the unused `runnerCtx` from the runner debug turn

> Xum, an AI agent, opened this PR for @ibetitsmike and wrote this
description.
ibetitsmike added a commit that referenced this pull request Oct 20, 2026
)

Deletes `chatdebug.Service.FinalizeStale` and everything that existed
only for it. Nothing in production calls `FinalizeStale` since #26270,
and #30988 removed the heartbeats that kept live runs out of its reach.
This is the second of three PRs that delete the stale debug-run sweep.

- `chatdebug/service.go`: deletes `FinalizeStale`, `SetStaleAfter`,
`DefaultStaleThreshold`, the `staleAfterNanos` field and `WithClock`.
The service keeps its real clock for the timestamps it writes.
- `chatd.go`: the debug service factory no longer calls
`SetStaleAfter(inFlightChatStaleAfter * 3)`. `inFlightChatStaleAfter`
stays, because the chat worker uses it.
- `chatdebug/types.go`: deletes `EventKindFinalize` and
`BroadcastPubsubChannel`. `FinalizeStale` was the only publisher without
a chat ID, so `PubsubChannel` loses its `uuid.Nil` branch. Nothing in
production subscribes to `chat_debug:*` channels.
- Deletes the four `FinalizeStale` service tests, and
`TestFinalizeStaleChatDebugRows_TerminalStatusAlignment` together with
`codersdk.ChatDebugTerminalStatuses`, which only that test used. The
test compared that list with the `NOT IN` lists in
`FinalizeStaleChatDebugRows` and nothing else: no other debug query or
constraint lists the statuses. The next PR deletes that query.
- Comments that named the stale sweep now say what happens instead:
`debugCleanupClockSkew`, `errRunFinalized`, `CreateStep`, `UpdateStep`,
the `InsertChatDebugStep` and `UpdateChatDebugStep` queries, the edit
cleanup in `chat_mutations.go`, the two notes on removed tests in
`chatd_internal_test.go`, and four comments in `chatd_test.go`.

**Why it is safe:** `FinalizeStale` and `SetStaleAfter` had no
production caller other than the factory line this PR removes, and
nothing calls `WithClock` after #30988. `make gen/db` changes only the
regenerated query comments. #30988 explains why no reader of
`updated_at` needs the sweep and why dbpurge cannot delete a live run.

**Review focus:** the `debugCleanupClockSkew` comment in
`chatd_debug.go`. The value stays `30 * time.Second`. Runs that start
inside that buffer survive the fast cleanup; the old comment said the
stale sweep would then finalize them, but dbpurge deletes them once they
age past the debug retention window.

Validation: `make gen/db`, `go build ./...`, and `go vet` on the touched
packages. `go test -count=1` passes for
`./coderd/x/chatd/chatdebug/...`, `./coderd/x/chatd/ -run
'Debug|EditMessage|ArchiveChat'`, `./coderd/database/ -run
'ChatDebug|TestIsAuthorizedQuery|TestInsertChatMessagesOrderContract|ConnectionLogs'`
and `./codersdk/`. The pre-commit hook passed.

## Stack

This stack deletes the stale debug-run sweep, then simplifies the
chatdebug debug-run, stream wrapper and step numbering code, drops the
indexes the sweep used, and last deletes a runner debug field that lost
its only reader with the heartbeats. The debug-run and stream wrapper
refactors each sit on a test-only layer that pins the behavior they
keep.

1. #30988 Delete the chat debug run and step heartbeats
2. This PR: Delete `FinalizeStale` and the stale threshold
3. #30995 Delete the `FinalizeStaleChatDebugRows` query
4. #30933 Pin the debug runs that turns, titles and status labels record
5. #30934 Create and finalize debug runs through one helper
6. #30980 Table the stream wrapper tests and cover their status rules
7. #30981 Share one finalize-once lifecycle between the stream wrappers
8. #31007 Number chat debug steps in the database
9. #31008 Delete the process-global step counters
10. #31009 Drop the chat debug stale-sweep indexes
11. #31010 Drop the unused `runnerCtx` from the runner debug turn

> Xum, an AI agent, opened this PR for @ibetitsmike and wrote this
description.
Base automatically changed from mike/chatdebug-delete-stale-query to main October 20, 2026 21:10
ibetitsmike added a commit that referenced this pull request Oct 20, 2026
Deletes the `FinalizeStaleChatDebugRows` query. #30989 deleted
`chatdebug.Service.FinalizeStale`, its only caller. This is the last of
three PRs that delete the stale debug-run sweep.

- Deletes the query from `chatdebug.sql` and regenerates the querier,
dbmock and dbmetrics code. `make gen/db` leaves the dbmetrics wrapper of
a deleted query behind, so I removed it by hand.
- Deletes the dbauthz wrapper and its `TestMethodTestSuite` case, and
`TestFinalizeStaleChatDebugRows` from `querier_test.go`.
- The `codersdk.ChatDebugStatus` comment no longer names the query.

**Why it is safe:** nothing calls the query after #30989. #30988
explains why no reader of the debug `updated_at` columns needs the sweep
and why dbpurge cannot delete a live run.

**Review focus:** the partial indexes `idx_chat_debug_runs_stale` and
`idx_chat_debug_steps_stale` from migration 000468 served only this
query. They stay in this PR; #31009, at the top of this stack, drops
them in a migration.

Validation: `make gen/db` and `go build ./...`. `go test -count=1`
passes for `./coderd/database/dbauthz/ -run TestMethodTestSuite`,
`./coderd/database/ -run ChatDebug`, `./codersdk/` and
`./coderd/database/dbpurge/`. The pre-commit hook passed.

## Stack

This stack deletes the stale debug-run sweep, then simplifies the
chatdebug debug-run, stream wrapper and step numbering code, drops the
indexes the sweep used, and last deletes a runner debug field that lost
its only reader with the heartbeats. The debug-run and stream wrapper
refactors each sit on a test-only layer that pins the behavior they
keep.

1. #30988 Delete the chat debug run and step heartbeats
2. #30989 Delete `FinalizeStale` and the stale threshold
3. This PR: Delete the `FinalizeStaleChatDebugRows` query
4. #30933 Pin the debug runs that turns, titles and status labels record
5. #30934 Create and finalize debug runs through one helper
6. #30980 Table the stream wrapper tests and cover their status rules
7. #30981 Share one finalize-once lifecycle between the stream wrappers
8. #31007 Number chat debug steps in the database
9. #31008 Delete the process-global step counters
10. #31009 Drop the chat debug stale-sweep indexes
11. #31010 Drop the unused `runnerCtx` from the runner debug turn

> Xum, an AI agent, opened this PR for @ibetitsmike and wrote this
description.
@ibetitsmike
ibetitsmike merged commit a1556f8 into main Oct 20, 2026
29 checks passed
@ibetitsmike
ibetitsmike deleted the mike/chatdebug-debug-run-tests branch October 20, 2026 21:43
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 20, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants