Repository navigation
feat: account chatd turn time by category - #29772
jscottmiller wants to merge 54 commits into
Conversation
Docs previewCheck off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here. |
6f34bf7 to
cf518ba
Compare
9e3bdda to
c058ef6
Compare
cf518ba to
8bb2f71
Compare
c058ef6 to
645f343
Compare
8bb2f71 to
c2202f0
Compare
645f343 to
2c98b6d
Compare
|
/coder-agents-review |
|
Chat: Review posted | View chat Review history
deep-review v0.13.0 | Round 7 | Last posted: Round 7, 115 findings (48 P2, 43 P3, 12 P4, 8 Nit, 4 Note), COMMENT. Review Finding inventoryFinding inventory: PR #29772Findings
Law analysisRound 1. Effective LOC +1512 -29 (578 production, 934 test). Head SHA 2c98b6d. Verdict: Don't split. Enforcement: Advisory. Contested and acknowledgedCRF-4 (P3, metrics.go:304) - Unreachable elapsed<0 guard in RecordTurnCategory
CRF-10 (P2, turn_trace.go:221) - Another task's Ensure can close a turn before its final step ends
CRF-17 (P2, stage.go:428) - adoptTurnModel doc narrates the body
CRF-45 (P3, turnaccounting.go:277) - Hook dispatch to external endpoints lands in chatd_overhead
CRF-46 (P3, turnaccounting.go:274) - Tool steps classify decide, hook, finish, and retry bookkeeping as tool_execution
CRF-56 (P3, stage.go:430) - First-model stamp on chat_turn contradicts "a turn is not a model-scoped unit"
CRF-58 (P3, turnaccounting.go:375) - overattributed anomaly carries no marker to follow
CRF-60 (P3, turnaccounting.go:281) - FinishTurn transaction and history read land in chatd_overhead
CRF-61 (P3, stage.go:41) - chatloop owns one value of chatd's generation-action enum
CRF-77 (Note, stage.go:328) - Scope and accumulator are two turn-membership signals
CRF-10 (P2, turn_trace.go:221) - R5 response to the R4 re-raise
CRF-86 (P3, turn_trace.go:292) - Committed turn counted as abandoned
CRF-90 (P4, turnaccounting.go:225) - Root-only use of EndTurn is doc-only
CRF-92 (P4, turnaccounting.go:251) - Recorded-stage subtraction has no production path
Round logRound 1Netero first pass: 6 P3, 1 Nit, 1 Note. Law: don't split (advisory). Panel of 24 (22 trigger-matched, wildcards Kurapika and Knuckle). New: 35 P2 (7 behavioral or structural, 28 comment-quality from Gon), 27 P3, 4 Nit, 4 Note posted. CRF-8 and CRF-79 dropped with reasons. Killua and Kurapika reported no findings. Reviewed against c2202f0..2c98b6d. Round 2 updateBLOCKED. All findings silent. No review. Round 3 updateBLOCKED. CRF-59 and CRF-75 silent. No review. Round 4Churn guard PROCEED: 67 addressed, 1 acknowledged (CRF-10), 9 contested, 0 silent. Round type: restructure (files added to the diff: chatd_test.go, metrics_internal_test.go, export_test.go, generation_retry_internal_test.go; exported API changed: Category renamed TurnCategory, MarkCompleted removed, EndTurn added). Law not run (effective additions 1512 to 1582). Netero resumed (CRF-4 contested). Full panel of 21: dispute check-in agents resumed for Gon, Mafuuu, Chopper, Knov, Kite, Leorio, Luffy, Knuckle, Ryosuke, Meruem; Komugi and Takumi resumed for acknowledged CRF-10 (orchestrator choice: an acknowledged-not-fixed finding by an agent author gets the same check-in); new agents for Ging-Go, Melody, Pen Botter, Killua (specialists), Pariston (core), Bisky (tests), Hisoka, Mafu-san (pool), Razor (wildcard, drawn from Zoro and Razor). Closed: CRF-4, 17, 45, 46, 56, 61, 77. Re-raised: CRF-10 (P2), CRF-14 (P2), CRF-58 (P4), CRF-60 (P4). New: 7 P2, 7 P3, 6 P4, 1 Nit posted; 2 out of scope; 6 verification or praise Notes dropped. Ging-Go and Killua reported no findings. Reviewed against 282311c..a62c34a. Round 5Churn guard PROCEED: 21 addressed, 2 acknowledged (CRF-10, CRF-86), 2 contested (CRF-90, CRF-92), 0 silent. Round type: fix round (every change maps to a round 4 finding; chatloop.go entered the diff for the CRF-82 fix; no file created and no exported signature changed, only the new constant AttrOverattributed). Law not run (effective additions 1582 to 1702). Netero new agent (no contested Netero finding). Panel of 14: resumed Knov, Ryosuke, Meruem, Bisky (contested), Komugi, Takumi (acknowledged); new Ging-Go, Melody, Pen Botter (specialists for changed Go and docs), Gon, Leorio (comments, doc comments, and Help changed), Pariston, Mafuuu (raised CRF-83, whose fix Netero questioned), Chopper (wildcard from Mafuuu, Chopper, Kite, Zoro, Razor). Killua not triggered: no new loop, cache, or allocation pattern. Closed: CRF-86 (1/1, root cause out of scope), CRF-90 (3/3), CRF-92 (3/3). Re-raised: CRF-10 (Komugi; Takumi accepted). New: 6 P2, 5 P3, 4 P4, 2 Nit posted; 1 merged into the CRF-10 reply; 1 out of scope; 4 verification Notes dropped. Ging-Go, Knov, Ryosuke, Takumi, Melody, Pariston, Chopper reported no new findings. Reviewed against 676c4fd..ab6ebb9. Round 6 updateBLOCKED. CRF-109, CRF-110, CRF-112, CRF-114, CRF-115, CRF-117, CRF-126 silent. No review. Round 7 updateBLOCKED. CRF-109 deferred without a ticket. No review. About deep-reviewCRF = Coder Review Finding (P0-P4, Nit, Note)
|
Add a StageTracer that emits one span and one stage_duration_seconds observation per chat lifecycle stage, with typed Stage, Scope, and ChatKind values, a model_stage_duration_seconds histogram for the provider-bound stages labeled by provider_type and model, and a stage_anomalies_total counter for dropped observations. The metric families are registered when the chat-stage-metrics experiment is enabled; spans are emitted regardless. Extract the generation step body into its own function so the step has a boundary for a deferred stage end.
c2202f0 to
7a5cd22
Compare
2c98b6d to
e070f77
Compare
7a5cd22 to
9185854
Compare
e070f77 to
ca0e2f4
Compare
|
/coder-agents-review |
9185854 to
5ffb40e
Compare
ca0e2f4 to
11a1826
Compare
There was a problem hiding this comment.
This PR turns overlapping stage histograms into a disjoint partition of turn wall time, and the core mechanism holds. Children report their full duration to the parent, TTFT nests under stream, the compaction summary stream stays out of TTFT through a nil tracer, and the ScopeTurn gate keeps background work out of a turn (Knov, Razor, Knuckle, and Meruem each verified that gate independently). Cardinality is bounded, TestRunnerTurnSpanCountsOutcomes checks exactly-once counting against closed spans, and the generated metrics docs match the scanner byte for byte. Killua and Kurapika found nothing.
Counts: 35 P2, 33 P3, 5 Nit, 4 Note. 28 of the P2s are comment-quality findings from Gon, and comment quality is a pattern here: the tool-execution rule is stated in four places, the emit-every-category rule in three, and the End, EndWithoutObservation, SetModel, and RecordAs docs never picked up their new turn-accounting side effects.
The behavioral findings share one theme: several categories hold time their labels do not describe. unattributed absorbs requires_action client waits (CRF-9), the finishing step's time when another task's Ensure closes the turn first (CRF-10), per-step state reloads (CRF-59), and task-level retry sleeps (CRF-47). scheduling absorbs prior failed attempts and failover (CRF-11) and the previous turn's runtime through queue_wait (CRF-12). chatd_overhead absorbs external hook dispatch and the FinishTurn transaction (CRF-45, CRF-60). Taken together, unattributed cannot yet serve as an instrumentation-gap signal, and scheduling and chatd_overhead cannot be read as chatd-owned costs. Separately, the outcome counter is split across two packages over a duplicated state machine (CRF-13, raised by ten reviewers), the concurrency rationale and its test describe a topology production never builds (CRF-14), and no chatd test reads the category family (CRF-15, with CRF-1).
Several findings interact. The CRF-13 restructure (have closeLocked own emission and record every outcome) also dissolves CRF-66 and simplifies CRF-56 and CRF-5, and CRF-55's alternative (partition every closed turn) removes the same flags. Dropping turns_total (CRF-54) changes the Help fix in CRF-53. Deleting stageNode.failed (CRF-49) conflicts with one fix option in CRF-48. Ryosuke and Zoro disagree on CRF-61; both positions are in the comment.
Rebase note: this review ran against 2c98b6dab. The head is now ca0e2f4da, rebased onto the updated base branch. Within this PR's own patch only the stage_anomalies_total Help string changed. Line numbers in stage.go, metrics.go, turn_trace.go, and stage_internal_test.go shifted because of base-stack changes, so some inline comments may show as outdated. The base changes affect CRF-48, CRF-49, CRF-63, and CRF-69; each of those comments says how.
Law assessed decomposition and advises against splitting: the partition, the outcome counter, the model stamp, and the generation-action plumbing form one reviewable idea.
CI: the failed Storybook job aborted after the snapshot service returned HTTP 500 on 11 uploads. It is unrelated to this PR, which touches no site/ files, and a rerun should clear it.
PR description: "every category has a series: 16 per chat kind" reads as a category count. There are 11 categories; 16 is 11 category series plus 4 outcome series plus 1 turns_total series (Mafu-san; Chopper confirmed the arithmetic).
From Hisoka: "I pulled the partition apart at every seam I could find. Most of it held."
coderd/x/chatd/generation.go:443
P3 [CRF-59] From step 2 on, the full-history state reload between steps lands in unattributed. It is uncovered persistence time, not an instrumentation remainder. (Knuckle)
Each later step first runs
loadGenerationState(generation.go:633-660) on the plain taskctx. That opens a transaction, takesSELECT ... FOR SHAREon the chat row (chatstate/machine.go:264), and reads the whole history withGetChatMessagesByChatID(AfterID: 0). Only after that doesEnsurehand back the turn context.
Capture
loadStartandloadEndaroundloadGenerationState, and pass them intoEnsure. In theif t.openbranch (turn_trace.go:119),Recorda reconstructed turn-scoped stage (for exampleload_state) over that window. Then map it inrecordedStageCategoriestoCategoryPersistence.
Knuckle had no production data, so the magnitude is predicted, not measured. The cost grows with chat length, so unattributed trends upward on long chats while persistence stays flat.
🤖
coderd/x/chatd/chatloop/stage.go:406
P3 [CRF-67] The End doc says End closes the span and records its duration, but End now also stamps the turn model on chat_turn, folds the stage into the turn partition, and, on chat_turn, emits the turn's metrics. (Gon P3, Leorio Nit)
The same gap appears in three sibling docs whose declarations this PR changed.
SetModel(:374) does not say that the first model set inside a turn names the chat_turn span. TheStageSpantype doc (:186) still says "End must be called exactly once; the duration observation happens there", which contradicts End's own "calls after the first are ignored" and leaves out the turn report.RecordAs(:472) does not say that turn-scoped recorded stages feed scheduling. (Gon)
A caller who adds
SetModelto a new turn-scoped stage that runs beforepreparechanges which model the whole turn is labelled with, and the doc gives no warning. (Leorio)
Line numbers refer to 2c98b6dab; the rebase shifted stage.go.
🤖
coderd/x/chatd/chatloop/stage.go:417
P3 [CRF-68] The EndWithoutObservation doc says it "makes no duration observation", but it now feeds the stage's full elapsed time into turn_time_seconds_total, and "exactly as End does" is false because it skips adoptTurnModel. (Leorio P3, Gon P3)
The next developer reads "no duration observation", sees a duration landing in a counter, and "fixes" the inconsistency. After that fix, failed-TTFT time silently moves into
streaming, because the stream is no longer marked failed. (Leorio)
Leorio's proposed doc:
// EndWithoutObservation closes the stage span like End but makes no
// histogram observation. It is for stages whose window is only
// comparable across runs when it completed, so a truncated window
// would skew the histogram while the span still needs to report the
// failure. The window still reports to the turn's time partition, so
// a failed time_to_first_token lands in provider_error rather than
// in its stream's own time.🤖
coderd/x/chatd/turn_trace.go:280
Note [CRF-75] A promoted turn's partition overlaps the tail of the turn that released it, so summing turn_time_seconds_total across turns counts that wall time twice. (Hisoka)
The ratio
rate(time)/rate(turns)means what the PR says it means, butsum(rate(turn_time_seconds_total))is not "chatd busy seconds". The same thing happens acrosschat_kind: a parent'stool_executionincludes the subagent turns it waited on, and those subagent turns are also accounted underchat_kind="subagent". Anyone building a capacity dashboard on these counters needs to know this.
One sentence in the ARCHITECTURE section would stop anyone building a capacity panel on the sum.
🤖
🤖 This review was automatically generated with Coder Agents.
|
/coder-agents-review |
There was a problem hiding this comment.
This PR partitions the wall time of completed chatd turns into categories on turn_time_seconds_total and adds turn_outcomes_total and turns_total; 77 findings are open: 35 P2, 33 P3, 5 Nit, 4 Note.
Head 11a18260 rebases 2c98b6dab onto the new base; turnaccounting.go, turnaccounting_internal_test.go, and generation.go are unchanged, and no review thread has a reply. Further review is blocked until fixes are pushed or the threads get replies. For each finding below, reply on its thread with the fixing commit, a GitHub issue that tracks it, or the reason the code should stay. Line numbers are from 2c98b6dab, where the threads are anchored.
The failing doc-check job (its Coder user is suspended) and Storybook job (snapshot service HTTP 500, "api key lookup failed") are not caused by this PR.
P2, behavior and structure:
turnaccounting.go:372: the requires_action client wait stays inside the turn and is counted asunattributed[CRF-9]turn_trace.go:221: another task'sEnsurecan close a finished turn before its final step ends, so where the step's time is counted depends on goroutine scheduling [CRF-10]turnaccounting.go:73: acquisition re-covers earlier attempts (reopen after invalidation, failover) and counts them asscheduling[CRF-11]turnaccounting.go:74:queue_waitinschedulingmixes pickup latency with the previous turn's remaining runtime [CRF-12]turnaccounting.go:111: the accumulator'scompletedandinvalidflags duplicate therunnerTurnSpanoutcome, andturn_outcomes_totalis counted from two packages [CRF-13]turnaccounting.go:105: the concurrency comment names tool calls, which never reach the accumulator, and the concurrency test asserts an overattributed turn [CRF-14]stage_internal_test.go:457: no chatd test readsturn_time_seconds_total; removing the accumulator fromcontextLockedfails no test [CRF-15]
P2, comments that restate the code or state a false or duplicated fact (one finding per comment):
stage.go: 324 [CRF-16], 428 [CRF-17]turnaccounting.go: 9 [CRF-18], 30 [CRF-19], 47 [CRF-20], 65 [CRF-21], 77 [CRF-22], 101 [CRF-23], 115 [CRF-24], 133 [CRF-25], 181 [CRF-26], 213 [CRF-27], 265 [CRF-28], 348 [CRF-29]generation.go: 128 [CRF-30]turn_trace.go: 146 [CRF-31], 162 [CRF-32], 190 [CRF-33], 227 [CRF-34]turnaccounting_internal_test.go: 70 [CRF-35], 76 [CRF-36], 104 [CRF-37], 119 [CRF-38], 171 [CRF-39], 217 [CRF-40], 297 [CRF-41], 352 [CRF-42], 691 [CRF-43]
P3:
generation.go:575: no test drivesrunGenerationStepinto the turn partition [CRF-1]turnaccounting_internal_test.go:434: theNilTracersubtest passes because of nil-receiver guards, not accounting logic [CRF-2]turnaccounting_internal_test.go:692: hand-written registry walkers duplicate thesumshelper andpromhelp[CRF-3]metrics.go:304: the unreachableelapsed < 0guard inRecordTurnCategorywould drop a category series without any signal [CRF-4]turnaccounting.go:33:TurnTimeCategories(a mutable slice) andTurnAccumulator.Modelare exported with no callers outsidechatloop[CRF-5]turn_trace.go:28: therunnerTurnSpandoc dropped the fact that stages still open atCompleteend inside the turn span [CRF-6]turnaccounting_internal_test.go:116: the main partition fixture recordsqueue_waitbefore the turn starts, and the sum check passes because two errors cancel [CRF-44]turnaccounting.go:277: hook dispatch to external endpoints is counted aschatd_overhead(ortool_execution) [CRF-45]turnaccounting.go:274: tool steps count decide, hook, finish, and retry bookkeeping astool_execution[CRF-46]turnaccounting.go:284:retry_backoffcovers only generation retries; phase and task-level backoff are counted in other categories [CRF-47]turnaccounting.go:292: a failed attempt's TTFT and discarded responses are counted astime_to_first_tokenorstreaming, which contradicts the ARCHITECTURE.md table [CRF-48]turnaccounting.go:287:stageNode.failedhas no test that fails without it and moves only microseconds [CRF-49]turnaccounting.go:317: theown < 0clamp duplicates theaddCategoryguard, soTestTurnAccountingClampsOwnTimecannot detect its removal [CRF-50]metrics.go:191: theturn_time_seconds_totalHelp says categories sum to the turn duration, which is false for overattributed turns [CRF-51]metrics.go:191: theturn_time_seconds_totalHelp lists no category values or definitions [CRF-52]metrics.go:203: theturn_outcomes_totalHelp says completed turns are the ones with a recorded partition, which is false fornonpositive_turn, and no test checks the difference [CRF-53]metrics.go:193:turns_totalduplicatesturn_outcomes_total{outcome="completed"}except after a clock step [CRF-54]turn_trace.go:245: the partition covers only completed turns, so time in turns that end in a provider error or an interrupt is counted in no category [CRF-55]stage.go:430: stamping the first model onchat_turncontradicts "a turn is not a model-scoped unit" [CRF-56]turnaccounting.go:54:attributingStagesand thecategoryswitch must list the same stages, and a mismatch fails no test [CRF-57]turnaccounting.go:375: theoverattributedanomaly has nochat_kindlabel or span attribute that identifies the turn [CRF-58]generation.go:443: from step 2 on,loadGenerationState(a full history read) is counted asunattributed[CRF-59]turnaccounting.go:281: theFinishTurntransaction and theafterGenerationOutcomehistory read are counted aschatd_overhead, notpersistence[CRF-60]stage.go:41:chatloopnow owns one value of chatd's generation-action enum, andSetGenerationActionhas an effect only ongeneration_step[CRF-61]stage_internal_test.go:668: theafterInterruptturn anchored attime.Now()can have zero duration on coarse clocks [CRF-62]turnaccounting_internal_test.go:33:newTurnFixtureduplicatesnewStageMetricsFixture[CRF-63]turnaccounting.go:13:Categorylacks theTurnprefix its sibling types use and shares a name withrefusal.Category[CRF-64]turnaccounting.go:340:recordAttributionhandles exactly the stages that are not inattributingStages[CRF-65]turnaccounting.go:304:reportemits the whole turn's metrics, but its name and doc describe folding one stage into its parent [CRF-66]stage.go:406: theEnd,SetModel,StageSpan, andRecordAsdocs omit the new turn-accounting effects [CRF-67]stage.go:417: theEndWithoutObservationdoc says "exactly as End does" and "no duration observation", and both are false now [CRF-68]metrics.go:291: theRecordStageAnomalydoc dropped "adjusted", and theRecordAnomalydoc (stage.go:506) was not updated [CRF-69]stage.go:197: theaccfield doc is wrong for background-scoped stages that run inside a turn [CRF-70]
Nit:
capacity_internal_test.go:23(andturn_trace_internal_test.go:119,:144): thenewStageTestTracertonewStageMetricsTracerswaps discard the registry and change no behavior [CRF-7]turnaccounting_internal_test.go:106:syntheticTurnreturns an accumulator no caller uses [CRF-71]ARCHITECTURE.md:844: "emit nothing" is contradicted two lines later by outcome counting [CRF-72]turnaccounting_internal_test.go:532: usewg.Go[CRF-73]turnaccounting.go:197: usemaps.Clone[CRF-74]
Note:
turn_trace.go:280: summing categories across turns or chat kinds counts overlapping wall time twice [CRF-75]ARCHITECTURE.md:852: thequeue_waitrecorded byPromoteQueuednever reachesscheduling[CRF-76]stage.go:328: the scope and the accumulator are two turn-membership signals that can disagree [CRF-77]turnaccounting.go:342: theacc == nilbranch ofrecordAttributionhas nochatlooptest coverage [CRF-78]
🤖 This review was automatically generated with Coder Agents.
|
/coder-agents-review |
There was a problem hiding this comment.
This PR partitions the wall time of every chatd turn into categories on turn_time_seconds_total. The new commits and replies answer 17 of the 18 open findings; the reply on the stand-in tracer finding contests it, and the reviewers who raised it will judge that defense when review resumes.
One P3 remains. Its reply accepts the behavior as a limitation and leaves the fix for a follow-up with no ticket. Further review is blocked until it has a fix in this PR, a linked GitHub issue that tracks the follow-up, or a maintainer's explicit acceptance of the gap. The line number is from ff9b9809.
- P3 [CRF-109]
turnaccounting.go:207: any error that wrapscontext.Canceledskipsprovider_error, so when the silence guard or a transport reset cancels a stream while the turn is live, up toStreamSilenceTimeoutof provider stall per attempt is counted asstreaming. The documented rule gives a chat stop and a shutdown as its examples of a context cancellation and does not mention either provider-side case.
🤖 This review was automatically generated with Coder Agents.
- Delete the failed flag on stage nodes: a stream's own time is provider_error only when the stream ends with an error, so the rest of a stream whose first-token window failed is streaming. - Take a recorded stage's window out of the own time of the attributing stage it is recorded under. - Extend the stage set consistency test to the turn accounting sets and the turn_time_seconds_total help text. - Remove capacity_wait from the accounting docs and tests.
…k overattributed turns
…n accounting limits
… end Closing a turn fixes its end time on the accumulator. A stage that reports afterwards counts only the part of its window before that time, for parents and children alike, so a step still unwinding when an interrupt settles its turn keeps its category up to the interrupt instead of landing in unattributed. EndTurn takes the turn's end time. Generated by Coder Agents.
…mits Generated by Coder Agents.
…ng tests Generated by Coder Agents.
ff9b980 to
dd49b8e
Compare
1be3031 to
93913d8
Compare
**TL;DR:** Adds a stage tracer that turns each chat lifecycle stage into
both an OpenTelemetry span and a Prometheus histogram observation, so
traces and metrics always agree. The metrics are only registered with
the `chat-stage-metrics` experiment. Nothing calls the tracer until the
next PRs.
Adds `chatloop.StageTracer`, which emits each chat lifecycle stage as an
OpenTelemetry span and a `coderd_chatd_stage_duration_seconds{stage,
scope, chat_kind}` observation from one `End` call, so trace and metric
durations cannot disagree. The provider-bound stages (`stream`,
`provider_attempt`) are also observed on
`coderd_chatd_model_stage_duration_seconds{stage, provider_type,
chat_kind, model}`, the only family carrying `provider_type` and
`model`, and only for turn-scoped stages. Time to first token per model
stays on the existing `coderd_chatd_ttft_seconds`. Nothing calls the
tracer yet; the next PRs instrument the provider call and the runner.
- The families are registered only when the `chat-stage-metrics`
experiment is enabled (safe and known, so
`--experiments=chat-stage-metrics` or `--experiments=*` turns it on).
Spans are emitted regardless. `NewMetrics` leaves the families off;
`chatd.New` opts in from the experiment, and a test asserts the
experiment decides whether they reach the server's registry.
- `Stage`, `Scope`, `ChatKind`, and `StageAnomaly` are typed string
constants. `scope` separates latency attributable to a prompt (`turn`)
from detached `background` work; `chat_kind` is `root` or `subagent`,
empty without a known chat.
- `StageModel` carries the wire `provider`, `provider_type`, `model`,
and `reasoning_effort`. All are span attributes; only `provider_type`
and `model` are metric labels, on the model family. No family carries an
organization label.
- Observed stages: `chat_turn`, `queue_wait`, `acquisition`,
`mcp_connect`, `stream`, `time_to_first_token`, `provider_attempt`,
`tool_call`, `commit`, `retry_backoff`. `generation_step`, `prepare`,
`thinking`, and `compaction` are span-only. A test keeps the stage sets
and help text consistent.
- API: `Start` / `StartRootAt` for live spans (span timestamps come from
the tracer clock), `Record` for stages reconstructed from timestamps
(the scope comes from the context). `End` is idempotent;
`EndWithoutObservation` ends the span without a sample. A nil tracer,
provider, or metrics discards.
- `coderd_chatd_stage_anomalies_total{reason}` counts observations
dropped, adjusted, or inconsistent (`negative_elapsed`,
`inverted_window`, `missing_timestamp`, `future_start`, `stale_anchor`).
Span-only stages never feed it.
- Buckets are 12 round edges (`0.1 0.25 0.5 1 2.5 5 10 30 60 300 1800
3600`).
- Moves the body of the `for` loop in `StartGeneration` into
`runGenerationStep` so the step has a function boundary for a deferred
stage end. Control flow and error wrapping are unchanged.
<details>
<summary>Series cost (approximate, whole stack)</summary>
A histogram label set is 12 edges + `+Inf` + `_sum` + `_count`: 15
series. Per replica, each `chat_kind` costs about 150 stage series and 4
turn outcome counters; each `(chat_kind, model)` costs 30 (2 stages x
15). Both chat kinds on `M` models: about `308 + 60*M` per replica, plus
background-scope stage series.
Dogfood `main`, 3 replicas: ~3.4k `coderd_*` series per replica (~10.1k
total).
| Models used per replica | Added series (3 replicas) | vs. baseline |
|---:|---:|---:|
| 1 | ~1.1k | +11% |
| 5 | ~1.8k | +18% |
| 10 | ~2.7k | +27% |
| 35 | ~7.2k | +72% |
Most deployments run four or fewer models, so a typical replica adds
roughly 370 to 550 series. The experimental turn accounting in #29772
would add 88 more per replica (11 categories x 4 outcomes x 2 chat
kinds). Series appear lazily and do not grow with organizations.
</details>
<details>
<summary>Review decisions</summary>
- Experiment gate only; no deployment flag or metric levels. If the
experiment graduates, the families become always on like the other chatd
families; a flag is added then only if dogfood shows a cardinality
problem.
- Organization is on neither metrics nor spans, and `effort` is
span-only; `provider_type` (configured type), not the wire `provider`,
is the model-family label.
- `time_to_first_token` is not on the model family; per-model TTFT is
`ttft_seconds`, which observes the same measured window.
- The model family records only turn-scoped stages.
- Nil guards are kept so a nil or zero tracer is always safe to call.
- HTTP attribute keys stay snake_case like the rest of the chatd span
attributes.
</details>
Supersedes #28968 and #28969.
🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.
**TL;DR:** Wraps each prompt in a `chat_turn` root span with every lifecycle stage parented under it. The span records one outcome per turn (`completed`, `interrupted`, `error`, `abandoned`), and the new `coderd_chatd_turn_outcomes_total` counter counts those outcomes. Turns stay correct when tasks are canceled, replaced, or retried. Each prompt runs inside a `chat_turn` root span owned by the runner (`runnerTurnSpan`). The generation step, prepare, MCP connect, stream, TTFT, provider attempt, commit, thinking, tool call, compaction, and retry backoff stages are parented under it, activating the stages from #29770. `chat_turn` spans are standalone roots: the turn runs on a worker, often on another replica, and no trace context is persisted. **Lifecycle** - **Start.** Each generation loop iteration calls `Ensure`, which opens the turn if none is open and returns a `turnToken`. The span is backdated to the trigger: the latest of the last user prompt, a dynamic tool's result message after it, and a pending compaction request. An `acquisition` stage covers trigger to pickup. - **Stale trigger / takeover.** A trigger at or before the previous anchor opens the turn at now with no `acquisition` and counts `stale_anchor`. A runner that took the chat over from an owner with a stale heartbeat (`TakenOver`, from acquisition) opens its first turn at now with no `acquisition`, so the dead owner's run is not reported as pickup delay. - **Newer prompt.** `Ensure` on the current turn with a newer trigger closes it and opens a new one. The closed turn is `abandoned` unless the task that ran it records an outcome before releasing it. - **Finish.** After `FinishTurn` or `EnterRequiresAction` commits, the step calls `Complete`; `Settle` closes the span after the step's own stage ends, so the finishing step is inside the turn. A chat waiting in `requires_action` is outside any turn; the generation resumed by its tool results opens a new one. - **Emission.** Closing a turn fixes its end time; its span is emitted once every task that joined it (`Ensure` takes the task ID) has called `Release`, which `runTask` does after the task returns. A finishing task whose `FinishTurn` promoted a queued prompt therefore still counts its turn as `completed` even when the promoted task's `Ensure` runs before its `Complete`. `End` at runner exit emits whatever is left. - **Failure.** A step that commits `FinishError`, or fails closed, invalidates the turn as `error` after the commit. Errors the task runner retries, including task timeouts, keep the same turn open; an expected, non-retryable exit records no outcome and leaves the turn open until the interrupt task, a newer prompt, or runner exit closes it. A canceled generation task records no outcome and cannot obtain a turn token, so a replaced task never touches its successor's turn. - **Interrupt.** The interrupt task marks the open turn `interrupted` before `FinishInterruption` commits and closes it after. Only a stop (by the user or a parent agent) is `interrupted`; it ends with unset span status, since error status is reserved for `error` turns. runner shutdown, loss of ownership, and message edits close the turn as `abandoned` with unset status. - **Promotion.** Every promoting transition (`FinishTurn`, `FinishInterruption`, a send to an errored chat, `PromoteQueued`) returns the promoted message's queued time in its result, and one `recordQueueWait` helper records a standalone `queue_wait` to the promotion. The promoted turn opens at the promotion time. `chat_turn` is observed on `coderd_chatd_stage_duration_seconds` only for completed turns, so the histogram is completed-turn latency; every turn still emits its span. `coderd_chatd_turn_outcomes_total{outcome, chat_kind}` counts every closed turn exactly once (`completed`, `interrupted`, `error`, `abandoned`), matching the span's `turn_outcome`, under the `chat-stage-metrics` experiment. Turn time accounting is in draft #29772. **Stages and identity** - `tool_call` is a live stage around each local tool call in `chatloop`; the tool runs on its context, and it ends errored only when the tool fails to execute. Calls that never run (inactive, unknown, or rejected by policy) record no stage. - `mcp_connect` carries connected and failed server counts and ends errored only when servers were configured and none connected. - The compaction stage carries the model that runs the summary. - `withStageIdentity` puts scope and chat kind on the turn context. Detached background work runs in `background` scope as its own trace roots. - The `advisor` tool's nested model call appears only as its enclosing `tool_call`. ARCHITECTURE.md gains a "Lifecycle tracing" section. <details> <summary>Review decisions</summary> - Turn identity stays timestamp-based with targeted fixes (clamped acquisition, rotation on a newer trigger, invalidate after commit, fail-closed invalidation, settle closing invalidated turns) plus a takeover flag; keying turns on message IDs would not detect takeover and cannot express interrupted/abandoned outcomes. - Outcome classification: the interrupt task owns `interrupted`, and cancellation or a fence mismatch alone never records an outcome, so shutdown, ownership loss, and edits are `abandoned`. Only `error` turns set error span status. Remaining ordering races between a step and an interrupt are documented as known limitations in ARCHITECTURE.md. - Turns are emitted after the tasks that ran them return, not when they close, so an outcome a replaced task records while unwinding reaches its own turn; the successor task is not delayed. Chosen over an after-commit, pre-publish hook on `ChatMachine.Update`, which would fix only the `FinishTurn` ordering. - `requires_action` ends the turn rather than pausing it, so `chat_turn` measures chatd time; a prompt with client tools counts as several turns. - Promoted turns anchor at promotion time and `queue_wait` is always standalone, so `chat_turn` never includes time spent queued behind another turn. - The former `capacity_wait` stage (#29805) was dropped: it was a per-worker lower bound, excluded from accounting, and needed its own SQL and bookkeeping. #29805's remaining helpers are folded into this PR. </details> Supersedes #28972, #28973, and #29805. 🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.
TL;DR: Experimental. When a turn closes, splits its wall time into categories (scheduling, time to first token, streaming, provider errors, tools, preparation, persistence, chatd overhead, unattributed) and emits them on
coderd_chatd_turn_time_seconds_total. This shows where users' wait time goes in aggregate.Note
Experimental follow-up, not part of the lifecycle tracing stack (#29769, #29770, #29771). Based on #29771, which now owns
turn_outcomes_total; this PR adds only the time partition on top of it.When a turn closes, whatever its outcome, its wall time is partitioned into disjoint categories and emitted once.
runnerTurnSpandecides the outcome when it emits the span and passes it toStageSpan.EndTurn, which stampsturn_outcome, emits the partition, and counts the outcome onturn_outcomes_total(from #29771). The turn families carry nomodelor organization label: a turn is not a model-scoped unit, and per-model time is on the stage families. The first model a turn resolves is stamped on thechat_turnspan.turn_time_seconds_total{category, chat_kind, outcome}turn_outcomes_totalwith the same labels, mean seconds per turn by categoryFilter
outcome="completed"for turns that finished normally.interruptedis a user stop; shutdown, ownership loss, and edits areabandoned. Content-filter and no-output attempts count asprovider_error; a stream ended by cancellation (stop, edit, shutdown) keeps its normal category. A turn whose categories exceed its duration carriesoverattributed=trueon its span. A parent'stool_executionincludes time spent waiting on subagent turns, which are also accounted underchat_kind="subagent", so query one chat kind at a time. Every category is emitted for every accounted turn, so every category has a series. A turn with a non-positive duration counts its outcome but records no partition (stage_anomalies_total{reason="nonpositive_turn"}). Per-turn distributions and per-stage totals are left to traces.How the partition is built
Nested stages share wall time with their parents, so summing durations over-counts. Each stage's own time (duration minus its attributing children) goes to one
TurnCategory: aTurnAccumulatorrides on the turn context, each attributing stage reports its full duration to its parent when it ends, and the parent's category receives what is left.Only turn-scoped stages report to the accumulator, so background work never lands in a turn, and
queue_wait(recorded outside the turn) never reaches a category. A recorded stage inside an attributing stage is subtracted from that stage's own time. Categories that sum to more than the turn are emitted as measured and counted asoverattributed. From the second step on, the history load before each step lands inunattributed. Closing a turn fixes its end time on the accumulator, and a stage that ends later counts only its window before that time, so a step still unwinding at an interrupt keeps its category up to the interrupt. A test keeps the stage-to-category maps and help text consistent.ARCHITECTURE.md gains a "Turn accounting" section.
Review decisions
outcomelabel;turns_totalwas removed becauseturn_outcomes_totalis the denominator.schedulingisacquisitiononly, sincequeue_waitis never inside a turn.failedflag was removed: it only moved microseconds between categories in production.overattributed.Supersedes #28974.
🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.