Skip to content

feat: account chatd turn time by category - #29772

Closed
jscottmiller wants to merge 54 commits into
scott/chatd-lifecycle/3-turn-spanfrom
scott/chatd-lifecycle/4-turn-accounting
Closed

jscottmiller wants to merge 54 commits into
scott/chatd-lifecycle/3-turn-spanfrom
scott/chatd-lifecycle/4-turn-accounting

Conversation

@jscottmiller

@jscottmiller jscottmiller commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

TL;DR: Experimental. When a turn closes, splits its wall time into categories (scheduling, time to first token, streaming, provider errors, tools, preparation, persistence, chatd overhead, unattributed) and emits them on coderd_chatd_turn_time_seconds_total. This shows where users' wait time goes in aggregate.

Note

Experimental follow-up, not part of the lifecycle tracing stack (#29769, #29770, #29771). Based on #29771, which now owns turn_outcomes_total; this PR adds only the time partition on top of it.

When a turn closes, whatever its outcome, its wall time is partitioned into disjoint categories and emitted once. runnerTurnSpan decides the outcome when it emits the span and passes it to StageSpan.EndTurn, which stamps turn_outcome, emits the partition, and counts the outcome on turn_outcomes_total (from #29771). The turn families carry no model or organization label: a turn is not a model-scoped unit, and per-model time is on the stage families. The first model a turn resolves is stamped on the chat_turn span.

Family What
turn_time_seconds_total{category, chat_kind, outcome} an exclusive partition of turn wall time; divided by turn_outcomes_total with the same labels, mean seconds per turn by category

Filter outcome="completed" for turns that finished normally. interrupted is a user stop; shutdown, ownership loss, and edits are abandoned. Content-filter and no-output attempts count as provider_error; a stream ended by cancellation (stop, edit, shutdown) keeps its normal category. A turn whose categories exceed its duration carries overattributed=true on its span. A parent's tool_execution includes time spent waiting on subagent turns, which are also accounted under chat_kind="subagent", so query one chat kind at a time. Every category is emitted for every accounted turn, so every category has a series. A turn with a non-positive duration counts its outcome but records no partition (stage_anomalies_total{reason="nonpositive_turn"}). Per-turn distributions and per-stage totals are left to traces.

How the partition is built

Nested stages share wall time with their parents, so summing durations over-counts. Each stage's own time (duration minus its attributing children) goes to one TurnCategory: a TurnAccumulator rides on the turn context, each attributing stage reports its full duration to its parent when it ends, and the parent's category receives what is left.

chat_turn                                        own time -> category
├── acquisition                                  full     -> scheduling
└── generation_step (repeats per step)           own      -> chatd_overhead, or
    │                                                        tool_execution when the
    │                                                        step ran local tools
    ├── prepare                                  own      -> preparation
    │   └── mcp_connect                          own      -> preparation
    ├── retry_backoff (incl. phase backoff)      own      -> retry_backoff
    ├── stream                                   own      -> streaming, or
    │   │                                                    provider_error if it errored
    │   ├── time_to_first_token                  own      -> time_to_first_token, or
    │   │                                                    provider_error without output
    │   └── provider_attempt  (overlaps TTFT)    not categorized
    ├── thinking              (window inside stream) not categorized
    ├── tool_call             (inside step)      not categorized
    ├── commit                                   own      -> persistence
    └── compaction                               own      -> compaction
                                                 remainder -> unattributed

Only turn-scoped stages report to the accumulator, so background work never lands in a turn, and queue_wait (recorded outside the turn) never reaches a category. A recorded stage inside an attributing stage is subtracted from that stage's own time. Categories that sum to more than the turn are emitted as measured and counted as overattributed. From the second step on, the history load before each step lands in unattributed. Closing a turn fixes its end time on the accumulator, and a stage that ends later counts only its window before that time, so a step still unwinding at an interrupt keeps its category up to the interrupt. A test keeps the stage-to-category maps and help text consistent.

ARCHITECTURE.md gains a "Turn accounting" section.

Review decisions
  • Every outcome emits its partition, with an outcome label; turns_total was removed because turn_outcomes_total is the denominator.
  • scheduling is acquisition only, since queue_wait is never inside a turn.
  • The per-stream failed flag was removed: it only moved microseconds between categories in production.
  • Stages that end after their turn closed are clamped to the turn's end time rather than dropped. Emission waits for the tasks that ran the turn to return (feat: open a chat_turn span per prompt #29771), so the clamped time is counted and the partition still sums to the turn without firing overattributed.

Supersedes #28974.

🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Docs preview

Check off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here.

@jscottmiller
jscottmiller force-pushed the scott/chatd-lifecycle/3-turn-span branch from 6f34bf7 to cf518ba Compare September 23, 2026 04:06
@jscottmiller
jscottmiller force-pushed the scott/chatd-lifecycle/4-turn-accounting branch from 9e3bdda to c058ef6 Compare September 23, 2026 04:06
@jscottmiller
jscottmiller force-pushed the scott/chatd-lifecycle/3-turn-span branch from cf518ba to 8bb2f71 Compare September 23, 2026 04:25
@jscottmiller
jscottmiller force-pushed the scott/chatd-lifecycle/4-turn-accounting branch from c058ef6 to 645f343 Compare September 23, 2026 04:25
@jscottmiller
jscottmiller marked this pull request as ready for review September 23, 2026 04:26
@jscottmiller
jscottmiller force-pushed the scott/chatd-lifecycle/3-turn-span branch from 8bb2f71 to c2202f0 Compare September 23, 2026 14:24
@jscottmiller
jscottmiller force-pushed the scott/chatd-lifecycle/4-turn-accounting branch from 645f343 to 2c98b6d Compare September 23, 2026 14:24
@jscottmiller

Copy link
Copy Markdown
Contributor Author

/coder-agents-review

@coder-agents-review

coder-agents-review Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Chat: Review posted | View chat
Requested: 2026-09-29 02:23 UTC by @jscottmiller

Review history
  • R2 (2026-09-24), 5 Nit, 4 Note, 35 P2, 33 P3, COMMENT. Review
  • R3 (2026-09-25), 5 Nit, 4 Note, 35 P2, 33 P3, COMMENT. Review
  • R4 (2026-09-25): 21 reviewers, 6 Nit, 4 Note, 42 P2, 38 P3, 8 P4, COMMENT. Review
  • R5 (2026-09-28): 14 reviewers, 8 Nit, 4 Note, 48 P2, 43 P3, 12 P4, COMMENT. Review
  • R6 (2026-09-29), 8 Nit, 4 Note, 48 P2, 43 P3, 12 P4, COMMENT. Review
  • R7 (2026-09-29), 8 Nit, 4 Note, 48 P2, 43 P3, 12 P4, COMMENT. Review

deep-review v0.13.0 | Round 7 | 562baea..ff9b980

Last posted: Round 7, 115 findings (48 P2, 43 P3, 12 P4, 8 Nit, 4 Note), COMMENT. Review

Finding inventory

Finding inventory: PR #29772

Findings

# Sev Status Location Summary Round Reviewer Posted
CRF-1 P3 Author fixed (at a62c34a) generation.go:575 No test drives production stage wiring (runGenerationStep) into the turn partition R1 Netero Yes
CRF-2 P3 Author fixed (at a62c34a) turnaccounting_internal_test.go:434 NilTracer subtest passes by construction, not by accounting logic R1 Netero Yes
CRF-3 P3 Author fixed (at a62c34a) turnaccounting_internal_test.go:692 Hand-rolled metric walkers duplicate the sums helper and promhelp R1 Netero P3, Bisky Nit, Robin Note Yes
CRF-4 P3 Author contested; panel closed R4 (1/1 accept) metrics.go:304 Unreachable elapsed<0 guard in RecordTurnCategory would silently drop a series R1 Netero Yes
CRF-5 P3 Author fixed (at a62c34a) turnaccounting.go:33 TurnTimeCategories (mutable slice) and TurnAccumulator.Model exported without external callers R1 Netero Yes
CRF-6 P3 Author fixed (at a62c34a) turn_trace.go:28 runnerTurnSpan doc dropped the fact that late stages nest inside the turn span R1 Netero Yes
CRF-7 Nit Author fixed (at a62c34a; capacity test left the diff) capacity_internal_test.go:23 No-op newStageTestTracer to newStageMetricsTracer swaps that discard the registry R1 Netero Yes
CRF-8 Note Dropped by orchestrator (verified: Storybook snapshot upload got HTTP 500 from the snapshot service; infra, unrelated to PR; noted in body) CI Storybook job failed on a PR that touches no site/ files R1 Netero No
CRF-9 P2 Author fixed (8a6db7b, in the base branch) turnaccounting.go:372 requires_action client wait stays inside the turn and is emitted as unattributed R1 Pariston P2, Mafuuu P2, Kite P2, Melody P2, Pen Botter P2, Ryosuke P2, Hisoka P3 Yes
CRF-10 P2 Author acknowledged R5; panel re-raised R5 (Komugi re-raised, Takumi accepted) turn_trace.go:221 Another task's Ensure can close a finished turn before its final step ends; partition depends on scheduling R1 Komugi P2, Takumi P3 Yes
CRF-11 P2 Author fixed (8a6db7b, in the base branch) turnaccounting.go:73 Acquisition re-covers prior attempts (reopen after invalidation, failover) and books them as scheduling R1 Razor P2, Ryosuke P3 Yes
CRF-12 P2 Author fixed (at a62c34a) turnaccounting.go:74 queue_wait in scheduling mixes pickup latency with the previous turn's remaining runtime R1 Knov Yes
CRF-13 P2 Author fixed (at a62c34a) turnaccounting.go:111 Accumulator completed/invalid duplicates runnerTurnSpan outcome; turn_outcomes_total counted from two packages R1 Zoro P2, Meruem P3, Bisky P3, Mafu-san P3, Chopper P3, Gon P3, Kite P3, Knov P3, Luffy P3, Ryosuke P3 Yes
CRF-14 P2 Author fixed (at ab6ebb9) turnaccounting.go:105 Concurrency rationale names tool calls that never reach the accumulator; concurrency test pins an overattributed turn R1 Mafu-san P2, Razor P3, Bisky P3, Ryosuke Nit, Takumi Note Yes
CRF-15 P2 Author fixed (at a62c34a) stage_internal_test.go:457 No chatd test reads turn_time_seconds_total; removing the accumulator from contextLocked stays green R1 Bisky Yes
CRF-16 P2 Author fixed (at a62c34a) stage.go:324 Comment restates the scope == ScopeTurn check R1 Gon Yes
CRF-17 P2 Author contested; panel closed R4 (1/1 accept) stage.go:428 adoptTurnModel doc narrates the body R1 Gon Yes
CRF-18 P2 Author fixed (at a62c34a) turnaccounting.go:9 Category doc states sum-to-duration invariant that overattributed turns break, in three clauses R1 Gon Yes
CRF-19 P2 Author fixed (at a62c34a) turnaccounting.go:30 TurnTimeCategories doc repeats the emit-every-category rule with a different reason R1 Gon Yes
CRF-20 P2 Author fixed (at a62c34a) turnaccounting.go:47 attributingStages doc restates itself and gets two facts wrong R1 Gon Yes
CRF-21 P2 Author fixed (at a62c34a) turnaccounting.go:65 recordedStageCategories doc restates the map lookup R1 Gon Yes
CRF-22 P2 Author fixed (at a62c34a) turnaccounting.go:77 turnAccumulatorKey doc explains the private-key idiom unlike sibling keys R1 Gon Yes
CRF-23 P2 Author fixed (at a62c34a) turnaccounting.go:101 TurnAccumulator doc restates its first clause in reverse R1 Gon Yes
CRF-24 P2 Author fixed (at a62c34a) turnaccounting.go:115 NewTurnAccumulator doc restates "not accounted until MarkCompleted" R1 Gon Yes
CRF-25 P2 Author fixed (at a62c34a) turnaccounting.go:133 setModel doc restates first-wins R1 Gon Yes
CRF-26 P2 Author fixed (at a62c34a) turnaccounting.go:181 turnAccounting doc adds nothing to its field names R1 Gon Yes
CRF-27 P2 Author fixed (at a62c34a) turnaccounting.go:213 Fourth statement of the tool-execution rule on stageNode.action R1 Gon Yes
CRF-28 P2 Author fixed (at a62c34a) turnaccounting.go:265 category doc repeats the switch R1 Gon Yes
CRF-29 P2 Author fixed (at a62c34a) turnaccounting.go:348 emitTurnAccounting doc narrates arithmetic and omits completed counting and nonpositive drop R1 Gon Yes
CRF-30 P2 Author fixed (at a62c34a) generation.go:128 Comment restates the shared constant and the classification rule R1 Gon Yes
CRF-31 P2 Author fixed (at a62c34a) turn_trace.go:146 "with a fresh accumulator" narrates the assignment R1 Gon Yes
CRF-32 P2 Author fixed (at a62c34a) turn_trace.go:162 Comment lists the ContextWith calls under it R1 Gon Yes
CRF-33 P2 Author fixed (at a62c34a) turn_trace.go:190 Comment restates the four ContextWith calls by name R1 Gon Yes
CRF-34 P2 Author fixed (at a62c34a) turn_trace.go:227 Invalidate doc copies the drop rationale owned elsewhere R1 Gon Yes
CRF-35 P2 Author fixed (at a62c34a) turnaccounting_internal_test.go:70 categorySums doc restates a one-line body R1 Gon Yes
CRF-36 P2 Author fixed (at a62c34a) turnaccounting_internal_test.go:76 stageSums doc restates a one-line body R1 Gon Yes
CRF-37 P2 Author fixed (at a62c34a) turnaccounting_internal_test.go:104 syntheticTurn doc restates return types R1 Gon Yes
CRF-38 P2 Author fixed (at a62c34a) turnaccounting_internal_test.go:119 Comment narrates the Start/End sequence R1 Gon Yes
CRF-39 P2 Author fixed (at a62c34a) turnaccounting_internal_test.go:171 "A compaction step." restates the code R1 Gon Yes
CRF-40 P2 Author fixed (at a62c34a) turnaccounting_internal_test.go:217 Comment restates the require.Equal under it R1 Gon Yes
CRF-41 P2 Author fixed (at a62c34a) turnaccounting_internal_test.go:297 Comment restates two require.Zero lines R1 Gon Yes
CRF-42 P2 Author fixed (at a62c34a) turnaccounting_internal_test.go:352 Comment restates require.InDelta R1 Gon Yes
CRF-43 P2 Author fixed (at a62c34a) turnaccounting_internal_test.go:691 Comment restates recorder calls and assertions R1 Gon Yes
CRF-44 P3 Author fixed (at a62c34a) turnaccounting_internal_test.go:116 Flagship fixture records queue_wait before turn start; sum check passes by cancellation R1 Bisky P3, Hisoka P3, Meruem P3 Yes
CRF-45 P3 Author contested; panel closed R4 (3/3 accept) turnaccounting.go:277 Hook dispatch to external endpoints lands in chatd_overhead (or tool_execution) R1 Mafuuu P3, Chopper P3, Knov P3 Yes
CRF-46 P3 Author contested; panel closed R4 (2/2 accept) turnaccounting.go:274 Tool steps classify decide, hook, finish, and retry bookkeeping as tool_execution R1 Kite P3, Leorio Note Yes
CRF-47 P3 Author fixed (at a62c34a) turnaccounting.go:284 retry_backoff covers only generation retries; phase and task-level backoff land elsewhere R1 Mafuuu P3, Melody P3, Chopper P3, Razor Note Yes
CRF-48 P3 Author fixed (at a62c34a) turnaccounting.go:292 Failed attempts' TTFT and discarded responses land in time_to_first_token/streaming, contrary to docs R1 Chopper P3, Razor P3 Yes
CRF-49 P3 Author fixed (at a62c34a) turnaccounting.go:287 stageNode.failed/markFailed has no distinguishing test and moves near-zero time R1 Bisky Yes
CRF-50 P3 Author fixed (at a62c34a) turnaccounting.go:317 own<0 clamp duplicates addCategory guard; ClampsOwnTime cannot detect its removal R1 Bisky P3, Ging-Go Note Yes
CRF-51 P3 Author fixed (at a62c34a) metrics.go:191 turn_time_seconds_total Help promises categories sum to the turn duration; overattributed turns do not R1 Mafu-san P3, Gon P3, Leorio P3, Pen Botter P3, Mafuuu Note, Kite Note Yes
CRF-52 P3 Author fixed (at a62c34a) metrics.go:191 turn_time_seconds_total Help lists no category values or definitions R1 Pen Botter P3, Luffy Nit Yes
CRF-53 P3 Author fixed (at a62c34a) metrics.go:203 turn_outcomes_total Help says completed equals partition-recorded; false for nonpositive_turn; untested R1 Leorio P3, Chopper P3 Yes
CRF-54 P3 Author fixed (at a62c34a) metrics.go:193 turns_total duplicates turn_outcomes_total{completed} except on clock step R1 Ryosuke P3, Zoro P3, Luffy P3 Yes
CRF-55 P3 Author fixed (at a62c34a) turn_trace.go:245 Completed-only partition drops the turns operators most need (provider errors, interrupts) R1 Pariston P3, Luffy P3, Pen Botter Note Yes
CRF-56 P3 Author contested; panel closed R4 (1/1 accept) stage.go:430 First-model stamp on chat_turn contradicts "a turn is not a model-scoped unit" R1 Luffy Yes
CRF-57 P3 Author fixed (at a62c34a) turnaccounting.go:54 attributingStages and the category switch are two tables that must agree; drift is silent R1 Meruem P3, Razor P3, Robin P3, Zoro P3 Yes
CRF-58 P4 Author fixed (at ab6ebb9) turnaccounting.go:375 overattributed anomaly carries no chat_kind or span marker to follow R1 Chopper Yes
CRF-59 P3 Author fixed (a62c34a; documented in the unattributed row, no stage added) generation.go:443 Per-step loadGenerationState (full history read) lands in unattributed from step 2 on R1 Knuckle Yes
CRF-60 P4 Author fixed (at ab6ebb9) turnaccounting.go:281 FinishTurn transaction and afterGenerationOutcome history read land in chatd_overhead, not persistence R1 Knuckle Yes
CRF-61 P3 Author contested; panel closed R4 (1/1 accept) stage.go:41 chatloop now owns one value of chatd's generation-action enum; SetGenerationAction works on one stage R1 Ryosuke P3 (Zoro Note disagrees) Yes
CRF-62 P3 Author fixed (at a62c34a) stage_internal_test.go:668 afterInterrupt turn anchored at time.Now() can have zero duration on coarse clocks R1 Komugi Yes
CRF-63 P3 Author fixed (at a62c34a) turnaccounting_internal_test.go:33 newTurnFixture duplicates newStageMetricsFixture R1 Robin Yes
CRF-64 P3 Author fixed (at a62c34a) turnaccounting.go:13 Category lacks the Turn prefix its siblings use and collides with refusal.Category R1 Gon Yes
CRF-65 P3 Author fixed (at a62c34a) turnaccounting.go:340 recordAttribution handles the stages that are not attributing R1 Gon Yes
CRF-66 P3 Author fixed (at a62c34a) turnaccounting.go:304 report hides the turn emission point behind a per-stage name and doc R1 Gon Yes
CRF-67 P3 Author fixed (at a62c34a) stage.go:406 End, SetModel, StageSpan, and RecordAs docs omit the new turn-accounting side effects R1 Gon P3, Leorio Nit Yes
CRF-68 P3 Author fixed (at a62c34a) stage.go:417 EndWithoutObservation doc says "exactly as End" and "no duration observation"; both now false R1 Gon P3, Leorio P3 Yes
CRF-69 P3 Author fixed (at a62c34a) metrics.go:291 RecordStageAnomaly doc dropped "adjusted"; RecordAnomaly doc (stage.go:506) not updated R1 Gon Yes
CRF-70 P3 Author fixed (at a62c34a) stage.go:197 acc field doc is wrong for background-scoped stages inside a turn R1 Gon Yes
CRF-71 Nit Author fixed (at a62c34a) turnaccounting_internal_test.go:106 syntheticTurn returns an accumulator no caller uses R1 Gon Yes
CRF-72 Nit Author fixed (at a62c34a) ARCHITECTURE.md:844 "emit nothing" contradicted two lines later by outcome counting R1 Leorio Yes
CRF-73 Nit Author fixed (at a62c34a) turnaccounting_internal_test.go:532 Use wg.Go instead of Add/Done R1 Ging-Go Yes
CRF-74 Nit Author fixed (at a62c34a) turnaccounting.go:197 Use maps.Clone in snapshot R1 Ging-Go Yes
CRF-75 Note Author fixed (a62c34a; chat-kind overlap documented) turn_trace.go:280 Summing categories across turns or chat kinds double counts overlapping wall time R1 Hisoka Yes
CRF-76 Note Author fixed (at a62c34a) ARCHITECTURE.md:852 PromoteQueued queue_wait is observed but never reaches scheduling R1 Melody Yes
CRF-77 Note Author contested; panel closed R4 (1/1 accept) stage.go:328 Scope and accumulator are two turn-membership signals that can disagree R1 Meruem Yes
CRF-78 Note Author fixed (at a62c34a) turnaccounting.go:342 recordAttribution acc==nil branch has no chatloop coverage R1 Bisky Yes
CRF-79 Note Dropped by orchestrator (Chopper confirmed 16 = 11 categories + 4 outcomes + 1 turns_total; PR-description wording only; noted in body) PR description "16 per chat kind" reads as a category count R1 Mafu-san No
CRF-80 P4 Author fixed (at ab6ebb9) turnaccounting.go:232 EndTurn's if !ok after closeSpan cannot run; the nil/ended check is duplicated R4 Netero Yes
CRF-81 OOS Out of scope (body) Makefile:1344 markdown-table-formatter re-pads all 245 rows of prometheus.md when one Help widens the column R4 Netero No
CRF-82 P2 Author fixed (at ab6ebb9) stage.go:339 Compaction summary's first-token window counted as time_to_first_token/provider_error, not compaction R4 Melody, Hisoka, Razor Yes
CRF-83 P3 Author fixed (at ab6ebb9) turnaccounting.go:202 Stream canceled by user stop, edit, or shutdown counted as provider_error R4 Pariston, Mafuuu Yes
CRF-84 P3 Author fixed (at ab6ebb9; stage definition updated, observation kept) generation.go:769 Phase-backoff retry_backoff span changes the retry_backoff histogram; stage definition not updated R4 Knov, Kite, Hisoka Yes
CRF-85 P3 Author fixed (at ab6ebb9) chatd_test.go:15002 ToolCall subtest waits on the outcome count, which is written before the categories; can fail R4 Komugi, Bisky Yes
CRF-86 P3 Author acknowledged R5; panel closed R5 (1/1 accept); root cause out of scope (CRF-127), needs a human decision turn_trace.go:292 Committed turn counted as abandoned when the promoted task's Ensure runs before Complete R4 Komugi Yes
CRF-87 P3 Author fixed (at ab6ebb9) metrics.go:198 Help does not scope the division to chat_kind or warn that sums across chat_kind count subagent time twice R4 Leorio P3, Pen Botter P3, Luffy P4 Yes
CRF-88 P3 Author fixed (at ab6ebb9) metrics.go:204 turn_outcomes_total Help defines none of the four outcomes R4 Pen Botter Yes
CRF-89 P3 Author fixed (at ab6ebb9) metrics.go:210 stage_anomalies_total Help lost the negative_elapsed and future_start definitions R4 Leorio Yes
CRF-90 P4 Author contested; panel closed R5 (3/3 accept) turnaccounting.go:225 EndTurn works on any StageSpan and End works on chat_turn; the root-only rule is doc-only R4 Knov, Ryosuke, Meruem Yes
CRF-91 P4 Author fixed (at ab6ebb9) stage.go:202 StageSpan doc names End and EndWithoutObservation but not EndTurn R4 Gon Yes
CRF-92 P4 Author contested; panel closed R5 (3/3 accept) turnaccounting.go:251 Recorded-stage subtraction from an enclosing stage has no production path; test builds an impossible tree R4 Meruem, Bisky, Ryosuke Yes
CRF-93 P4 Author fixed (at ab6ebb9) stage.go:444 adoptTurnModel guard is never true in production R4 Bisky Yes
CRF-94 P4 Author fixed (at ab6ebb9) metrics.go:198 Help calls retry_backoff "retry delays" but the task-level retry sleep is unattributed R4 Razor Yes
CRF-95 P2 Author fixed (at ab6ebb9) turnaccounting.go:209 report doc restates the three-line body R4 Gon Yes
CRF-96 P2 Author fixed (at ab6ebb9) turnaccounting.go:138 snapshot doc restates maps.Clone and the nil branch R4 Gon Yes
CRF-97 P2 Author fixed (at ab6ebb9) turnaccounting_internal_test.go:88 startTurn doc lists the calls in its body R4 Gon Yes
CRF-98 P2 Author fixed (at ab6ebb9) turnaccounting_internal_test.go:148 Comment calls a step failed whose stream ends without error R4 Gon Yes
CRF-99 P2 Author fixed (at ab6ebb9) chatd_test.go:14998 Second comment sentence restates the assertion and the tool-execution rule R4 Gon Yes
CRF-100 P2 Author fixed (at ab6ebb9) stage_internal_test.go:299 Comment restates the require.Equal under it R4 Gon Yes
CRF-101 Nit Author fixed (at ab6ebb9) turnaccounting_internal_test.go:154 failed* variable names for stages that end without error R4 Gon Yes
CRF-102 OOS Out of scope (body) coderd.go:918 Hook dispatcher has a nil HTTP client and no span; hook time is an unexplained gap in traces R4 Chopper No
CRF-103 Note Dropped by orchestrator (verification only, asks for nothing: 88 series, bounded) metrics.go:194 outcome label cardinality R4 Chopper No
CRF-104 Note Dropped by orchestrator (praise, asks for nothing) metrics.go:49 Anomaly and EndTurn docs as a model R4 Leorio No
CRF-105 Note Dropped by orchestrator (verification only, asks for nothing) turn_trace_internal_test.go:775 New runner tests are deterministic R4 Komugi No
CRF-106 Note Dropped by orchestrator (verification only, asks for nothing) turnaccounting.go:238 Round 1 findings fixed R4 Pen Botter No
CRF-107 Note Dropped by orchestrator (process praise, asks for nothing) PR Author replied on every thread R4 Mafu-san No
CRF-108 Note Dropped by orchestrator (verification only, asks for nothing) turnaccounting.go:52 Stage, outcome, and Help pairings agree R4 Melody No
CRF-109 P3 Open turnaccounting.go:204 CRF-83 fix counts provider-side cancellation (silence timeout, transport reset) as streaming R5 Netero (Pariston, Mafuuu, Chopper, Melody concur) Yes
CRF-110 P4 Open chatloop.go:827 Any StageTracer reports into the turn accumulator on ctx; the nil-tracer stand-in is blocked by hand R5 Meruem Yes
CRF-111 P4 Open turnaccounting_internal_test.go:575 Race test cannot detect a missing addCategory lock because maps.Clone is invisible to -race R5 Bisky Yes
CRF-112 P3 Open metrics.go:204 turn_outcomes_total Help omits the documented cases where finished or interrupted turns count as abandoned R5 Pen Botter Yes
CRF-113 P3 Open metrics.go:198 unattributed Help omits the open-stage time that lands there on interrupted turns R5 Pen Botter Yes
CRF-114 P3 Open turnaccounting.go:281 "a counted outcome implies its partition is recorded" is false (nonpositive turns, concurrent Gather); same clause in ARCHITECTURE.md:844 R5 Gon P3, Leorio P3 Yes
CRF-115 P3 Open turn_trace.go:286 Deleted base comment took the "reply latency" reason for observing only completed turns R5 Gon Yes
CRF-116 P3 Dropped by orchestrator (merged into the CRF-10 R5 re-raise reply, which quotes it) PR description Review decisions call the stale-step loss "small"; ARCHITECTURE.md says the whole step R5 Leorio No
CRF-117 P4 Open ARCHITECTURE.md:848 "categories that sum to the turn duration" is unqualified; overattributed turns sum to more R5 Leorio Yes
CRF-118 P4 Open stage.go:205 StageSpan, End, EndWithoutObservation docs say every turn-scoped stage reports; only attributing stages do R5 Mafuuu Yes
CRF-119 P2 Open chatloop.go:821 Comment restates ContextWithTurnAccumulator(parent, nil) and omits why R5 Gon Yes
CRF-120 P2 Open turnaccounting.go:202 Cancellation comment repeats ARCHITECTURE.md examples and the default branch R5 Gon Yes
CRF-121 P2 Open turnaccounting_internal_test.go:349 TestTurnAccountingCanceledStream doc repeats the name and the stop example R5 Gon Yes
CRF-122 P2 Open turnaccounting_internal_test.go:415 TestTurnAccountingNilTracerStream doc restates setup and assertions R5 Gon Yes
CRF-123 P2 Open turnaccounting_internal_test.go:549 TestTurnAccountingConcurrentStageEnds doc copies the TurnAccumulator rationale R5 Gon Yes
CRF-124 P2 Open turnaccounting_internal_test.go:579 "the partition sums to the turn either way" restates the assertion R5 Gon Yes
CRF-125 Nit Open turn_trace.go:204 Invalidate doc rewrapped with no word changed R5 Gon Yes
CRF-126 Nit Open turnaccounting.go:284 Parameter named categories, doc calls it partition R5 Gon Yes
CRF-127 OOS Out of scope (body) chatstate/machine.go:205 Update publishes before the caller acts on the commit; cause of the CRF-86 outcome flip R5 Komugi No
CRF-128 Note Dropped by orchestrator (verification only, asks for nothing) turnaccounting.go:284 CRF-85 fix verified R5 Komugi No
CRF-129 Note Dropped by orchestrator (verification only, asks for nothing) chatloop.go:824 CRF-82 fix and pairings verified R5 Melody No
CRF-130 Note Dropped by orchestrator (verification only, asks for nothing) metrics.go:204 CRF-87 and CRF-88 fixes verified R5 Pen Botter No
CRF-131 Note Dropped by orchestrator (verification only, asks for nothing) metrics.go:198 CRF-87 and CRF-89 fixes verified R5 Leorio No

Law analysis

Round 1. Effective LOC +1512 -29 (578 production, 934 test). Head SHA 2c98b6d. Verdict: Don't split. Enforcement: Advisory.

Contested and acknowledged

CRF-4 (P3, metrics.go:304) - Unreachable elapsed<0 guard in RecordTurnCategory

  • Finding: The elapsed < 0 clause in RecordTurnCategory cannot fire; if it did, it would drop one category's sample for the turn without a signal, contradicting "every category is emitted for every accounted turn". Proposed removing the clause and keeping the m == nil guard.
  • Author defense (R3 thread reply): Keeps the guard because Counter.Add panics on a negative value, so the guard stops a caller bug from crashing the server; emitTurnAccounting clamps the remainder and counts overattributed, so the guard is not the only signal.
  • Status: contested, panel pending (R4). Raised by Netero.
  • Panel closure (R4, 1/1): Netero, the agent that raised it (resumed), accepted: without the guard Counter.Add panics inside coderd, dropping one sample is the smaller failure, and the only caller never passes a negative value (addCategory rejects values <= 0; unattributed is clamped at turnaccounting.go:272).

CRF-10 (P2, turn_trace.go:221) - Another task's Ensure can close a turn before its final step ends

  • Finding: Two call sites close a turn: the finishing task's Settle, and Ensure in another task. Nothing orders the finishing task's deferred step End before the other task's Ensure, so the finishing step's time moves to unattributed (Komugi measured chatd_overhead: 0, unattributed: 0.1007 with a 50ms hold), and a mid-turn variant counts an overlap window in two steps (Takumi). The loss is silent because the partition still sums to the turn.
  • Author defense (R3 thread reply, PR description): Not changing; the loss is accepted as a small unattributed share rather than delaying emission. ARCHITECTURE.md now says a stage that ends after its turn has closed is not in the partition.
  • Status: acknowledged, not fixed, no ticket; panel pending (R4). Raised by Komugi P2, Takumi P3.
  • Panel re-raise (R4): Komugi (resumed) accepted the defense for the category shift of a finishing step's milliseconds, and raised the outcome flip as CRF-86. Takumi (resumed) re-raised at P2: when EndTurn closes the turn before an attributing stage ends, that stage's whole own time goes to unattributed; on interrupts the open step is the long one (scratch test: a 300 ms execute_local_tools step moves entirely from tool_execution to unattributed with a 1 ms ordering difference). Proposed fix: the accumulator tracks open nodes and EndTurn categorizes their own time up to the turn end.

CRF-17 (P2, stage.go:428) - adoptTurnModel doc narrates the body

  • Finding: The adoptTurnModel doc narrates the lines under it; proposed deleting it (and removing it with the model stamp if CRF-56 is fixed).
  • Author defense (R3 thread reply): Keeps the one-sentence doc because it states the non-obvious condition that the root adopts a model only when it never received one.
  • Status: contested, panel pending (R4). Raised by Gon.
  • Panel closure (R4, 1/1): Gon (resumed) accepted: the condition is visible in the body, so the doc duplicates SetModel and model() docs, but it has no runtime effect.

CRF-45 (P3, turnaccounting.go:277) - Hook dispatch to external endpoints lands in chatd_overhead

  • Finding: Time waiting on external hook endpoints (session_start, stop, pre_compact/post_compact, pre_tool_use) goes to chatd_overhead, while post_tool_use goes to tool_execution, so the category meant to isolate chatd's own cost points at chatd when a hook is slow. Proposed an attributing hook_dispatch stage with its own category.
  • Author defense (R3 thread reply): Hook dispatch runs inside the step and is documented at chatd_overhead granularity in the Turn accounting table; the trace carries per-hook detail.
  • Status: contested, panel pending (R4). Raised by Mafuuu P3, Chopper P3, Knov P3.
  • Panel closure (R4, 3/3): Mafuuu, Chopper, and Knov (all resumed) accepted: the chatd_overhead row names hook dispatch, and coderd_chatd_hook_dispatch_seconds{event} separates hook latency. Mafuuu and Chopper found the defense's "trace carries per-hook detail" false: the dispatcher starts no span and uses an uninstrumented client (coderd.go:918), recorded as out of scope CRF-102.

CRF-46 (P3, turnaccounting.go:274) - Tool steps classify decide, hook, finish, and retry bookkeeping as tool_execution

  • Finding: On steps that run local tools, the step's whole own time (hook dispatch, decide phase, FinishTurn, recordGenerationRetry) goes to tool_execution; on other steps the same work goes to chatd_overhead, which contradicts the ARCHITECTURE.md chatd_overhead row. Proposed deriving tool_execution from the union of tool_call windows, or stating the coarse split in the table.
  • Author defense (R3 thread reply): The category follows the step's action, as the Turn accounting table documents; per-phase detail inside a tool step is a trace question.
  • Status: contested, panel pending (R4). Raised by Kite P3, Leorio Note.
  • Panel closure (R4, 2/2): Kite and Leorio (both resumed) accepted: the table, the chatd_overhead row ("every other action"), and the Help now all define tool_execution at step granularity.

CRF-56 (P3, stage.go:430) - First-model stamp on chat_turn contradicts "a turn is not a model-scoped unit"

  • Finding: The PR argues a turn has no single model, then stamps the first model on the chat_turn span, which is wrong for the turns the PR calls out; proposed removing it or naming it honestly (for example initial_model) with the reason written down.
  • Author defense (R3 thread reply): The model is only a trace attribute on chat_turn (the first model resolved) to aid navigation; the turn metric families carry no model label.
  • Status: contested, panel pending (R4). Raised by Luffy.
  • Panel closure (R4, 1/1): Luffy (resumed) accepted: the value is a trace attribute only, child spans keep their own model, and the PR description and ARCHITECTURE.md say "first model resolved".

CRF-58 (P3, turnaccounting.go:375) - overattributed anomaly carries no marker to follow

  • Finding: When overattributed fires, neither the metric (reason label only) nor the trace identifies the turn or its chat kind; proposed computing the partition before closeSpan and stamping an attribute on chat_turn.
  • Author defense (R3 thread reply): stage_anomalies_total keeps a single reason label by design to bound cardinality; the overattributed turn can be found in traces by comparing the chat_turn span with its children.
  • Status: contested, panel pending (R4). Raised by Chopper.
  • Panel re-raise (R4, P4): Chopper (resumed) re-raised: the cardinality reason covers the metric label, not the proposed span attribute; EndTurn already writes turn_outcome on the span before closeSpan, so an overattributed marker can be set at the same point. Lowered to P4 because no production path over-attributes today.

CRF-60 (P3, turnaccounting.go:281) - FinishTurn transaction and history read land in chatd_overhead

  • Finding: persistence counts only commit; each completed turn's FinishTurn transaction and the synchronous afterGenerationOutcome prompt-history read go to chatd_overhead (or tool_execution on a tool step). Proposed a turn-scoped stage mapped to persistence, or naming these reads in the chatd_overhead row.
  • Author defense (R3 thread reply): persistence is the commit stage by definition, and the finishing transition is documented as chatd_overhead in the Turn accounting table.
  • Status: contested, panel pending (R4). Raised by Knuckle.
  • Panel re-raise (R4, P4): Knuckle (resumed) accepted the FinishTurn half (the row lists "transitions other than CommitStep") and re-raised the read: afterGenerationOutcome runs GetChatMessagesForPromptByChatID in the final step, after Complete and before the step ends, and the chatd_overhead row does not name it. Fix: name the read in the row.

CRF-61 (P3, stage.go:41) - chatloop owns one value of chatd's generation-action enum

  • Finding: chatloop owns one value of chatd's generationActionKind enum, and SetGenerationAction exists on every StageSpan but only does something on generation_step; proposed keeping the vocabulary in chatd and passing chatloop a category. Zoro argued the opposite in R1.
  • Author defense (R3 thread reply): chatloop needs only the one action value that changes categorization, and chatd's constant is defined from chatloop.GenerationActionExecuteLocalTools, so the two cannot drift.
  • Status: contested, panel pending (R4). Raised by Ryosuke.
  • Panel closure (R4, 1/1): Ryosuke (resumed) accepted: chatd's constant is defined from the chatloop constant, so the values cannot drift; a new tool-time action costs one chatloop line and none exists.

CRF-77 (Note, stage.go:328) - Scope and accumulator are two turn-membership signals

  • Finding: Turn membership is carried by the scope and the accumulator, which can disagree; proposed clearing the accumulator in inflightContext so a nil accumulator is the single signal.
  • Author defense (R3 thread reply): Both entry points (startSpan, RecordAs) read the accumulator only when the scope is turn, so a background stage never reports to a turn; TestTurnAccountingIgnoresWorkOutsideTheTurn covers it.
  • Status: contested, panel pending (R4). Raised by Meruem.
  • Panel closure (R4, 1/1): Meruem (resumed) accepted: RecordAs with a caller-supplied scope was removed in the base, Record and startSpan both read scope from ctx, inflightContext sets ScopeBackground, and the only disagreeing path Meruem had cited is gone.

CRF-10 (P2, turn_trace.go:221) - R5 response to the R4 re-raise

  • Finding (R4 re-raise): When a turn closes before an attributing stage ends, that stage's whole own time goes to unattributed; on interrupts the open step is the long one. Proposed tracking open nodes so EndTurn categorizes their own time up to the turn end without delaying emission.
  • Author defense (R5 thread reply): Not fixed in code; documented in ARCHITECTURE.md "Known limitations of turn accounting": a stage still open when its turn closes contributes nothing, so the step's whole own time lands in unattributed. No reason given for rejecting the proposed fix; no ticket.
  • Status: acknowledged, not fixed, no ticket; panel pending (R5). Raised by Komugi P2, Takumi P3 (Takumi re-raised R4).
  • Panel re-raise (R5): Komugi (resumed) re-raised at P2 for interrupts: StartInterrupt calls Settle without waiting for the canceled task, and a scratch runnerTurnSpan test moved a 300 s execute_local_tools step between tool_execution=300 and unattributed=300.001 on a 1 ms ordering difference; nothing marks the affected turn, and the author gave no reason for rejecting the open-node fix. Takumi (resumed) accepted: the known-limitations text now states the real size of the loss. Leorio (new agent) found the PR description's Review decisions still call the loss "small" (CRF-116, merged into the re-raise reply).

CRF-86 (P3, turn_trace.go:292) - Committed turn counted as abandoned

  • Finding: ChatMachine.Update publishes during commit, before Complete, and the promoted task's Ensure can close the open, unfinished turn as abandoned, moving a completed turn between outcome series. Proposed calling Complete inside the Update callback after FinishTurn succeeds.
  • Author defense (R5 thread reply): Not fixed; documented as a known limitation. "A fix would need new ordering between the commit and the promoted task, which is out of scope for this stack." No ticket.
  • Status: acknowledged, not fixed, no ticket; panel pending (R5). Raised by Komugi.
  • Panel closure (R5, 1/1): Komugi (resumed) accepted: the race needs pubsub delivery, a spawn, and loadGenerationState to beat a function return, it is documented at ARCHITECTURE.md:871, and the ordering fix belongs in ChatMachine.Update, which the PR does not change (out of scope CRF-127). The accepted gap has no ticket, so it needs a human decision.

CRF-90 (P4, turnaccounting.go:225) - Root-only use of EndTurn is doc-only

  • Finding: EndTurn works on any StageSpan (double-counting the turn on a non-root span) and End works on chat_turn (emitting nothing). Proposed an early return when s.stage != StageChatTurn, or a turn span type exposing only EndTurn.
  • Author defense (R5 thread reply): EndTurn is used only on turn roots by convention and has one caller, runnerTurnSpan.closeLocked; a type-level split would add a new mechanism for a single call site. The reply does not address the early-return option.
  • Status: contested, panel pending (R5). Raised by Knov, Ryosuke, Meruem.
  • Panel closure (R5, 3/3): Knov, Ryosuke, and Meruem (all resumed) accepted: EndTurn still has one production caller, runnerTurnSpan.closeLocked, which always passes the chat_turn span from startLocked; a type split protects one call site, and Meruem noted the early return would turn misuse into silently discarded data rather than prevent it.

CRF-92 (P4, turnaccounting.go:251) - Recorded-stage subtraction has no production path

  • Finding: categorizeRecordedStage subtracts a recorded stage from the attributing stage on ctx, but acquisition, the only categorized recorded stage, is always recorded on the turn context, so the line never runs in production and its test builds a tree production cannot produce. Proposed deleting the line, the test, and the matching doc sentences.
  • Author defense (R5 thread reply): Kept as a guard that keeps the categories disjoint if a categorized recorded stage is ever recorded under an attributing stage; three lines; TestTurnAccountingRecordedStageUnderStep covers it.
  • Status: contested, panel pending (R5). Raised by Meruem, Bisky, Ryosuke.
  • Panel closure (R5, 3/3): Meruem, Ryosuke (resumed), and Bisky (resumed R4 agent) accepted: the line enforces one rule for all categorized stages, costs three lines and one test, and a future misuse without it would surface as overattributed (metric and span attribute).

Round log

Round 1

Netero first pass: 6 P3, 1 Nit, 1 Note. Law: don't split (advisory). Panel of 24 (22 trigger-matched, wildcards Kurapika and Knuckle). New: 35 P2 (7 behavioral or structural, 28 comment-quality from Gon), 27 P3, 4 Nit, 4 Note posted. CRF-8 and CRF-79 dropped with reasons. Killua and Kurapika reported no findings. Reviewed against c2202f0..2c98b6d.

Round 2 update

BLOCKED. All findings silent. No review.
Churn guard: 0 addressed, 0 acknowledged, 0 contested, 0 deferred, 77 silent. Head 11a1826 rebases the round 1 patch onto base 5ffb40e; range-diff shows only rebase hunks, and turnaccounting.go, turnaccounting_internal_test.go, and generation.go are unchanged since 2c98b6d. No thread replies. CI failures are infrastructure: doc-check Coder user suspended (Unauthorized), Storybook snapshot service HTTP 500 "api key lookup failed". Reviewed against: 5ffb40e..11a1826.

Round 3 update

BLOCKED. CRF-59 and CRF-75 silent. No review.
Churn guard: 65 addressed, 1 acknowledged, 9 contested, 0 deferred, 2 silent out of 77 open findings. Both silent findings were folded into the round 1 review body and have no thread; the author replied on every thread. CRF-59 still applies at head: generation.go:439 runs loadGenerationState on the task ctx while the turn is open, before Ensure. CRF-75 asked for one ARCHITECTURE.md sentence that is not present. Statuses are not updated on a blocked round. CI passed. Reviewed against: 61546de..35203c4.

Round 4

Churn guard PROCEED: 67 addressed, 1 acknowledged (CRF-10), 9 contested, 0 silent. Round type: restructure (files added to the diff: chatd_test.go, metrics_internal_test.go, export_test.go, generation_retry_internal_test.go; exported API changed: Category renamed TurnCategory, MarkCompleted removed, EndTurn added). Law not run (effective additions 1512 to 1582). Netero resumed (CRF-4 contested). Full panel of 21: dispute check-in agents resumed for Gon, Mafuuu, Chopper, Knov, Kite, Leorio, Luffy, Knuckle, Ryosuke, Meruem; Komugi and Takumi resumed for acknowledged CRF-10 (orchestrator choice: an acknowledged-not-fixed finding by an agent author gets the same check-in); new agents for Ging-Go, Melody, Pen Botter, Killua (specialists), Pariston (core), Bisky (tests), Hisoka, Mafu-san (pool), Razor (wildcard, drawn from Zoro and Razor). Closed: CRF-4, 17, 45, 46, 56, 61, 77. Re-raised: CRF-10 (P2), CRF-14 (P2), CRF-58 (P4), CRF-60 (P4). New: 7 P2, 7 P3, 6 P4, 1 Nit posted; 2 out of scope; 6 verification or praise Notes dropped. Ging-Go and Killua reported no findings. Reviewed against 282311c..a62c34a.

Round 5

Churn guard PROCEED: 21 addressed, 2 acknowledged (CRF-10, CRF-86), 2 contested (CRF-90, CRF-92), 0 silent. Round type: fix round (every change maps to a round 4 finding; chatloop.go entered the diff for the CRF-82 fix; no file created and no exported signature changed, only the new constant AttrOverattributed). Law not run (effective additions 1582 to 1702). Netero new agent (no contested Netero finding). Panel of 14: resumed Knov, Ryosuke, Meruem, Bisky (contested), Komugi, Takumi (acknowledged); new Ging-Go, Melody, Pen Botter (specialists for changed Go and docs), Gon, Leorio (comments, doc comments, and Help changed), Pariston, Mafuuu (raised CRF-83, whose fix Netero questioned), Chopper (wildcard from Mafuuu, Chopper, Kite, Zoro, Razor). Killua not triggered: no new loop, cache, or allocation pattern. Closed: CRF-86 (1/1, root cause out of scope), CRF-90 (3/3), CRF-92 (3/3). Re-raised: CRF-10 (Komugi; Takumi accepted). New: 6 P2, 5 P3, 4 P4, 2 Nit posted; 1 merged into the CRF-10 reply; 1 out of scope; 4 verification Notes dropped. Ging-Go, Knov, Ryosuke, Takumi, Melody, Pariston, Chopper reported no new findings. Reviewed against 676c4fd..ab6ebb9.

Round 6 update

BLOCKED. CRF-109, CRF-110, CRF-112, CRF-114, CRF-115, CRF-117, CRF-126 silent. No review.
Churn guard: 11 addressed (CRF-10, CRF-111, CRF-113, CRF-118, CRF-119 to CRF-125), 0 acknowledged, 0 contested, 0 deferred, 7 silent out of 18 open findings. Head e85c63d rebases onto base 562baea (the #29771 stack) and adds commits after ab6ebb9. The author replied on two threads only: CRF-10, claimed fixed by clamping late stages to the turn end (b1e1dc7, TurnAccumulator.SetEnd, TestRunnerTurnSpanCountsStepRunningAtInterrupt); and CRF-86, claimed fixed in base fb6a859 (#29771), where the finishing task holds its turn until it returns and the outcome is decided at emission. The round 5 threads (CRF-110 to CRF-126) and the CRF-109 reply have no author reply. The churn guard left three addressed classifications for panel judgment: CRF-111 (doc deleted; the race test still cannot detect a missing lock), CRF-113 (resolved through the CRF-10 fix; Help unchanged), CRF-118 (EndWithoutObservation still says the stage counts toward its turn). Orchestrator checked the seven silent findings at head: category still skips provider_error for any context.Canceled (turnaccounting.go:207, comment changed only); guardedStream still blocks the stand-in by hand (chatloop.go:815); the turn_outcomes_total Help still omits the interrupting-edit abandoned case (metrics.go:182, ARCHITECTURE.md:815); the emitTurnAccounting doc and ARCHITECTURE.md:845 still claim a counted outcome implies a recorded partition; the EndTurn doc still lacks the reply-latency reason (turnaccounting.go:227); ARCHITECTURE.md:849 still has no overattributed exception; the categories parameter is unchanged. Statuses are not updated on a blocked round. CI passed (3 passed, 1 skipped). Reviewed against: 562baea..e85c63d.

Round 7 update

BLOCKED. CRF-109 deferred without a ticket. No review.
Churn guard: 16 addressed (CRF-10, CRF-111 to CRF-115, CRF-117 to CRF-126), 0 acknowledged, 1 contested (CRF-110), 0 deferred with ticket, 1 silent (CRF-109) out of the 18 findings open after round 5. Head ff9b980 adds ac01fda and ff9b980 on base 562baea after the round 6 head e85c63d. CRF-109 author reply (PRRC_kwDOGkVX1s72G15Z): "Accepted limitation, not fixing in this stack", citing the Turn accounting table rule that a stream ended by a context cancellation counts as streaming, and leaving the provider-side split "for follow-up" with no ticket; the only Linear link on the PR is CODAGT-1102, the PR's own linkback. Orchestrator checked at head: turnaccounting.go:207 still skips provider_error for any errors.Is(err, context.Canceled), and ARCHITECTURE.md:865 gives a chat stop and a shutdown as its examples of a context cancellation, and neither it nor the table rows at :854-856 name the silence guard or a transport reset. CRF-110 author defense (PRRC_kwDOGkVX1s72G1_B): the stand-in is the only tracer that must stay out of the turn, and a tracer-level opt-out adds a mechanism for one call site; code unchanged. Author replies claim fixes for CRF-112, CRF-114, CRF-115, CRF-117, CRF-126 (ac01fda, ff9b980); CRF-111, CRF-113, CRF-118 to CRF-125 have no reply and were classified from the code. Statuses are not updated on a blocked round. CI passed (3 passed, 1 skipped). Reviewed against: 562baea..ff9b980.

About deep-review

CRF = Coder Review Finding (P0-P4, Nit, Note)

Reviewer Focus
Bisky tests
Chopper ops/errors
Churn-guard change verification
Ging language modernization
Gon naming
Hisoka edge cases
Killua perf
Kite change integrity
Knov contracts
Knuckle SQL
Komugi flake/determinism
Kurapika security
Law decomposition
Leorio docs
Luffy product
Mafu-san process
Mafuuu contracts
Melody dispatch/pairing
Meruem structural
Nami frontend
Netero mechanical checks
Pariston premise testing
Pen-botter product gaps
Razor verification
Robin duplication
Ryosuke Go arch
Takumi concurrency
Zoro shape

🤖 Managed by Coder Agents.

Add a StageTracer that emits one span and one stage_duration_seconds
observation per chat lifecycle stage, with typed Stage, Scope, and
ChatKind values, a model_stage_duration_seconds histogram for the
provider-bound stages labeled by provider_type and model, and a
stage_anomalies_total counter for dropped observations. The metric
families are registered when the chat-stage-metrics experiment is
enabled; spans are emitted regardless. Extract the generation step body
into its own function so the step has a boundary for a deferred stage
end.
@jscottmiller
jscottmiller force-pushed the scott/chatd-lifecycle/3-turn-span branch from c2202f0 to 7a5cd22 Compare September 23, 2026 19:37
@jscottmiller
jscottmiller force-pushed the scott/chatd-lifecycle/4-turn-accounting branch from 2c98b6d to e070f77 Compare September 23, 2026 19:37
@jscottmiller
jscottmiller force-pushed the scott/chatd-lifecycle/3-turn-span branch from 7a5cd22 to 9185854 Compare September 23, 2026 20:27
@jscottmiller
jscottmiller force-pushed the scott/chatd-lifecycle/4-turn-accounting branch from e070f77 to ca0e2f4 Compare September 23, 2026 20:27
@jscottmiller

Copy link
Copy Markdown
Contributor Author

/coder-agents-review

@jscottmiller
jscottmiller force-pushed the scott/chatd-lifecycle/3-turn-span branch from 9185854 to 5ffb40e Compare September 23, 2026 21:04
@jscottmiller
jscottmiller force-pushed the scott/chatd-lifecycle/4-turn-accounting branch from ca0e2f4 to 11a1826 Compare September 23, 2026 21:05
@jscottmiller
jscottmiller removed this pull request from stack #29773 September 23, 2026 21:05
@linear-code

linear-code Bot commented Sep 24, 2026

Copy link
Copy Markdown

CODAGT-1102

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR turns overlapping stage histograms into a disjoint partition of turn wall time, and the core mechanism holds. Children report their full duration to the parent, TTFT nests under stream, the compaction summary stream stays out of TTFT through a nil tracer, and the ScopeTurn gate keeps background work out of a turn (Knov, Razor, Knuckle, and Meruem each verified that gate independently). Cardinality is bounded, TestRunnerTurnSpanCountsOutcomes checks exactly-once counting against closed spans, and the generated metrics docs match the scanner byte for byte. Killua and Kurapika found nothing.

Counts: 35 P2, 33 P3, 5 Nit, 4 Note. 28 of the P2s are comment-quality findings from Gon, and comment quality is a pattern here: the tool-execution rule is stated in four places, the emit-every-category rule in three, and the End, EndWithoutObservation, SetModel, and RecordAs docs never picked up their new turn-accounting side effects.

The behavioral findings share one theme: several categories hold time their labels do not describe. unattributed absorbs requires_action client waits (CRF-9), the finishing step's time when another task's Ensure closes the turn first (CRF-10), per-step state reloads (CRF-59), and task-level retry sleeps (CRF-47). scheduling absorbs prior failed attempts and failover (CRF-11) and the previous turn's runtime through queue_wait (CRF-12). chatd_overhead absorbs external hook dispatch and the FinishTurn transaction (CRF-45, CRF-60). Taken together, unattributed cannot yet serve as an instrumentation-gap signal, and scheduling and chatd_overhead cannot be read as chatd-owned costs. Separately, the outcome counter is split across two packages over a duplicated state machine (CRF-13, raised by ten reviewers), the concurrency rationale and its test describe a topology production never builds (CRF-14), and no chatd test reads the category family (CRF-15, with CRF-1).

Several findings interact. The CRF-13 restructure (have closeLocked own emission and record every outcome) also dissolves CRF-66 and simplifies CRF-56 and CRF-5, and CRF-55's alternative (partition every closed turn) removes the same flags. Dropping turns_total (CRF-54) changes the Help fix in CRF-53. Deleting stageNode.failed (CRF-49) conflicts with one fix option in CRF-48. Ryosuke and Zoro disagree on CRF-61; both positions are in the comment.

Rebase note: this review ran against 2c98b6dab. The head is now ca0e2f4da, rebased onto the updated base branch. Within this PR's own patch only the stage_anomalies_total Help string changed. Line numbers in stage.go, metrics.go, turn_trace.go, and stage_internal_test.go shifted because of base-stack changes, so some inline comments may show as outdated. The base changes affect CRF-48, CRF-49, CRF-63, and CRF-69; each of those comments says how.

Law assessed decomposition and advises against splitting: the partition, the outcome counter, the model stamp, and the generation-action plumbing form one reviewable idea.

CI: the failed Storybook job aborted after the snapshot service returned HTTP 500 on 11 uploads. It is unrelated to this PR, which touches no site/ files, and a rerun should clear it.

PR description: "every category has a series: 16 per chat kind" reads as a category count. There are 11 categories; 16 is 11 category series plus 4 outcome series plus 1 turns_total series (Mafu-san; Chopper confirmed the arithmetic).

From Hisoka: "I pulled the partition apart at every seam I could find. Most of it held."


coderd/x/chatd/generation.go:443

P3 [CRF-59] From step 2 on, the full-history state reload between steps lands in unattributed. It is uncovered persistence time, not an instrumentation remainder. (Knuckle)

Each later step first runs loadGenerationState (generation.go:633-660) on the plain task ctx. That opens a transaction, takes SELECT ... FOR SHARE on the chat row (chatstate/machine.go:264), and reads the whole history with GetChatMessagesByChatID(AfterID: 0). Only after that does Ensure hand back the turn context.

Capture loadStart and loadEnd around loadGenerationState, and pass them into Ensure. In the if t.open branch (turn_trace.go:119), Record a reconstructed turn-scoped stage (for example load_state) over that window. Then map it in recordedStageCategories to CategoryPersistence.

Knuckle had no production data, so the magnitude is predicted, not measured. The cost grows with chat length, so unattributed trends upward on long chats while persistence stays flat.

🤖

coderd/x/chatd/chatloop/stage.go:406

P3 [CRF-67] The End doc says End closes the span and records its duration, but End now also stamps the turn model on chat_turn, folds the stage into the turn partition, and, on chat_turn, emits the turn's metrics. (Gon P3, Leorio Nit)

The same gap appears in three sibling docs whose declarations this PR changed. SetModel (:374) does not say that the first model set inside a turn names the chat_turn span. The StageSpan type doc (:186) still says "End must be called exactly once; the duration observation happens there", which contradicts End's own "calls after the first are ignored" and leaves out the turn report. RecordAs (:472) does not say that turn-scoped recorded stages feed scheduling. (Gon)

A caller who adds SetModel to a new turn-scoped stage that runs before prepare changes which model the whole turn is labelled with, and the doc gives no warning. (Leorio)

Line numbers refer to 2c98b6dab; the rebase shifted stage.go.

🤖

coderd/x/chatd/chatloop/stage.go:417

P3 [CRF-68] The EndWithoutObservation doc says it "makes no duration observation", but it now feeds the stage's full elapsed time into turn_time_seconds_total, and "exactly as End does" is false because it skips adoptTurnModel. (Leorio P3, Gon P3)

The next developer reads "no duration observation", sees a duration landing in a counter, and "fixes" the inconsistency. After that fix, failed-TTFT time silently moves into streaming, because the stream is no longer marked failed. (Leorio)

Leorio's proposed doc:

// EndWithoutObservation closes the stage span like End but makes no
// histogram observation. It is for stages whose window is only
// comparable across runs when it completed, so a truncated window
// would skew the histogram while the span still needs to report the
// failure. The window still reports to the turn's time partition, so
// a failed time_to_first_token lands in provider_error rather than
// in its stream's own time.

🤖

coderd/x/chatd/turn_trace.go:280

Note [CRF-75] A promoted turn's partition overlaps the tail of the turn that released it, so summing turn_time_seconds_total across turns counts that wall time twice. (Hisoka)

The ratio rate(time)/rate(turns) means what the PR says it means, but sum(rate(turn_time_seconds_total)) is not "chatd busy seconds". The same thing happens across chat_kind: a parent's tool_execution includes the subagent turns it waited on, and those subagent turns are also accounted under chat_kind="subagent". Anyone building a capacity dashboard on these counters needs to know this.

One sentence in the ARCHITECTURE section would stop anyone building a capacity panel on the sum.

🤖

🤖 This review was automatically generated with Coder Agents.

Comment thread coderd/x/chatd/generation.go Resolved
Comment thread coderd/x/chatd/chatloop/metrics.go Resolved
Comment thread coderd/x/chatd/ARCHITECTURE.md Resolved
Comment thread coderd/x/chatd/chatloop/stage.go Resolved
@jscottmiller

Copy link
Copy Markdown
Contributor Author

/coder-agents-review

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR partitions the wall time of completed chatd turns into categories on turn_time_seconds_total and adds turn_outcomes_total and turns_total; 77 findings are open: 35 P2, 33 P3, 5 Nit, 4 Note.

Head 11a18260 rebases 2c98b6dab onto the new base; turnaccounting.go, turnaccounting_internal_test.go, and generation.go are unchanged, and no review thread has a reply. Further review is blocked until fixes are pushed or the threads get replies. For each finding below, reply on its thread with the fixing commit, a GitHub issue that tracks it, or the reason the code should stay. Line numbers are from 2c98b6dab, where the threads are anchored.

The failing doc-check job (its Coder user is suspended) and Storybook job (snapshot service HTTP 500, "api key lookup failed") are not caused by this PR.

P2, behavior and structure:

  • turnaccounting.go:372: the requires_action client wait stays inside the turn and is counted as unattributed [CRF-9]
  • turn_trace.go:221: another task's Ensure can close a finished turn before its final step ends, so where the step's time is counted depends on goroutine scheduling [CRF-10]
  • turnaccounting.go:73: acquisition re-covers earlier attempts (reopen after invalidation, failover) and counts them as scheduling [CRF-11]
  • turnaccounting.go:74: queue_wait in scheduling mixes pickup latency with the previous turn's remaining runtime [CRF-12]
  • turnaccounting.go:111: the accumulator's completed and invalid flags duplicate the runnerTurnSpan outcome, and turn_outcomes_total is counted from two packages [CRF-13]
  • turnaccounting.go:105: the concurrency comment names tool calls, which never reach the accumulator, and the concurrency test asserts an overattributed turn [CRF-14]
  • stage_internal_test.go:457: no chatd test reads turn_time_seconds_total; removing the accumulator from contextLocked fails no test [CRF-15]

P2, comments that restate the code or state a false or duplicated fact (one finding per comment):

  • stage.go: 324 [CRF-16], 428 [CRF-17]
  • turnaccounting.go: 9 [CRF-18], 30 [CRF-19], 47 [CRF-20], 65 [CRF-21], 77 [CRF-22], 101 [CRF-23], 115 [CRF-24], 133 [CRF-25], 181 [CRF-26], 213 [CRF-27], 265 [CRF-28], 348 [CRF-29]
  • generation.go: 128 [CRF-30]
  • turn_trace.go: 146 [CRF-31], 162 [CRF-32], 190 [CRF-33], 227 [CRF-34]
  • turnaccounting_internal_test.go: 70 [CRF-35], 76 [CRF-36], 104 [CRF-37], 119 [CRF-38], 171 [CRF-39], 217 [CRF-40], 297 [CRF-41], 352 [CRF-42], 691 [CRF-43]

P3:

  • generation.go:575: no test drives runGenerationStep into the turn partition [CRF-1]
  • turnaccounting_internal_test.go:434: the NilTracer subtest passes because of nil-receiver guards, not accounting logic [CRF-2]
  • turnaccounting_internal_test.go:692: hand-written registry walkers duplicate the sums helper and promhelp [CRF-3]
  • metrics.go:304: the unreachable elapsed < 0 guard in RecordTurnCategory would drop a category series without any signal [CRF-4]
  • turnaccounting.go:33: TurnTimeCategories (a mutable slice) and TurnAccumulator.Model are exported with no callers outside chatloop [CRF-5]
  • turn_trace.go:28: the runnerTurnSpan doc dropped the fact that stages still open at Complete end inside the turn span [CRF-6]
  • turnaccounting_internal_test.go:116: the main partition fixture records queue_wait before the turn starts, and the sum check passes because two errors cancel [CRF-44]
  • turnaccounting.go:277: hook dispatch to external endpoints is counted as chatd_overhead (or tool_execution) [CRF-45]
  • turnaccounting.go:274: tool steps count decide, hook, finish, and retry bookkeeping as tool_execution [CRF-46]
  • turnaccounting.go:284: retry_backoff covers only generation retries; phase and task-level backoff are counted in other categories [CRF-47]
  • turnaccounting.go:292: a failed attempt's TTFT and discarded responses are counted as time_to_first_token or streaming, which contradicts the ARCHITECTURE.md table [CRF-48]
  • turnaccounting.go:287: stageNode.failed has no test that fails without it and moves only microseconds [CRF-49]
  • turnaccounting.go:317: the own < 0 clamp duplicates the addCategory guard, so TestTurnAccountingClampsOwnTime cannot detect its removal [CRF-50]
  • metrics.go:191: the turn_time_seconds_total Help says categories sum to the turn duration, which is false for overattributed turns [CRF-51]
  • metrics.go:191: the turn_time_seconds_total Help lists no category values or definitions [CRF-52]
  • metrics.go:203: the turn_outcomes_total Help says completed turns are the ones with a recorded partition, which is false for nonpositive_turn, and no test checks the difference [CRF-53]
  • metrics.go:193: turns_total duplicates turn_outcomes_total{outcome="completed"} except after a clock step [CRF-54]
  • turn_trace.go:245: the partition covers only completed turns, so time in turns that end in a provider error or an interrupt is counted in no category [CRF-55]
  • stage.go:430: stamping the first model on chat_turn contradicts "a turn is not a model-scoped unit" [CRF-56]
  • turnaccounting.go:54: attributingStages and the category switch must list the same stages, and a mismatch fails no test [CRF-57]
  • turnaccounting.go:375: the overattributed anomaly has no chat_kind label or span attribute that identifies the turn [CRF-58]
  • generation.go:443: from step 2 on, loadGenerationState (a full history read) is counted as unattributed [CRF-59]
  • turnaccounting.go:281: the FinishTurn transaction and the afterGenerationOutcome history read are counted as chatd_overhead, not persistence [CRF-60]
  • stage.go:41: chatloop now owns one value of chatd's generation-action enum, and SetGenerationAction has an effect only on generation_step [CRF-61]
  • stage_internal_test.go:668: the afterInterrupt turn anchored at time.Now() can have zero duration on coarse clocks [CRF-62]
  • turnaccounting_internal_test.go:33: newTurnFixture duplicates newStageMetricsFixture [CRF-63]
  • turnaccounting.go:13: Category lacks the Turn prefix its sibling types use and shares a name with refusal.Category [CRF-64]
  • turnaccounting.go:340: recordAttribution handles exactly the stages that are not in attributingStages [CRF-65]
  • turnaccounting.go:304: report emits the whole turn's metrics, but its name and doc describe folding one stage into its parent [CRF-66]
  • stage.go:406: the End, SetModel, StageSpan, and RecordAs docs omit the new turn-accounting effects [CRF-67]
  • stage.go:417: the EndWithoutObservation doc says "exactly as End does" and "no duration observation", and both are false now [CRF-68]
  • metrics.go:291: the RecordStageAnomaly doc dropped "adjusted", and the RecordAnomaly doc (stage.go:506) was not updated [CRF-69]
  • stage.go:197: the acc field doc is wrong for background-scoped stages that run inside a turn [CRF-70]

Nit:

  • capacity_internal_test.go:23 (and turn_trace_internal_test.go:119, :144): the newStageTestTracer to newStageMetricsTracer swaps discard the registry and change no behavior [CRF-7]
  • turnaccounting_internal_test.go:106: syntheticTurn returns an accumulator no caller uses [CRF-71]
  • ARCHITECTURE.md:844: "emit nothing" is contradicted two lines later by outcome counting [CRF-72]
  • turnaccounting_internal_test.go:532: use wg.Go [CRF-73]
  • turnaccounting.go:197: use maps.Clone [CRF-74]

Note:

  • turn_trace.go:280: summing categories across turns or chat kinds counts overlapping wall time twice [CRF-75]
  • ARCHITECTURE.md:852: the queue_wait recorded by PromoteQueued never reaches scheduling [CRF-76]
  • stage.go:328: the scope and the accumulator are two turn-membership signals that can disagree [CRF-77]
  • turnaccounting.go:342: the acc == nil branch of recordAttribution has no chatloop test coverage [CRF-78]

🤖 This review was automatically generated with Coder Agents.

@jscottmiller

Copy link
Copy Markdown
Contributor Author

/coder-agents-review

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR partitions the wall time of every chatd turn into categories on turn_time_seconds_total. The new commits and replies answer 17 of the 18 open findings; the reply on the stand-in tracer finding contests it, and the reviewers who raised it will judge that defense when review resumes.

One P3 remains. Its reply accepts the behavior as a limitation and leaves the fix for a follow-up with no ticket. Further review is blocked until it has a fix in this PR, a linked GitHub issue that tracks the follow-up, or a maintainer's explicit acceptance of the gap. The line number is from ff9b9809.

  • P3 [CRF-109] turnaccounting.go:207: any error that wraps context.Canceled skips provider_error, so when the silence guard or a transport reset cancels a stream while the turn is live, up to StreamSilenceTimeout of provider stall per attempt is counted as streaming. The documented rule gives a chat stop and a shutdown as its examples of a context cancellation and does not mention either provider-side case.

🤖 This review was automatically generated with Coder Agents.

Scott Miller added 18 commits September 29, 2026 03:19
- Delete the failed flag on stage nodes: a stream's own time is
  provider_error only when the stream ends with an error, so the rest of
  a stream whose first-token window failed is streaming.
- Take a recorded stage's window out of the own time of the attributing
  stage it is recorded under.
- Extend the stage set consistency test to the turn accounting sets and
  the turn_time_seconds_total help text.
- Remove capacity_wait from the accounting docs and tests.
… end

Closing a turn fixes its end time on the accumulator. A stage that
reports afterwards counts only the part of its window before that time,
for parents and children alike, so a step still unwinding when an
interrupt settles its turn keeps its category up to the interrupt
instead of landing in unattributed. EndTurn takes the turn's end time.

Generated by Coder Agents.
@jscottmiller
jscottmiller force-pushed the scott/chatd-lifecycle/4-turn-accounting branch from ff9b980 to dd49b8e Compare September 29, 2026 03:26
@jscottmiller
jscottmiller removed this pull request from stack #29877 September 29, 2026 03:26
@jscottmiller
jscottmiller marked this pull request as draft September 29, 2026 03:27
@jscottmiller
jscottmiller force-pushed the scott/chatd-lifecycle/3-turn-span branch 2 times, most recently from 1be3031 to 93913d8 Compare September 30, 2026 14:02
An error occurred while trying to automatically change base from scott/chatd-lifecycle/3-turn-span to scott/chatd-lifecycle/2-provider-stages September 30, 2026 16:54
jscottmiller pushed a commit that referenced this pull request Sep 30, 2026
**TL;DR:** Adds a stage tracer that turns each chat lifecycle stage into
both an OpenTelemetry span and a Prometheus histogram observation, so
traces and metrics always agree. The metrics are only registered with
the `chat-stage-metrics` experiment. Nothing calls the tracer until the
next PRs.

Adds `chatloop.StageTracer`, which emits each chat lifecycle stage as an
OpenTelemetry span and a `coderd_chatd_stage_duration_seconds{stage,
scope, chat_kind}` observation from one `End` call, so trace and metric
durations cannot disagree. The provider-bound stages (`stream`,
`provider_attempt`) are also observed on
`coderd_chatd_model_stage_duration_seconds{stage, provider_type,
chat_kind, model}`, the only family carrying `provider_type` and
`model`, and only for turn-scoped stages. Time to first token per model
stays on the existing `coderd_chatd_ttft_seconds`. Nothing calls the
tracer yet; the next PRs instrument the provider call and the runner.

- The families are registered only when the `chat-stage-metrics`
experiment is enabled (safe and known, so
`--experiments=chat-stage-metrics` or `--experiments=*` turns it on).
Spans are emitted regardless. `NewMetrics` leaves the families off;
`chatd.New` opts in from the experiment, and a test asserts the
experiment decides whether they reach the server's registry.
- `Stage`, `Scope`, `ChatKind`, and `StageAnomaly` are typed string
constants. `scope` separates latency attributable to a prompt (`turn`)
from detached `background` work; `chat_kind` is `root` or `subagent`,
empty without a known chat.
- `StageModel` carries the wire `provider`, `provider_type`, `model`,
and `reasoning_effort`. All are span attributes; only `provider_type`
and `model` are metric labels, on the model family. No family carries an
organization label.
- Observed stages: `chat_turn`, `queue_wait`, `acquisition`,
`mcp_connect`, `stream`, `time_to_first_token`, `provider_attempt`,
`tool_call`, `commit`, `retry_backoff`. `generation_step`, `prepare`,
`thinking`, and `compaction` are span-only. A test keeps the stage sets
and help text consistent.
- API: `Start` / `StartRootAt` for live spans (span timestamps come from
the tracer clock), `Record` for stages reconstructed from timestamps
(the scope comes from the context). `End` is idempotent;
`EndWithoutObservation` ends the span without a sample. A nil tracer,
provider, or metrics discards.
- `coderd_chatd_stage_anomalies_total{reason}` counts observations
dropped, adjusted, or inconsistent (`negative_elapsed`,
`inverted_window`, `missing_timestamp`, `future_start`, `stale_anchor`).
Span-only stages never feed it.
- Buckets are 12 round edges (`0.1 0.25 0.5 1 2.5 5 10 30 60 300 1800
3600`).
- Moves the body of the `for` loop in `StartGeneration` into
`runGenerationStep` so the step has a function boundary for a deferred
stage end. Control flow and error wrapping are unchanged.

<details>
<summary>Series cost (approximate, whole stack)</summary>

A histogram label set is 12 edges + `+Inf` + `_sum` + `_count`: 15
series. Per replica, each `chat_kind` costs about 150 stage series and 4
turn outcome counters; each `(chat_kind, model)` costs 30 (2 stages x
15). Both chat kinds on `M` models: about `308 + 60*M` per replica, plus
background-scope stage series.

Dogfood `main`, 3 replicas: ~3.4k `coderd_*` series per replica (~10.1k
total).

| Models used per replica | Added series (3 replicas) | vs. baseline |
|---:|---:|---:|
| 1 | ~1.1k | +11% |
| 5 | ~1.8k | +18% |
| 10 | ~2.7k | +27% |
| 35 | ~7.2k | +72% |

Most deployments run four or fewer models, so a typical replica adds
roughly 370 to 550 series. The experimental turn accounting in #29772
would add 88 more per replica (11 categories x 4 outcomes x 2 chat
kinds). Series appear lazily and do not grow with organizations.

</details>

<details>
<summary>Review decisions</summary>

- Experiment gate only; no deployment flag or metric levels. If the
experiment graduates, the families become always on like the other chatd
families; a flag is added then only if dogfood shows a cardinality
problem.
- Organization is on neither metrics nor spans, and `effort` is
span-only; `provider_type` (configured type), not the wire `provider`,
is the model-family label.
- `time_to_first_token` is not on the model family; per-model TTFT is
`ttft_seconds`, which observes the same measured window.
- The model family records only turn-scoped stages.
- Nil guards are kept so a nil or zero tracer is always safe to call.
- HTTP attribute keys stay snake_case like the rest of the chatd span
attributes.

</details>

Supersedes #28968 and #28969.

🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.
jscottmiller pushed a commit that referenced this pull request Sep 30, 2026
**TL;DR:** Wraps each prompt in a `chat_turn` root span with every
lifecycle stage parented under it. The span records one outcome per turn
(`completed`, `interrupted`, `error`, `abandoned`), and the new
`coderd_chatd_turn_outcomes_total` counter counts those outcomes. Turns
stay correct when tasks are canceled, replaced, or retried.

Each prompt runs inside a `chat_turn` root span owned by the runner
(`runnerTurnSpan`). The generation step, prepare, MCP connect, stream,
TTFT, provider attempt, commit, thinking, tool call, compaction, and
retry backoff stages are parented under it, activating the stages from
#29770. `chat_turn` spans are standalone roots: the turn runs on a
worker, often on another replica, and no trace context is persisted.

**Lifecycle**

- **Start.** Each generation loop iteration calls `Ensure`, which opens
the turn if none is open and returns a `turnToken`. The span is
backdated to the trigger: the latest of the last user prompt, a dynamic
tool's result message after it, and a pending compaction request. An
`acquisition` stage covers trigger to pickup.
- **Stale trigger / takeover.** A trigger at or before the previous
anchor opens the turn at now with no `acquisition` and counts
`stale_anchor`. A runner that took the chat over from an owner with a
stale heartbeat (`TakenOver`, from acquisition) opens its first turn at
now with no `acquisition`, so the dead owner's run is not reported as
pickup delay.
- **Newer prompt.** `Ensure` on the current turn with a newer trigger
closes it and opens a new one. The closed turn is `abandoned` unless the
task that ran it records an outcome before releasing it.
- **Finish.** After `FinishTurn` or `EnterRequiresAction` commits, the
step calls `Complete`; `Settle` closes the span after the step's own
stage ends, so the finishing step is inside the turn. A chat waiting in
`requires_action` is outside any turn; the generation resumed by its
tool results opens a new one.
- **Emission.** Closing a turn fixes its end time; its span is emitted
once every task that joined it (`Ensure` takes the task ID) has called
`Release`, which `runTask` does after the task returns. A finishing task
whose `FinishTurn` promoted a queued prompt therefore still counts its
turn as `completed` even when the promoted task's `Ensure` runs before
its `Complete`. `End` at runner exit emits whatever is left.
- **Failure.** A step that commits `FinishError`, or fails closed,
invalidates the turn as `error` after the commit. Errors the task runner
retries, including task timeouts, keep the same turn open; an expected,
non-retryable exit records no outcome and leaves the turn open until the
interrupt task, a newer prompt, or runner exit closes it. A canceled
generation task records no outcome and cannot obtain a turn token, so a
replaced task never touches its successor's turn.
- **Interrupt.** The interrupt task marks the open turn `interrupted`
before `FinishInterruption` commits and closes it after. Only a stop (by
the user or a parent agent) is `interrupted`; it ends with unset span
status, since error status is reserved for `error` turns. runner
shutdown, loss of ownership, and message edits close the turn as
`abandoned` with unset status.
- **Promotion.** Every promoting transition (`FinishTurn`,
`FinishInterruption`, a send to an errored chat, `PromoteQueued`)
returns the promoted message's queued time in its result, and one
`recordQueueWait` helper records a standalone `queue_wait` to the
promotion. The promoted turn opens at the promotion time.

`chat_turn` is observed on `coderd_chatd_stage_duration_seconds` only
for completed turns, so the histogram is completed-turn latency; every
turn still emits its span.

`coderd_chatd_turn_outcomes_total{outcome, chat_kind}` counts every
closed turn exactly once (`completed`, `interrupted`, `error`,
`abandoned`), matching the span's `turn_outcome`, under the
`chat-stage-metrics` experiment. Turn time accounting is in draft
#29772.

**Stages and identity**

- `tool_call` is a live stage around each local tool call in `chatloop`;
the tool runs on its context, and it ends errored only when the tool
fails to execute. Calls that never run (inactive, unknown, or rejected
by policy) record no stage.
- `mcp_connect` carries connected and failed server counts and ends
errored only when servers were configured and none connected.
- The compaction stage carries the model that runs the summary.
- `withStageIdentity` puts scope and chat kind on the turn context.
Detached background work runs in `background` scope as its own trace
roots.
- The `advisor` tool's nested model call appears only as its enclosing
`tool_call`.

ARCHITECTURE.md gains a "Lifecycle tracing" section.

<details>
<summary>Review decisions</summary>

- Turn identity stays timestamp-based with targeted fixes (clamped
acquisition, rotation on a newer trigger, invalidate after commit,
fail-closed invalidation, settle closing invalidated turns) plus a
takeover flag; keying turns on message IDs would not detect takeover and
cannot express interrupted/abandoned outcomes.
- Outcome classification: the interrupt task owns `interrupted`, and
cancellation or a fence mismatch alone never records an outcome, so
shutdown, ownership loss, and edits are `abandoned`. Only `error` turns
set error span status. Remaining ordering races between a step and an
interrupt are documented as known limitations in ARCHITECTURE.md.
- Turns are emitted after the tasks that ran them return, not when they
close, so an outcome a replaced task records while unwinding reaches its
own turn; the successor task is not delayed. Chosen over an
after-commit, pre-publish hook on `ChatMachine.Update`, which would fix
only the `FinishTurn` ordering.
- `requires_action` ends the turn rather than pausing it, so `chat_turn`
measures chatd time; a prompt with client tools counts as several turns.
- Promoted turns anchor at promotion time and `queue_wait` is always
standalone, so `chat_turn` never includes time spent queued behind
another turn.
- The former `capacity_wait` stage (#29805) was dropped: it was a
per-worker lower bound, excluded from accounting, and needed its own SQL
and bookkeeping. #29805's remaining helpers are folded into this PR.

</details>

Supersedes #28972, #28973, and #29805.

🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 1, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant