Repository navigation
feat: trace chatd provider stages and count only output parts as first token - #29770
Conversation
26cae76 to
9e8add5
Compare
Docs previewCheck off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here. |
9e8add5 to
aacc28a
Compare
|
/coder-agents-review |
|
Chat: Review posted | View chat Review historydeep-review v0.13.0 | Round 4 | Last posted: Round 4, 40 findings (2 P2, 16 P3, 9 P4, 12 Nit, 1 Note), COMMENT. Review Finding inventoryFinding inventory: PR #29770Findings
Contested and acknowledgedCRF-2 (P2, chatloop.go:126) - No production caller sets Stages/StageModel
CRF-5 (P3, stage_internal_test.go:128) - Duplicated stage test fixtures
CRF-8 (P3, model_routing_aibridge.go:157) - Failed windows observed into stage histograms
CRF-10 (P3, model_routing_aibridge.go:132) - provider_attempt spans the gateway call
CRF-12 (P3, chatloop.go:863) - Start markers close the TTFT window
CRF-19 (P3, chatloop.go:830) - TTFT observed into multiple histograms
CRF-24 (Nit, chatd/stage_internal_test.go:73) - wantStatusCode naming
CRF-47 (P4, modelcall.go:89) - Stage identity derived twice
Law analysisR4 (first Law run for this PR): effective +1231 -274 (15 files) at head e6f36ac; Law counts about 169 hand-written production lines (the rest of the 418 is the generated Round logRound 1Netero-only. 2 P2, 3 P3, 1 Note. Panel skipped per Netero gate (P2 present). Orchestrator reproduced CRF-1 with a throwaway test (Warnings then Error, and Finish then Error, both observed as successful TTFT with span status Unset) and confirmed CRF-2 by grepping production Round 2Tool v0.12.0. Stack base changed from 76818c9 to 5e32cbb (revised step 1 commit adding Round type: full panel. No prior panel round existed, so the whole diff was new to the panel (not a fix round). Panel (13): specialists ging-go, Ryosuke, Takumi, Melody, Killua, Luffy, Pen Botter (docs row for a changed operator-facing metric); core Pariston, Mafuuu; Bisky (tests changed); Gon and Leorio (once-per-PR floor, first panel round); wildcard Kite (random from distilled set). Size exceeds the focused target (6-8) because seven specialists triggered plus mandatory core, Bisky, Gon, Leorio. Cross-check: 13 P3, 1 P4, 7 Nit posted; 4 Note (body); 3 out of scope; 5 dropped. Convergence: CRF-8 (4 reviewers, Kite reproduced a 429 observation), CRF-9 (4 reviewers, 3 reproduced the constant host), CRF-11 (Mafuuu and Takumi both reproduced). CRF-9 set to P3 by the tiebreaker (Ryosuke P3 vs three P4). CRF-10 set to P3 over Pen Botter's Note. Severity adjustments with the keep-argument written first: CRF-13, CRF-14, CRF-15, CRF-16 (Gon P2 on comment accuracy): keep-argument is that each comment states behavior the code does not have and no follow-up will correct it; set to P3 because the consequence is a misread by a code reader with no runtime effect, and Gon applied P2 uniformly to all 20 comment items without consequence reasoning. CRF-14 also over Leorio P4/Ryosuke Nit/Melody Note. Gon's verbosity items grouped into CRF-25, CRF-26, CRF-27 as Nits under the AGENTS.md rule that comments be substantive and concise; keep-argument for P2 is the same repo rule, rejected because verbosity has no consequence beyond reading time. Contradictions: Bisky's prior-findings remark that all non-error/warnings/finish parts are model output versus Pariston's reproduced start-marker case (CRF-12); Pariston's evidence (reproduction plus the help text's "content part" wording) kept CRF-12. Pariston's premise that the nil tracer keeps compaction out of turn accounting contradicts Gon/Leorio grep evidence (CRF-14). Leorio praised the comments Gon flagged at model_routing_aibridge.go:155/161 and chatloop.go:358, so those Nits were dropped (CRF-35 to CRF-37). Praise-only notes (Luffy x2, Kite, Leorio model_routing_aibridge.go:155) not recorded as findings. Killua: no findings. Reviewed against 5e32cbb..3809e82. Round 3Tool v0.13.0. Stack base changed from 5e32cbb to ced30e1 (revised #29769). New commits: 9026572 (rebased feature), 5bb414f, e761a9d, adeadee. Churn guard PROCEED: 17 addressed, 4 contested (CRF-8, CRF-10, CRF-12, CRF-24); orchestrator also treated CRF-19 as contested because the author kept two of the three series and defended the remainder (churn guard had classified it addressed). CRF-38 reopened: its R2 drop assumed CRF-10 would rewrite the same doc, but CRF-10 was contested. CRF-28 superseded (Netero: Round type: restructure (full panel). Dispute check-in: all five contested findings closed by every persona that raised them (CRF-8 4/4, CRF-10 2/2, CRF-12 1/1, CRF-19 1/1, CRF-24 1/1). Gon re-raised CRF-38 at model_routing_aibridge.go:134; Pen Botter added that the doc's first sentence says the round trip goes "to the model provider" when it goes to the AI Gateway. Cross-check: new 6 P4 and 3 Nit (CRF-41 to CRF-49), plus Netero's CRF-40 Nit and reopened CRF-38 Nit; 11 posted. Leorio's PR-description P4 folded into Meruem's CRF-47 (same root: stage identity derived in two places; Meruem's fix makes the description true). Meruem's consequence claim that Round 4Tool v0.13.0. Stack base changed from ced30e1 to b7d8353 (revised #29769, drops unused stage tracer API). Round 3 commits rebased (9c79842, 807436b, 5e26b8d, 4b47b43; range-diff shows only base-driven hunk changes). New commit: e6f36ac. Churn guard PROCEED: 10 addressed (CRF-38, CRF-40 to CRF-46, CRF-48, CRF-49), 1 contested (CRF-47). Orchestrator also marked Note (body) items CRF-29 (stream span now errored for content-filter/no-output), CRF-30 (title names the TTFT change), and CRF-31 (description states the #29771 dependency) as author fixed, and CRF-32 resolved in the base (Ryosuke). Inventory validation: dropped CRF-35 to CRF-37 and CRF-39 re-checked; e6f36ac extends the chatloop.go:358 ordering comment (CRF-37) but the drop's reason still holds. CRF-2 dependency re-checked: #29771 ( Law ran for the first time (effective additions 1231 > 1000 and no prior Law analysis; round 3 at 1146 met the same condition and Law did not run). Verdict Don't split, advisory. Netero (new agent) raised CRF-50. Round type: fix round. No files added, renamed, or moved ( Dispute check-in: CRF-47 closed 2/2 (Meruem, Leorio). Meruem withdrew its round 3 claim that metric series would diverge. Cross-check: 2 P4 posted (CRF-50, CRF-51). CRF-51 combines Pen Botter's P4 (description says the stream span records the returned error; lines 351 and 375 record the unwrapped error, pinned by About deep-reviewCRF = Coder Review Finding (P0-P4, Nit, Note)
|
There was a problem hiding this comment.
First-pass review only. These are mechanical findings from Netero. The full review panel has not reviewed this PR yet and will review after these findings are addressed.
Closing the TTFT window through one sync.Once is a clean way to keep both histograms success-only, and TestNewModelTracesEachProviderAttempt drives a 429 and then a 200 through the SDK client that newModel builds, so the one-span-per-attempt claim is tested end to end rather than against a stub. TestResolveModelCallResolvedEffort also checks the clamped effort in the provider options, not only on the span.
2 P2, 3 P3, 1 Note. CRF-1 was reproduced with a throwaway test against aacc28a: a warnings part followed by an error part is observed as a successful first token on both histograms. CRF-2 needs a decision on the advisor runner, which no PR in the stack wires.
"The result is correct only because
ResolveReasoningEffortis idempotent on its own output." (Netero)
🤖 This review was automatically generated with Coder Agents.
71d3e86 to
3809e82
Compare
|
/coder-agents-review |
There was a problem hiding this comment.
This PR adds stream, time_to_first_token, and provider_attempt stages around the chatd provider call and narrows what closes the coderd_chatd_ttft_seconds window. This round has 13 P3, 1 P4, and 7 Nit findings.
The warnings and finish handling, the endStream ordering tests, the single effort input to ProviderOptionsForCall, and the shared mock clock in the test fixtures are in 3809e82. The production chatd.New still builds the tracer without the server clock (comment on chatd.go:2996). The reviewer that raised the unwired Stages fields and the duplicated test fixtures accepted both replies; the first depends on #29771 merging with this PR, which sets Stages in generation.go and documents the advisor exclusion in ARCHITECTURE.md.
Out of scope (needs a ticket or explicit acceptance by a human):
coderd/x/chatd/chatloop/stage.go:151: theStageModel.Effortdoc says the effort is "sent to the provider", butapplyReasoningEffortmaps or drops it for some models, so every stage that carriesreasoning_effortinherits a wrong definition.coderd/x/chatd/chatloop/metrics.go:175:stage_duration_secondsandmodel_stage_duration_secondshave no outcome label, so no query can separate failed windows from completed ones for any stage that observes failures.- CI: the
StorybookandAnalyze PR for Documentation Updates Neededjobs fail on this head although the PR changes nosite/files; the cause is unverified because the job logs could not be read.
Notes:
coderd/x/chatd/chatloop/stage_internal_test.go:468: the ordering assertion compares OTel span end times, which come from the wall clock rather than the fixture's mock clock, so a reorder that ends both spans on the same tick still passes.coderd/x/chatd/chatloop/chatloop.go:378:endStream(nil)runs before the content-filter and no-output checks, so thestreamspan reads success for attempts thatGenerateAssistantthen returns as errors.coderd/x/chatd/chatloop/metrics.go:166:coderd_chatd_ttft_secondshas shipped since v2.33.0 and the PR title does not mention its changed meaning, so release notes built from the title will not explain the step in the series.- PR description: it does not say that without #29771 production emits only
provider_attemptstages.
🤖 This review was automatically generated with Coder Agents.
3809e82 to
adeadee
Compare
|
/coder-agents-review |
There was a problem hiding this comment.
This PR adds stream, time_to_first_token, and provider_attempt stages around the chatd provider call and changes what closes the coderd_chatd_ttft_seconds window. This round has 6 P4 and 5 Nit findings.
The round 2 fixes in 5bb414f and adeadee are in place and covered by tests, including the server clock on the tracer, the silence-timeout error on the TTFT span, and the end-to-end identity test with the copilot row. The reviewers that raised failed-window observation, the gateway-level provider_attempt span, start markers closing the TTFT window, the two TTFT series, and the wantStatusCode name accepted the replies. The stageSpanRoundTripper doc comment is raised again because the doc was not rewritten. Accepting the unwired Stages fields still depends on #29771 merging with this PR.
Out of scope (needs a ticket or explicit acceptance by a human):
coderd/x/chatd/chatloop/stage.go:149: theStageModel.Effortdoc says the effort is "sent to the provider", while theresolvedEffortcomment inmodelcall.gonow says it is the value before provider mapping, so the two comments contradict each other and span readers get the wrong definition.coderd/x/chatd/chatloop/metrics.go:174:stage_duration_secondsandmodel_stage_duration_secondshave no outcome label, so no query can separate failedstreamandprovider_attemptwindows from completed ones now that both are observed.
Notes:
coderd/x/chatd/chatloop/chatloop.go:380:endStream(nil)runs before the content-filter and no-output checks, so thestreamspan reads success for attempts thatGenerateAssistantthen returns as errors.coderd/x/chatd/chatloop/metrics.go:167:coderd_chatd_ttft_secondshas shipped since v2.33.0 and the PR title does not mention its changed meaning, so release notes built from the title will not explain the step in the series.- PR description: it does not say that without #29771 production emits only
provider_attemptstages.
🤖 This review was automatically generated with Coder Agents.
adeadee to
e6f36ac
Compare
|
/coder-agents-review |
There was a problem hiding this comment.
This PR adds stream, time_to_first_token, and provider_attempt stages around the chatd provider call and changes what closes the coderd_chatd_ttft_seconds window. This round has 2 P4 findings.
The round 3 fixes in e6f36ac are in place: the stream span now ends from one deferred call after the attempt is released, content-filter and no-output attempts end it errored, and the new tests fail when the span context or the tracer provider is dropped. The title and description now name the TTFT change and the #29771 dependency. The reviewers that raised the two-place stage identity derivation accepted the reply. Accepting the unwired Stages fields still depends on #29771 merging with this PR.
Out of scope (needs a ticket or explicit acceptance by a human):
coderd/x/chatd/chatloop/metrics.go:174:stage_duration_secondsandmodel_stage_duration_secondshave no outcome label, so no query can separate failedstreamandprovider_attemptwindows from completed ones now that both are observed.
🤖 This review was automatically generated with Coder Agents.
0510e54 to
dc303a5
Compare
dc303a5 to
a434ea5
Compare
… once - Stamp the wire provider from StageModel on stream, time_to_first_token, and provider_attempt stages instead of setting it by hand. - Observe coderd_chatd_ttft_seconds with the elapsed time the time_to_first_token stage records, so both measure one window. - Regenerate metric docs.
…error on every path
a434ea5 to
99a939c
Compare
**TL;DR:** Wraps each prompt in a `chat_turn` root span with every lifecycle stage parented under it. The span records one outcome per turn (`completed`, `interrupted`, `error`, `abandoned`), and the new `coderd_chatd_turn_outcomes_total` counter counts those outcomes. Turns stay correct when tasks are canceled, replaced, or retried. Each prompt runs inside a `chat_turn` root span owned by the runner (`runnerTurnSpan`). The generation step, prepare, MCP connect, stream, TTFT, provider attempt, commit, thinking, tool call, compaction, and retry backoff stages are parented under it, activating the stages from #29770. `chat_turn` spans are standalone roots: the turn runs on a worker, often on another replica, and no trace context is persisted. **Lifecycle** - **Start.** Each generation loop iteration calls `Ensure`, which opens the turn if none is open and returns a `turnToken`. The span is backdated to the trigger: the latest of the last user prompt, a dynamic tool's result message after it, and a pending compaction request. An `acquisition` stage covers trigger to pickup. - **Stale trigger / takeover.** A trigger at or before the previous anchor opens the turn at now with no `acquisition` and counts `stale_anchor`. A runner that took the chat over from an owner with a stale heartbeat (`TakenOver`, from acquisition) opens its first turn at now with no `acquisition`, so the dead owner's run is not reported as pickup delay. - **Newer prompt.** `Ensure` on the current turn with a newer trigger closes it and opens a new one. The closed turn is `abandoned` unless the task that ran it records an outcome before releasing it. - **Finish.** After `FinishTurn` or `EnterRequiresAction` commits, the step calls `Complete`; `Settle` closes the span after the step's own stage ends, so the finishing step is inside the turn. A chat waiting in `requires_action` is outside any turn; the generation resumed by its tool results opens a new one. - **Emission.** Closing a turn fixes its end time; its span is emitted once every task that joined it (`Ensure` takes the task ID) has called `Release`, which `runTask` does after the task returns. A finishing task whose `FinishTurn` promoted a queued prompt therefore still counts its turn as `completed` even when the promoted task's `Ensure` runs before its `Complete`. `End` at runner exit emits whatever is left. - **Failure.** A step that commits `FinishError`, or fails closed, invalidates the turn as `error` after the commit. Errors the task runner retries, including task timeouts, keep the same turn open; an expected, non-retryable exit records no outcome and leaves the turn open until the interrupt task, a newer prompt, or runner exit closes it. A canceled generation task records no outcome and cannot obtain a turn token, so a replaced task never touches its successor's turn. - **Interrupt.** The interrupt task marks the open turn `interrupted` before `FinishInterruption` commits and closes it after. Only a stop (by the user or a parent agent) is `interrupted`; it ends with unset span status, since error status is reserved for `error` turns. runner shutdown, loss of ownership, and message edits close the turn as `abandoned` with unset status. - **Promotion.** Every promoting transition (`FinishTurn`, `FinishInterruption`, a send to an errored chat, `PromoteQueued`) returns the promoted message's queued time in its result, and one `recordQueueWait` helper records a standalone `queue_wait` to the promotion. The promoted turn opens at the promotion time. `chat_turn` is observed on `coderd_chatd_stage_duration_seconds` only for completed turns, so the histogram is completed-turn latency; every turn still emits its span. `coderd_chatd_turn_outcomes_total{outcome, chat_kind}` counts every closed turn exactly once (`completed`, `interrupted`, `error`, `abandoned`), matching the span's `turn_outcome`, under the `chat-stage-metrics` experiment. Turn time accounting is in draft #29772. **Stages and identity** - `tool_call` is a live stage around each local tool call in `chatloop`; the tool runs on its context, and it ends errored only when the tool fails to execute. Calls that never run (inactive, unknown, or rejected by policy) record no stage. - `mcp_connect` carries connected and failed server counts and ends errored only when servers were configured and none connected. - The compaction stage carries the model that runs the summary. - `withStageIdentity` puts scope and chat kind on the turn context. Detached background work runs in `background` scope as its own trace roots. - The `advisor` tool's nested model call appears only as its enclosing `tool_call`. ARCHITECTURE.md gains a "Lifecycle tracing" section. <details> <summary>Review decisions</summary> - Turn identity stays timestamp-based with targeted fixes (clamped acquisition, rotation on a newer trigger, invalidate after commit, fail-closed invalidation, settle closing invalidated turns) plus a takeover flag; keying turns on message IDs would not detect takeover and cannot express interrupted/abandoned outcomes. - Outcome classification: the interrupt task owns `interrupted`, and cancellation or a fence mismatch alone never records an outcome, so shutdown, ownership loss, and edits are `abandoned`. Only `error` turns set error span status. Remaining ordering races between a step and an interrupt are documented as known limitations in ARCHITECTURE.md. - Turns are emitted after the tasks that ran them return, not when they close, so an outcome a replaced task records while unwinding reaches its own turn; the successor task is not delayed. Chosen over an after-commit, pre-publish hook on `ChatMachine.Update`, which would fix only the `FinishTurn` ordering. - `requires_action` ends the turn rather than pausing it, so `chat_turn` measures chatd time; a prompt with client tools counts as several turns. - Promoted turns anchor at promotion time and `queue_wait` is always standalone, so `chat_turn` never includes time spent queued behind another turn. - The former `capacity_wait` stage (#29805) was dropped: it was a per-worker lower bound, excluded from accounting, and needed its own SQL and bookkeeping. #29805's remaining helpers are folded into this PR. </details> Supersedes #28972, #28973, and #29805. 🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.
TL;DR: Traces the model provider call: a
streamstage, a nestedtime_to_first_tokenstage, and oneprovider_attemptper HTTP round trip, labelled with provider type and model. Behavior change:coderd_chatd_ttft_secondsnow counts only a real output part as the first token, so error, warning, and finish parts no longer count.Instruments the provider call.
Stream and time to first token.
GenerateAssistantrecords astreamstage from stream open to close and a nestedtime_to_first_tokenstage. The TTFT window closes once, on the first streamed output part: any part other than warnings, finish, or error, including start markers such astext_start. A closed window observes the stage and the pre-existingcoderd_chatd_ttft_secondswith the same elapsed time. A failed open, an error part, or a release without an output part ends the TTFT span with the error (a silence timeout is classified as such) and observes nothing. Thestreamspan ends once, with the errorGenerateAssistantreturns, so content-filter and no-output attempts end errored. Failedstreamwindows are observed like successful ones. The attempt is released before the stream span ends, so the TTFT window always falls inside its stream. The compaction summary stream passes a nil tracer, so it records neither.Without #29771, production emits only
provider_attempt; #29771 passes the tracer toGenerateAssistant.Provider attempts. The provider HTTP transport is wrapped in
stageSpanRoundTripper, which emits oneprovider_attemptstage per round trip, so retries each get a span. The span closes on response headers; a status >= 400 marks it errored without changing what the client sees.resolveModelCallresolves theStageModel(wire provider, configuredprovider_type, model, effective reasoning effort) and threads it to the transport, so span attributes and labels match the call.provider_typeis read fromai_providers.type, so types outside the fantasy set (copilot) keep their configured name.chatd.Newbuilds the tracer with the server clock.Behavior change to
coderd_chatd_ttft_seconds: only an output part counts as a first token. Error parts no longer close the window as a success, and warnings and finish parts are skipped; the help text says so.StageSpan.Endreturns the span's elapsed window, whichttft_secondsobserves. Callers passStagesandStageModelthroughGenerateAssistantOptions; a nil tracer discards (TTFT still recordsttft_seconds).Servergains thestagestracer and aTracerProvideroption, wired fromcoderd.Review decisions
streamandprovider_attemptwindows are observed: that time was spent. TTFT is the exception because a first token that never arrived has no duration.http_hostwas dropped fromprovider_attempt; every request goes through the local AI Gateway transport, so it was constant.stream/TTFT.Supersedes #28970 and #28971.
🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.