Repository navigation
feat: add chatd lifecycle stage tracer - #29769
Conversation
Docs previewCheck off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here. |
b86e409 to
76818c9
Compare
Documentation CheckUpdates Needed
Automated review via Coder Agents |
|
/coder-agents-review |
|
Chat: Review posted | View chat Review historydeep-review v0.13.0 | Round 4 | Last posted: Round 4, 60 findings (1 P2, 22 P3, 20 P4, 15 Nit, 2 Note), COMMENT. Review Finding inventoryFinding inventory, PR #29769Law analysisRound 1 (current)
Findings
Contested and acknowledgedCRF-1 (P2, coderd/x/chatd/chatloop/stage.go:87) - StageTracer has no production caller but the experiment and doc rows ship
CRF-2 (P3, coderd/x/chatd/chatloop/metrics.go:37) - stale_anchor advertised but not emitted
CRF-4 (P3, coderd/x/chatd/chatloop/stage.go:441) - redundant nil-field guards
CRF-11 (Note, coderd/x/chatd/generation.go:453) - runGenerationStep extraction has no consumer here
CRF-17 (P3, codersdk/deployment.go:5311) - series-cost gate is a safe experiment, not a Prometheus option
CRF-26 (P3, coderd/x/chatd/chatloop/stage.go:51) - HTTP attribute keys differ from semconv
CRF-30 (Nit, coderd/x/chatd/chatloop/metrics.go:74) - bucket comment omits the round-edge reason
CRF-33 (Nit, coderd/x/chatd/chatloop/stage.go:189) - context key comments restate names
CRF-38 (Nit, coderd/x/chatd/chatloop/stage.go:157) - StageModel.Effort naming
CRF-41 (Nit, coderd/x/chatd/chatloop/stage_test.go:587) - "ladder" wording and hardcoded count
CRF-44 (P4, coderd/x/chatd/chatloop/stage.go:343) - s.ended guards in setters
CRF-52 (P4, coderd/x/chatd/chatloop/metrics_internal_test.go:13) - knownStages duplicates the TestStageMembership list
CRF-17 (P3, codersdk/deployment.go:5311) - round 4 defense and closure
CRF-58 (P4, coderd/x/chatd/chatloop/metrics.go:181) - long help strings re-pad the prometheus.md table
Round logRound 1Netero-only (pre-panel gate on CRF-1 P2). Law ran (effective additions 1446 > 1000): Split, Advisory. 1 P2, 6 P3, 2 Nit, 2 Note. Orchestrator verified CRF-1 (grep: no caller of NewStageTracer, NopStageTracer, StartRoot, StartRootAt, RecordAs outside stage.go and stage_test.go), CRF-2 (only reference is the definition), CRF-8 (no _test.go references ExperimentChatStageMetrics), CRF-3 (stage_test.go is package chatloop_test, no export_test.go), CRF-9, CRF-11, CRF-12 against the code at head. Reviewed against 22a6d34..76818c9. Round 2Tool v0.12.0 (was v0.9.0); skill re-read. Churn guard: PROCEED (9 addressed, 3 contested; CRF-2 treated as contested because the author kept the constant). Dispute check-in: Netero's round 1 agent resumed with the round 2 prompt; it accepted CRF-1, CRF-2, CRF-4, CRF-11 (each 1/1). CRF-1 closure depends on the stack merging together, which nothing in the repository enforces. Netero also verified the author fixes (CRF-8 by mutation) and raised no new findings (1 Note, 1 OOS). Netero gate (now P0 only): no P0, panel proceeds. Law not re-run (effective additions +101 since round 1). Round type: full panel. First panel review of this PR, and a restructure since round 1 (new file Cross-check: 30 new in-scope findings (15 P3, 4 P4, 11 Nit), 1 Note (body), 2 new OOS, 3 dropped. Convergence: span-only anomaly counting (5 reviewers, incomplete fix of CRF-7 class; CRF-7 itself stays fixed for negative_elapsed); experiment as the cost gate (4); TTFT duplicate (3; Pariston verified at pr/29772 that Round 3Tool v0.13.0 (was v0.12.0); skill re-read (change: reviewer prompts point at nested AGENTS.md). Churn guard: PROCEED (24 addressed, 6 contested; CRF-41 treated as contested for its wording part). Netero (new agent, no contested Netero findings): 3 P4 test findings; verified CRF-15, CRF-16, CRF-18, CRF-19, CRF-20 by mutation. Law not re-run (effective additions 1921, +475 since Law's 1446). Round type: restructure (new file Dispute results: CRF-26, 30, 33, 38, 41, 44 closed 1/1. CRF-17 re-raised by Luffy (graduation precedent #26699/#27472 verified by orchestrator: #27472 deleted the gate and emptied ExperimentsSafe); Pen Botter, Ryosuke, Pariston accepted. Cross-check: 12 new in-scope P4 findings posted inline, 1 P4 in the review body (PR description, no file line), 3 fix-confirmation Notes dropped, 1 OOS. Convergence: help-text length re-pads the docs table (Leorio, Zoro; orchestrator verified 490 to 553 chars and +246 -243 in prometheus.md). Contradiction resolved: Melody cited Posted: #29769 (review) (12 inline, 1 reply). The CRF-17 reply posted, but Round 4Tool v0.13.0 (unchanged); skill re-read after context compaction. Churn guard: PROCEED (11 addressed, 1 acknowledged, 2 contested; CRF-63 fixed by a PR description edit with no reply). Netero: the round 3 agent resumed because CRF-52 was contested; no new findings; verified CRF-51 by mutation in both directions and CRF-53 by coverage. Law not re-run (effective additions 1928, +482 since Law's 1446). Round type: fix round. The only commit since round 3 (b7d8353) addresses prior findings: it removes Dispute results: CRF-17 closed 4/4 (lifecycle now recorded in the PR description). CRF-52 closed 1/1 (Netero). CRF-58 closed 2/2 (orchestrator verified the longest help string at each stack head: 553, 633, 823, 988). Cross-check: 4 new P4 and 2 new Nits posted; 1 Nit dropped (CRF-70: Gon P2 on two test section comments; keep-argument was AGENTS.md's substantive-comment rule; dropped because each comment accurately summarizes a multi-line block); 4 Notes dropped as fix confirmations or pattern observations; 1 OOS. Severity: Gon rated CRF-69 and CRF-70 P2; the reviewer prompt classes comment wording as Nit and both comments are accurate, as in round 2. Fix-chain findings: CRF-68 (the CRF-54 wording this review proposed misses About deep-reviewCRF = Coder Review Finding (P0-P4, Nit, Note)
|
There was a problem hiding this comment.
The core idea holds up: one call produces both the span and the histogram sample, so trace and metric durations cannot drift apart, and the typed Stage, Scope, and ChatKind constants keep string conversions at the label boundary. The series-cost table in the description is the cardinality accounting a new metric family should come with. Test density is about 50%, and the stage tests assert on real registry and span-recorder output rather than mock returns.
1 P2, 6 P3, 2 Nit, 2 Note.
This is a first-pass review only. These are mechanical findings from Netero; the full review panel has not reviewed this PR yet and will review after these findings are addressed. The P2 (CRF-1) is what holds the panel back: this PR ships a *-enabled experiment and three documented metric families with no production caller, so they never receive a sample unless the stacked PRs also land. CRF-2 (stale_anchor) has the same shape.
Split analysis (advisory, not blocking): at +1446 effective lines the PR was checked for decomposition. The recommendation is to keep the tracer, metric families, and experiment together, and to move the runGenerationStep extraction in generation.go into the stacked PR that adds the deferred stage end (or into a standalone refactor PR ahead of it), so its reason can be checked next to the defer it exists for (see CRF-11). That grouping argues against separating the experiment from the families, which favors CRF-1's first fix (bring the first caller into this PR) over its second (hold the experiment and doc rows back).
From Netero on CRF-5: "an alert on that series sends someone to look at NTP when the real problem is a missing timestamp."
🤖 This review was automatically generated with Coder Agents.
76818c9 to
5e32cbb
Compare
|
/coder-agents-review |
There was a problem hiding this comment.
This PR adds the chatd stage tracer, three experiment-gated Prometheus families, and the chat-stage-metrics experiment. This round: 15 P3, 4 P4, 11 Nit.
The round 1 fixes are verified: the injectable clock, the missing_timestamp and future_start reasons, the negative_elapsed ordering, the experiment wiring test (it fails when chatd.go:2981 is forced to true), the gather helpers, and the shared pool label values. The four contested threads (no production callers, stale_anchor, nil-field guards, runGenerationStep extraction) are closed on your replies. Three of those closures assume the stack merges together; nothing in the repository prevents this PR from merging alone, so that needs a human decision.
The reply on the stale_anchor thread says the help text now says the emitter comes with #29771, but no file under coderd/x/chatd mentions #29771 and the help text clause is unchanged; correct the reply or the text.
Out of scope (needs a ticket or explicit acceptance by a human):
coderd/coderd.go:2566:ReadExperimentslogs "ignoring unknown experiment" and then appends the experiment anyway, so the log says the opposite of what the server does.coderd/x/chatd/chatd.go:278:ptr.Refcalls predate this PR; Go 1.26go fixrewrites them tonew(expr), whileptr.Refis still the repository pattern.- CI
Storybookjob 107348848639 failed at this head. The onlysite/change is the generatedtypesGenerated.ts; whether this PR caused the failure is unverified.
Notes:
chat-stage-metricsis the firstExperimentsSafeentry, so deployments running--experiments=*get the families on upgrade, and every deployment lists the experiment on the Deployment overview page and exportscoderd_experiments{experiment="chat-stage-metrics"}.stage.go:141: thetime.Now()fallback inNow()is reachable only throughWithClock(nil).
🤖 This review was automatically generated with Coder Agents.
|
/coder-agents-review |
There was a problem hiding this comment.
This PR adds the chatd stage tracer, its experiment-gated Prometheus families, and the chat-stage-metrics experiment. This round adds 13 P4 findings, 12 inline and one below, and one P3 from the previous review stays open.
The fixes from the previous review are verified, including span-only anomaly filtering, span timestamps from the tracer clock, turn-only model observations, and NewMetrics leaving the stage families off. Six contested threads are closed on your replies. The thread on the experiment as the only switch for the series cost stays open and needs a human decision.
The API bullet in the PR description says End returns the elapsed time, but End(err error) returns nothing; the description becomes the squash commit message, so correct the bullet.
Out of scope (needs a ticket or explicit acceptance by a human):
coderd/x/chatd/generation.go:630:recordGenerationRetrypicks the larger delay with anif, whichgo fixwith the Go 1.26 modernizers rewrites to themaxbuiltin.
Notes:
- Correction to the previous review:
chat-stage-metricsis the only currentExperimentsSafeentry, not the first one;minimum-implicit-memberwas safe from #26699 until #27472.
🤖 This review was automatically generated with Coder Agents.
|
/coder-agents-review |
There was a problem hiding this comment.
This PR adds the chatd stage tracer, its experiment-gated Prometheus families, and the chat-stage-metrics experiment. This round: 4 P4, 2 Nit.
The fixes from the previous review are verified, including the two-way help text check, the removal of RecordAs, NopStageTracer, and AttrHTTPHost, and the queue_wait test that now takes scope and chat kind from the context. The three contested threads are closed on your replies: the experiment gate on the graduation lifecycle now in the description, the two stage lists, and the help string length.
Out of scope (needs a ticket or explicit acceptance by a human):
coderd/x/chatd/chatloop/metrics.goin #29770 to #29772: each later stack PR sets a new longest help string (633, 823, then 988 characters forcoderd_chatd_turn_time_seconds_totalat a62c34a), so each one re-pads about 245 rows ofdocs/admin/integrations/prometheus.md.
🤖 This review was automatically generated with Coder Agents.
ba38581 to
539484a
Compare
Add a StageTracer that emits one span and one stage_duration_seconds observation per chat lifecycle stage, with typed Stage, Scope, and ChatKind values, a model_stage_duration_seconds histogram for the provider-bound stages labeled by provider_type and model, and a stage_anomalies_total counter for dropped observations. The metric families are registered when the chat-stage-metrics experiment is enabled; spans are emitted regardless. Extract the generation step body into its own function so the step has a boundary for a deferred stage end.
Count stage anomalies only for observed stages, take live span timestamps from the tracer clock, leave the stage families off in NewMetrics, type anomaly reasons as StageAnomaly, and observe model_stage_duration_seconds only for turn-scoped stages. Cross-reference ttft_seconds and the time_to_first_token stage in their help text, and tighten the tracer doc comments.
- Carry the wire provider on StageModel so SetModel and Record stamp it. - Remove StartRoot and the unused links parameter of StartRootAt. - Drop time_to_first_token from the model stage family; per-model TTFT stays on coderd_chatd_ttft_seconds. - Remove the capacity_wait stage. - Define the turn scope as latency attributable to a prompt. - Add a consistency test over the stage sets and help text.
… stage tracer docs
…age spans Generated by Coder Agents.
539484a to
0be4ce3
Compare
TL;DR: Adds a stage tracer that turns each chat lifecycle stage into both an OpenTelemetry span and a Prometheus histogram observation, so traces and metrics always agree. The metrics are only registered with the
chat-stage-metricsexperiment. Nothing calls the tracer until the next PRs.Adds
chatloop.StageTracer, which emits each chat lifecycle stage as an OpenTelemetry span and acoderd_chatd_stage_duration_seconds{stage, scope, chat_kind}observation from oneEndcall, so trace and metric durations cannot disagree. The provider-bound stages (stream,provider_attempt) are also observed oncoderd_chatd_model_stage_duration_seconds{stage, provider_type, chat_kind, model}, the only family carryingprovider_typeandmodel, and only for turn-scoped stages. Time to first token per model stays on the existingcoderd_chatd_ttft_seconds. Nothing calls the tracer yet; the next PRs instrument the provider call and the runner.chat-stage-metricsexperiment is enabled (safe and known, so--experiments=chat-stage-metricsor--experiments=*turns it on). Spans are emitted regardless.NewMetricsleaves the families off;chatd.Newopts in from the experiment, and a test asserts the experiment decides whether they reach the server's registry.Stage,Scope,ChatKind, andStageAnomalyare typed string constants.scopeseparates latency attributable to a prompt (turn) from detachedbackgroundwork;chat_kindisrootorsubagent, empty without a known chat.StageModelcarries the wireprovider,provider_type,model, andreasoning_effort. All are span attributes; onlyprovider_typeandmodelare metric labels, on the model family. No family carries an organization label.chat_turn,queue_wait,acquisition,mcp_connect,stream,time_to_first_token,provider_attempt,tool_call,commit,retry_backoff.generation_step,prepare,thinking, andcompactionare span-only. A test keeps the stage sets and help text consistent.Start/StartRootAtfor live spans (span timestamps come from the tracer clock),Recordfor stages reconstructed from timestamps (the scope comes from the context).Endis idempotent;EndWithoutObservationends the span without a sample. A nil tracer, provider, or metrics discards.coderd_chatd_stage_anomalies_total{reason}counts observations dropped, adjusted, or inconsistent (negative_elapsed,inverted_window,missing_timestamp,future_start,stale_anchor). Span-only stages never feed it.0.1 0.25 0.5 1 2.5 5 10 30 60 300 1800 3600).forloop inStartGenerationintorunGenerationStepso the step has a function boundary for a deferred stage end. Control flow and error wrapping are unchanged.Series cost (approximate, whole stack)
A histogram label set is 12 edges +
+Inf+_sum+_count: 15 series. Per replica, eachchat_kindcosts about 150 stage series and 4 turn outcome counters; each(chat_kind, model)costs 30 (2 stages x 15). Both chat kinds onMmodels: about308 + 60*Mper replica, plus background-scope stage series.Dogfood
main, 3 replicas: ~3.4kcoderd_*series per replica (~10.1k total).Most deployments run four or fewer models, so a typical replica adds roughly 370 to 550 series. The experimental turn accounting in #29772 would add 88 more per replica (11 categories x 4 outcomes x 2 chat kinds). Series appear lazily and do not grow with organizations.
Review decisions
effortis span-only;provider_type(configured type), not the wireprovider, is the model-family label.time_to_first_tokenis not on the model family; per-model TTFT isttft_seconds, which observes the same measured window.Supersedes #28968 and #28969.
🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.