Skip to content

feat: add chatd lifecycle stage tracer - #29769

Merged
jscottmiller merged 8 commits into
mainfrom
scott/chatd-lifecycle/1-stage-tracer
Sep 30, 2026
Merged

jscottmiller merged 8 commits into
mainfrom
scott/chatd-lifecycle/1-stage-tracer

Conversation

@jscottmiller

@jscottmiller jscottmiller commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

TL;DR: Adds a stage tracer that turns each chat lifecycle stage into both an OpenTelemetry span and a Prometheus histogram observation, so traces and metrics always agree. The metrics are only registered with the chat-stage-metrics experiment. Nothing calls the tracer until the next PRs.

Adds chatloop.StageTracer, which emits each chat lifecycle stage as an OpenTelemetry span and a coderd_chatd_stage_duration_seconds{stage, scope, chat_kind} observation from one End call, so trace and metric durations cannot disagree. The provider-bound stages (stream, provider_attempt) are also observed on coderd_chatd_model_stage_duration_seconds{stage, provider_type, chat_kind, model}, the only family carrying provider_type and model, and only for turn-scoped stages. Time to first token per model stays on the existing coderd_chatd_ttft_seconds. Nothing calls the tracer yet; the next PRs instrument the provider call and the runner.

  • The families are registered only when the chat-stage-metrics experiment is enabled (safe and known, so --experiments=chat-stage-metrics or --experiments=* turns it on). Spans are emitted regardless. NewMetrics leaves the families off; chatd.New opts in from the experiment, and a test asserts the experiment decides whether they reach the server's registry.
  • Stage, Scope, ChatKind, and StageAnomaly are typed string constants. scope separates latency attributable to a prompt (turn) from detached background work; chat_kind is root or subagent, empty without a known chat.
  • StageModel carries the wire provider, provider_type, model, and reasoning_effort. All are span attributes; only provider_type and model are metric labels, on the model family. No family carries an organization label.
  • Observed stages: chat_turn, queue_wait, acquisition, mcp_connect, stream, time_to_first_token, provider_attempt, tool_call, commit, retry_backoff. generation_step, prepare, thinking, and compaction are span-only. A test keeps the stage sets and help text consistent.
  • API: Start / StartRootAt for live spans (span timestamps come from the tracer clock), Record for stages reconstructed from timestamps (the scope comes from the context). End is idempotent; EndWithoutObservation ends the span without a sample. A nil tracer, provider, or metrics discards.
  • coderd_chatd_stage_anomalies_total{reason} counts observations dropped, adjusted, or inconsistent (negative_elapsed, inverted_window, missing_timestamp, future_start, stale_anchor). Span-only stages never feed it.
  • Buckets are 12 round edges (0.1 0.25 0.5 1 2.5 5 10 30 60 300 1800 3600).
  • Moves the body of the for loop in StartGeneration into runGenerationStep so the step has a function boundary for a deferred stage end. Control flow and error wrapping are unchanged.
Series cost (approximate, whole stack)

A histogram label set is 12 edges + +Inf + _sum + _count: 15 series. Per replica, each chat_kind costs about 150 stage series and 4 turn outcome counters; each (chat_kind, model) costs 30 (2 stages x 15). Both chat kinds on M models: about 308 + 60*M per replica, plus background-scope stage series.

Dogfood main, 3 replicas: ~3.4k coderd_* series per replica (~10.1k total).

Models used per replica Added series (3 replicas) vs. baseline
1 ~1.1k +11%
5 ~1.8k +18%
10 ~2.7k +27%
35 ~7.2k +72%

Most deployments run four or fewer models, so a typical replica adds roughly 370 to 550 series. The experimental turn accounting in #29772 would add 88 more per replica (11 categories x 4 outcomes x 2 chat kinds). Series appear lazily and do not grow with organizations.

Review decisions
  • Experiment gate only; no deployment flag or metric levels. If the experiment graduates, the families become always on like the other chatd families; a flag is added then only if dogfood shows a cardinality problem.
  • Organization is on neither metrics nor spans, and effort is span-only; provider_type (configured type), not the wire provider, is the model-family label.
  • time_to_first_token is not on the model family; per-model TTFT is ttft_seconds, which observes the same measured window.
  • The model family records only turn-scoped stages.
  • Nil guards are kept so a nil or zero tracer is always safe to call.
  • HTTP attribute keys stay snake_case like the rest of the chatd span attributes.

Supersedes #28968 and #28969.

🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Docs preview

Check off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here.

@coderagents

coderagents Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Documentation Check

Updates Needed

  • docs/admin/integrations/prometheus.md (open) - Restore the "Monitor sessions by application" section. It exists on the base branch (and current main) and documents how to query coderd_agentstats_session_count joined with coderd_agentstats_app_info, including the multi-deployment variant, series-count planning, and the migration off the deprecated session gauges. The section was removed in this PR's commit c38acdc3b4 (a regeneration that rewrote the whole page from a stale base), while both metrics are still listed in the page's generated table.
  • docs/ai-coder/agents/platform-controls/chat-stage-metrics.md - New page (does not exist yet) for the chat-stage-metrics experiment: how to enable it (coder server --experiments=chat-stage-metrics or CODER_EXPERIMENTS), what it does (registers the coderd_chatd_stage_duration_seconds, coderd_chatd_model_stage_duration_seconds, and coderd_chatd_stage_anomalies_total metric families; spans are emitted regardless), and known limitations (opt-in only; metric cardinality grows with the number of models, and the provider_type label is the configured AI provider type, not the wire protocol provider label other chatd metrics use). Safe experiments need at least this single page, and the sibling convention (Advisor, Virtual desktop) plus docs/ai-coder/agents/platform-controls/index.md should enumerate it.

    ⚠️ Checked but no corresponding documentation changes found in this PR


Automated review via Coder Agents

@jscottmiller

Copy link
Copy Markdown
Contributor Author

/coder-agents-review

@coder-agents-review

coder-agents-review Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Chat: Review posted | View chat
Requested: 2026-09-25 17:29 UTC by @jscottmiller

Review history
  • R2 (2026-09-24): 12 reviewers, 13 Nit, 2 Note, 1 P2, 22 P3, 4 P4, COMMENT. Review
  • R3 (2026-09-25): 12 reviewers, 13 Nit, 2 Note, 1 P2, 22 P3, 16 P4, COMMENT. Review
  • R4 (2026-09-25): 11 reviewers, 15 Nit, 2 Note, 1 P2, 22 P3, 20 P4, COMMENT. Review

deep-review v0.13.0 | Round 4 | 22a6d34..b7d8353

Last posted: Round 4, 60 findings (1 P2, 22 P3, 20 P4, 15 Nit, 2 Note), COMMENT. Review

Finding inventory

Finding inventory, PR #29769

Law analysis

Round 1 (current)

  • Effective LOC: +1446 -113 across 9 files (719 production, 727 test, 50.3% test density).
  • Head SHA analyzed: 76818c9.
  • Verdict: Split, lightly. Keep the stage tracer, stage metric families, and chat-stage-metrics experiment together. Move the runGenerationStep extraction in coderd/x/chatd/generation.go out, preferably into the chat loop instrumentation PR that adds the deferred stage end, or into a standalone refactor PR ahead of it.
  • Enforcement: Advisory.
  • Basis: four concerns. Tracer (1) and metric families (2) are one API (the families key on Stage); the experiment (3) gates (2). The extraction (4) is independent of 1 to 3 and its only consumer is a later PR in the stack. Law read both sides of the extraction and found no behavior change (whitespace-insensitive diff about +38 -19); it did not run the chatd suite against it. Horizontal cuts of 1 to 3 were rejected because each leaves a flag gating nothing or a transient registration state.

Findings

# Sev Status Location Summary Round Reviewer Posted
CRF-1 P2 Author contested; panel closed R2 (1/1 accept; conditional on the stack merging together) coderd/x/chatd/chatloop/stage.go:87 StageTracer has zero production callers, yet the PR ships a safe experiment and doc rows for three metric families that never receive a sample R1 Netero Yes
CRF-2 P3 Author contested; panel closed R2 (1/1 accept; comment aligned in 5e32cbb, reply misstates help text change) coderd/x/chatd/chatloop/metrics.go:37 StageAnomalyStaleAnchor is advertised in help text and docs but nothing emits it R1 Netero Yes
CRF-3 P3 Author fixed (5e32cbb) coderd/x/chatd/chatloop/stage.go:90 quartz clock field cannot be injected, so tests use wall-clock time and the time.Now fallback is unreachable R1 Netero Yes
CRF-4 P3 Author contested; panel closed R2 (1/1 accept) coderd/x/chatd/chatloop/stage.go:441 Nil-field guards repeat invariants the constructor already enforces, hiding which layer owns them R1 Netero Yes
CRF-5 P3 Author fixed (5e32cbb) coderd/x/chatd/chatloop/stage.go:411 Missing timestamps are counted as inverted_window (help text says clock inconsistency); zero-width windows are observed despite the doc saying non-positive windows are dropped R1 Netero Yes
CRF-6 P3 Author fixed (5e32cbb) coderd/x/chatd/chatloop/stage.go:288 startSpan clamps a future start to now without counting an anomaly; branch untested R1 Netero Yes
CRF-7 P3 Author fixed (5e32cbb) coderd/x/chatd/chatloop/metrics.go:246 negative_elapsed is counted before the observedStages check, so span-only stages feed the dropped-observation counter R1 Netero Yes
CRF-8 P3 Author fixed (5e32cbb) coderd/x/chatd/chatd.go:2980 No test covers the experiment-to-MetricsOptions wiring in chatd.New R1 Netero Yes
CRF-9 Nit Author fixed (5e32cbb) coderd/x/chatd/chatloop/metrics.go:227 m := &Metrics{...}; return m adds an unused variable and widens the diff R1 Netero Yes
CRF-10 Nit Author fixed (5e32cbb) coderd/x/chatd/chatloop/stage_test.go:65 Four test helpers repeat the Gather-and-find loop that gatherFamily already implements R1 Netero Yes
CRF-11 Note Author contested; panel closed R2 (1/1 accept) coderd/x/chatd/generation.go:453 runGenerationStep extraction preserves control flow but has no consumer in this PR; hooks block shadows named return err R1 Netero Yes
CRF-12 Note Author fixed (5e32cbb) coderd/x/chatd/chatloop/stage.go:75 ChatKindRoot/ChatKindSubagent duplicate the raw pool label strings in capacity.go:80-83 with no shared constant R1 Netero Yes
CRF-13 Note Note (body) coderd/x/chatd/chatloop/stage.go:141 Now() time.Now fallback is reachable only through WithClock(nil); no caller or test does that R2 Netero No
CRF-14 OOS Out of scope (body) CI Storybook (107348848639) Storybook job failed at 5e32cbb; only site/ change is generated typesGenerated.ts; cause unverified R2 Netero No
CRF-15 P3 Author fixed (cbb8533) coderd/x/chatd/chatloop/stage.go:432 RecordAs and startSpan count missing_timestamp, inverted_window, and future_start for span-only stages; the earlier negative_elapsed fix covered one of four reasons R2 Melody, Ryosuke, Mafuuu, Bisky, Leorio Yes
CRF-16 P3 Author fixed (ced30e1) coderd/x/chatd/chatloop/metrics.go:69 time_to_first_token on model_stage_duration_seconds duplicates coderd_chatd_ttft_seconds with different labels and buckets; the stack observes both from one event R2 Pariston, Luffy, Pen Botter Yes
CRF-17 P3 Author contested; panel closed R4 (4/4 accept) codersdk/deployment.go:5311 Series-cost gate is an experiment in ExperimentsSafe instead of a permanent Prometheus deployment option like prometheus-collect-db-metrics R2 Luffy, Pen Botter, Ryosuke, Pariston Yes
CRF-18 P3 Author fixed (cbb8533) coderd/x/chatd/chatloop/stage.go:400 Live spans take timestamps from the OTel SDK clock while the histogram uses the tracer clock; with a mock clock span and histogram durations diverge and tests do not check it R2 Mafuuu P3, Bisky P3, Pariston P4 Yes
CRF-19 P3 Author fixed (cbb8533) coderd/x/chatd/chatloop/metrics.go:182 model_stage_duration_seconds drops the scope label, so background and turn observations of a stage share one series R2 Melody Yes
CRF-20 P3 Author fixed (cbb8533) coderd/x/chatd/chatloop/metrics.go:110 NewMetrics registers the experiment-gated families unconditionally; only NewMetricsWithOptions honors the gate R2 Knov P3, Ryosuke P4 Yes
CRF-21 P3 Author fixed (cbb8533) coderd/x/chatd/chatloop/stage.go:176 StageSpan doc says End must be called exactly once, contradicting idempotent End and omitting EndWithoutObservation R2 Gon P3, Leorio P4, Ryosuke Nit, Mafuuu Nit, Knov Nit Yes
CRF-22 P3 Author fixed (cbb8533) coderd/x/chatd/chatloop/stage.go:15 Stage doc says every value is a stage label value; four span-only values never are R2 Gon P3, Leorio P4 Yes
CRF-23 P3 Author fixed (cbb8533) coderd/x/chatd/chatloop/stage.go:82 StageTracer doc promises one observation per stage; false for span-only stages and EndWithoutObservation R2 Gon P3, Leorio P4 Yes
CRF-24 P3 Author fixed (cbb8533) coderd/x/chatd/chatloop/metrics.go:268 RecordStageAnomaly doc (and commit body) say dropped; future_start and stale_anchor are adjustments R2 Gon P3, Leorio P4 Yes
CRF-25 P3 Author fixed (cbb8533) coderd/x/chatd/chatloop/metrics.go:78 MetricsOptions doc names NewMetrics, which takes no options R2 Gon P3, Leorio P4 Yes
CRF-26 P3 Author contested; panel closed R3 (1/1 accept) coderd/x/chatd/chatloop/stage.go:51 AttrHTTPStatusCode/Method/Host use snake_case keys that differ from the semconv keys coderd HTTP spans record R2 Gon Yes
CRF-27 P3 Author fixed (cbb8533) coderd/x/chatd/chatloop/stage_test.go:598 Test comment says a 45 minute turn; the recorded stage is StageStream R2 Gon Yes
CRF-28 P3 Author fixed (cbb8533) coderd/x/chatd/chatloop/stage_test.go:668 Test comment ties the inverted_window count to commit; it comes from a direct RecordStageAnomaly call R2 Gon Yes
CRF-29 P3 Author fixed (cbb8533) coderd/x/chatd/chatloop/metrics.go:46 observedStages comment category list omits chat_turn and restates the map R2 Gon P2 Yes
CRF-30 Nit Author contested; panel closed R3 (1/1 accept) coderd/x/chatd/chatloop/metrics.go:74 stageDurationBuckets comment restates the values; the round-edge reason lives only in a test comment R2 Gon P2 Yes
CRF-31 Nit Author fixed (cbb8533) coderd/x/chatd/chatloop/metrics.go:241 RecordStageDuration doc repeats the ChatKind doc R2 Gon P2 Yes
CRF-32 Nit Author fixed (cbb8533) coderd/x/chatd/chatloop/stage.go:103 NewStageTracer doc restates the effect of each fallback R2 Gon P2 Yes
CRF-33 Nit Author contested; panel closed R3 (1/1 accept) coderd/x/chatd/chatloop/stage.go:189 Context key comments at :189, :192, :195 restate the identifiers R2 Gon P2 Yes
CRF-34 Nit Author fixed (cbb8533) coderd/x/chatd/chatloop/stage.go:326 stageIdentityAttributes comment ends with a preference clause that adds no information R2 Gon P2 Yes
CRF-35 Nit Author fixed (cbb8533) coderd/x/chatd/chatloop/stage.go:378 EndWithoutObservation doc takes five lines for one sentence of content R2 Gon P2 Yes
CRF-36 Nit Author fixed (cbb8533) coderd/x/chatd/chatloop/stage.go:387 closeSpan doc repeats End's double-count reason R2 Gon P2 Yes
CRF-37 Nit Author fixed (cbb8533) coderd/x/chatd/chatloop/stage_test.go:621 Garbled test comment restates its first clause R2 Gon P2 Yes
CRF-38 Nit Author contested; panel closed R3 (1/1 accept) coderd/x/chatd/chatloop/stage.go:157 StageModel.Effort should be ReasoningEffort to match AttrReasoningEffort R2 Gon Yes
CRF-39 Nit Author fixed (cbb8533) coderd/x/chatd/chatloop/metrics.go:28 StageAnomaly* comments call each reason constant a stage R2 Gon Yes
CRF-40 P4 Author fixed (cbb8533) coderd/x/chatd/chatloop/stage.go:458 Anomaly reason is an untyped string on exported methods; any string creates a new series R2 Knov P4, Gon Nit Yes
CRF-41 Nit Author partially fixed (ced30e1); contested wording part; panel closed R3 (1/1 accept) coderd/x/chatd/chatloop/stage_test.go:587 Test calls the buckets a ladder and hardcodes 12 R2 Gon Yes
CRF-42 P4 Author fixed (cbb8533) coderd/x/chatd/chatloop/stage.go:273 StartRootAt doc omits the future-start clamp and anomaly R2 Leorio Yes
CRF-43 P4 Author fixed (cbb8533) coderd/x/chatd/chatloop/stage.go:448 No test covers Record with a non-nil error R2 Bisky Yes
CRF-44 P4 Author contested; panel closed R3 (1/1 accept) coderd/x/chatd/chatloop/stage.go:343 s.ended guards in SetAttributes and SetModel duplicate OTel behavior and have no coverage R2 Bisky Yes
CRF-45 Note Note (body) codersdk/deployment.go:5310 First ExperimentsSafe entry: * deployments get the families, every deployment lists it on the overview page and exports a coderd_experiments series R2 Melody, Pen Botter, Knov No
CRF-46 Note Dropped by orchestrator (praise; no information or action for the author) coderd/x/chatd/chatloop/stage.go:372 One call produces span and metric; cost table justifies the gate R2 Luffy No
CRF-47 Note Dropped by orchestrator (praise; no information or action for the author) coderd/x/chatd/stage_metrics_internal_test.go:21 TestServerStageMetricsFollowExperiment and TestExperimentsSafeAreKnown are real tests R2 Bisky No
CRF-48 Note Dropped by orchestrator (PR description wording; no code line and no reader consequence beyond length) PR description Two bullets repeat the diff R2 Leorio No
CRF-49 OOS Out of scope (body) coderd/x/chatd/chatd.go:278 ptr.Ref predates the PR; go fix rewrites to new(expr) R2 Ging-Go No
CRF-50 OOS Out of scope (body) coderd/coderd.go:2566 ReadExperiments logs ignoring unknown experiment, then appends it anyway R2 Leorio No
CRF-51 P4 Author fixed (b7d8353) coderd/x/chatd/chatloop/metrics_internal_test.go:68 TestStageSetsConsistent checks help text one way only and skips the model family help R3 Netero Yes
CRF-52 P4 Author contested; panel closed R4 (1/1 accept) coderd/x/chatd/chatloop/metrics_internal_test.go:13 knownStages repeats the TestStageMembership stage list; neither fails on a new Stage constant R3 Netero Yes
CRF-53 P4 Author fixed (b7d8353) coderd/x/chatd/chatloop/stage.go:334 StageSpan.SetAttributes has 0% coverage R3 Netero Yes
CRF-54 P4 Author fixed (b7d8353) coderd/x/chatd/chatloop/stage.go:85 StageTracer doc says span timestamps come from the tracer clock; false for Record and RecordAs R3 Gon Yes
CRF-55 P4 Author fixed (b7d8353) coderd/x/chatd/chatloop/stage.go:457 recordStageAnomaly shares a name with Metrics.RecordStageAnomaly but filters span-only stages R3 Gon Yes
CRF-56 P4 Author fixed (b7d8353) coderd/x/chatd/chatloop/stage_test.go:397 Test comment says RecordAs sets the scope on the context; it does not R3 Gon Yes
CRF-57 P4 Author fixed (b7d8353) coderd/x/chatd/chatloop/stage_test.go:545 UsesExplicitTimestamps records queue_wait with chat_kind only as a span attribute and background scope, so span and sample disagree R3 Melody Yes
CRF-58 P4 Author contested; panel closed R4 (2/2 accept) coderd/x/chatd/chatloop/metrics.go:181 New help strings exceed the longest existing one, so make gen re-pads the whole prometheus.md metrics table R3 Leorio, Zoro Yes
CRF-59 P4 Author fixed (b7d8353) coderd/x/chatd/chatloop/stage.go:413 RecordAs has no production caller in the stack; Record with ContextWithScope does the same; empty scope behaves differently R3 Zoro Yes
CRF-60 P4 Author fixed (b7d8353) coderd/x/chatd/chatloop/stage.go:127 NopStageTracer is unused anywhere in the stack; a nil tracer already discards R3 Zoro Yes
CRF-61 P4 Author fixed (b7d8353) coderd/x/chatd/chatloop/stage.go:52 AttrHTTPHost has no user in this PR; #29770 removes it (5bb414f) and the stack tip does not define it R3 Zoro Yes
CRF-62 P4 Author fixed (b7d8353) coderd/x/chatd/stage_metrics_internal_test.go:40 Gather-and-scan loop reimplements promtestutil.GatherAndCount R3 Zoro Yes
CRF-63 P4 Author fixed (PR description edited before R4) PR description API bullet says End returns the elapsed time; End returns nothing in this PR R3 Leorio No
CRF-64 Note Dropped by orchestrator (fix confirmation; no action) coderd/x/chatd/chatloop/metrics.go:70 Time to first token removed from modelStages R3 Luffy No
CRF-65 Note Dropped by orchestrator (fix confirmation; no action) coderd/x/chatd/chatloop/stage.go:457 Span-only anomaly and model-scope fixes verified R3 Melody No
CRF-66 Note Dropped by orchestrator (fix confirmation; no action) coderd/x/chatd/chatloop/stage.go:15 Round 2 doc fixes verified R3 Leorio No
CRF-67 OOS Out of scope (body) coderd/x/chatd/generation.go:630 recordGenerationRetry delay if-block that max() replaces; unchanged by this PR R3 Ging-Go No
CRF-68 P4 Open coderd/x/chatd/chatloop/stage.go:84 StageTracer doc (CRF-54 fix) says live spans take timestamps from the tracer clock; StartRootAt takes its start from the caller R4 Gon Yes
CRF-69 Nit Open coderd/x/chatd/chatloop/stage.go:146 StageModel.Effort doc introduces "Coder-scale"; chatprovider calls it the global scale R4 Gon P2 Yes
CRF-70 Nit Dropped by orchestrator (accurate summaries of multi-line test blocks; no reader consequence) coderd/x/chatd/chatloop/metrics_internal_test.go:67 Test section comments at :67 and :77 restate the assertions below them R4 Gon P2 No
CRF-71 Nit Open coderd/x/chatd/chatloop/metrics_internal_test.go:72 strings.Split plus range where strings.SplitSeq fits (GO.md, go fix) R4 Ging-Go Yes
CRF-72 P4 Open coderd/x/chatd/chatloop/stage_test.go:202 Post-End half of SetAttributesBeforeEndOnly (CRF-53 fix) cannot fail; the SDK already drops the write R4 Bisky Yes
CRF-73 P4 Open coderd/x/chatd/chatloop/metrics.go:248 Exported RecordStageDuration, RecordStageAnomaly, and stage family fields let callers write samples with no span and an unnormalized scope R4 Knov Yes
CRF-74 P4 Open coderd/x/chatd/chatloop/stage.go:267 StartRootAt accepts any Stage but always records turn scope R4 Knov Yes
CRF-75 OOS Out of scope (body) coderd/x/chatd/chatloop/metrics.go (#29772) Later stack PRs lengthen the longest help string to 988 characters; each re-pads ~245 prometheus.md rows R4 Zoro No
CRF-76 Note Dropped by orchestrator (pattern observation; no action) coderd/x/chatd/chatloop/stage.go Comment quality pattern repeats across rounds R4 Gon No
CRF-77 Note Dropped by orchestrator (fix confirmation; no action) coderd/x/chatd/chatloop/stage.go:395 Removed API leaves Record as the only timestamp path R4 Luffy No
CRF-78 Note Dropped by orchestrator (fix confirmation; no action) coderd/x/chatd/chatloop/stage.go:146 Changed doc comments verified against the stack tip; CRF-63 fixed R4 Leorio No
CRF-79 Note Dropped by orchestrator (fix confirmation; no action) coderd/x/chatd/chatloop/metrics.go:117 CRF-20 and CRF-40 fixes verified R4 Knov No

Contested and acknowledged

CRF-1 (P2, coderd/x/chatd/chatloop/stage.go:87) - StageTracer has no production caller but the experiment and doc rows ship

CRF-2 (P3, coderd/x/chatd/chatloop/metrics.go:37) - stale_anchor advertised but not emitted

  • Finding: StageAnomalyStaleAnchor appears in the help text and docs but nothing emits it; the constant comment and help text described the clamp differently. Proposed: remove the constant and help text clause, or add the emitter.
  • Author defense (R2): Partially addressed; the comment and help text now say the emitter lands with the turn span in feat: open a chat_turn span per prompt #29771; the constant stays so the registered help string is complete.
  • Panel closure (R2, 1/1): Netero, the agent that raised it. Accepted on the same stack condition as CRF-1. Verified the constant comment (metrics.go:40-42) now matches the help text. Also found the reply misstates the code: grep -rn '29771' coderd/x/chatd returns nothing and the stale_anchor help text clause (metrics.go:187) is unchanged.

CRF-4 (P3, coderd/x/chatd/chatloop/stage.go:441) - redundant nil-field guards

  • Finding: Nil-field guards in observe, RecordAnomaly, and otelTracer repeat invariants NewStageTracer enforces. Proposed: keep only t == nil guards, or remove the constructor substitution.
  • Author defense (R2): The guards keep a nil or zero-value tracer safe from tests and from paths that pass a nil tracer on purpose (compaction summary in feat: trace chatd provider stages and count only output parts as first token #29770).
  • Panel closure (R2, 1/1): Netero, the agent that raised it. Closed because the consequence is acceptable, not because the defense is accurate: the nil-tracer case is handled by the t == nil guards (TestStageTracerWithoutProvider), and the nil-field guards only run for a zero-value StageTracer{}, which no code constructs (grep -rn 'StageTracer{' coderd finds only the constructor; the branches show 0 coverage). Three unreachable branches with no runtime effect.

CRF-11 (Note, coderd/x/chatd/generation.go:453) - runGenerationStep extraction has no consumer here

  • Finding: The extraction preserves control flow, but the deferred stage end it exists for is not in this PR.
  • Author defense (R2): The extraction is used by feat: open a chat_turn span per prompt #29771.
  • Panel closure (R2, 1/1): Netero, the agent that raised it. Accepted under the same stack condition as CRF-1; control-flow equivalence was verified in round 1 by Netero and the Law split analysis.

CRF-17 (P3, codersdk/deployment.go:5311) - series-cost gate is a safe experiment, not a Prometheus option

  • Finding: The only switch for the added series is chat-stage-metrics in ExperimentsSafe; the cost remains after graduation, --experiments=* deployments get the series on upgrade, and costly families elsewhere use permanent Prometheus options (--prometheus-collect-db-metrics). Proposed: a Prometheus option passed to MetricsOptions.StageMetrics, or state the gate's purpose and keep it out of ExperimentsSafe.
  • Author defense (R3): Not changing; the settled design gates the families only by the safe experiment, with no separate flag or level (PR body "Review decisions").
  • Panel re-raised (R3, 3/4 accept, 1 re-raise): All four raising agents were the round 2 agents, resumed. Luffy re-raised with new evidence: the only other recent safe experiment (minimum-implicit-member, safe in feat: mark minimum-implicit-member experiment as safe #26699) was promoted in feat: promote MinimumImplicitMember experiment to GA #27472, which deleted the gate and left ExperimentsSafe empty, so graduation removes the switch; the defense gives no reason. Pen Botter accepted (label sets are bounded like ungated ttft_seconds/message_count; operators can drop series with metric_relabel_configs; * is explicit opt-in; the db-metrics precedent gates per-query series, a different cost class). Ryosuke accepted on the consequence (series only with explicit *, cost stated, graduation PR must decide). Pariston accepted (* is explicit opt-in, series bounded and lazy, graduation needs a code change that can decide with production data). Stays open because one raising persona re-raised; the graduation-time decision the acceptances rely on is a follow-up with no ticket.

CRF-26 (P3, coderd/x/chatd/chatloop/stage.go:51) - HTTP attribute keys differ from semconv

  • Finding: AttrHTTPStatusCode/AttrHTTPMethod/AttrHTTPHost use snake_case keys unlike coderd's inbound HTTP spans (http.status_code). Proposed: semconv keys.
  • Author defense (R3): Chatd stage attributes on custom spans keep one lowercase snake_case key set rather than mixing in semconv keys.
  • Panel closure (R3, 1/1): Gon, the round 2 agent, resumed. Accepted: grep -rln otelhttp --include=*.go finds no outbound HTTP spans in coderd outside scaletest/, so no existing outbound span uses semconv keys that chatd diverges from; a cross-span query needs two keys, which is acceptable.

CRF-30 (Nit, coderd/x/chatd/chatloop/metrics.go:74) - bucket comment omits the round-edge reason

  • Finding: The comment restates the bucket values and omits why they are round.
  • Author defense (R3): TestStageDurationBuckets asserts the constraint and fails if an alert edge is removed.
  • Panel closure (R3, 1/1): Gon, resumed. Accepted: a developer learns the constraint from the test failure, at the cost of one CI run.

CRF-33 (Nit, coderd/x/chatd/chatloop/stage.go:189) - context key comments restate names

  • Finding: Three unexported context key type comments restate the type names.
  • Author defense (R3): One-line doc comments on unexported context key types are idiomatic Go.
  • Panel closure (R3, 1/1): Gon, resumed. Accepted: four lines with no behavioral effect.

CRF-38 (Nit, coderd/x/chatd/chatloop/stage.go:157) - StageModel.Effort naming

  • Finding: Effort should be ReasoningEffort to match AttrReasoningEffort.
  • Author defense (R3): The name is used across the stack; the StageModel doc defines it.
  • Panel closure (R3, 1/1): Gon, resumed. Accepted: the doc defines the field, and a rename touches every stacked PR.

CRF-41 (Nit, coderd/x/chatd/chatloop/stage_test.go:587) - "ladder" wording and hardcoded count

  • Finding: The test called bucket edges a "ladder" and hardcoded 12.
  • Author defense (R3): Hardcoded count removed; "ladder" wording left as cosmetic.
  • Panel closure (R3, 1/1): Gon, resumed. Accepted: the count is gone; the wording difference is confined to one test.

CRF-44 (P4, coderd/x/chatd/chatloop/stage.go:343) - s.ended guards in setters

  • Finding: The s.ended checks in SetAttributes and SetModel duplicate SDK behavior and have no coverage.
  • Author defense (R3): The guards keep the post-End contract local to StageSpan.
  • Panel closure (R3, 1/1): Bisky, the round 2 agent, resumed. Accepted: two early returns with no runtime effect; the missing test of the contract is tracked separately (SetAttributes 0% coverage).

CRF-52 (P4, coderd/x/chatd/chatloop/metrics_internal_test.go:13) - knownStages duplicates the TestStageMembership list

  • Finding: knownStages in the internal test repeats the 14-stage list in TestStageMembership, and neither fails when a new Stage constant is added. Proposed: one list, by driving TestStageMembership from knownStages or deriving the subset checks from the membership table.
  • Author defense (R4): The lists live in different packages (chatloop internal vs chatloop_test), Go cannot enumerate a type's constants, and the membership table pins each stage's sets, so it is updated with a new constant.
  • Panel closure (R4, 1/1): Netero, the round 3 agent that raised it, resumed. Accepted: after b7d8353, TestStageSetsConsistent requires the help text's Observed: list to equal observedStages, so a new observed stage missing from the lists or the help fails; a new span-only stage missing from both lists produces no observation, so leaving it unchecked has no metric effect. The package split is real.

CRF-17 (P3, codersdk/deployment.go:5311) - round 4 defense and closure

  • Finding: As in the round 2 entry above; round 3 re-raise asked what happens to the switch at graduation.
  • Author defense (R4): On graduation the families become always on like the other chatd families (ttft_seconds and others are already unconditional); a flag is added at graduation only if dogfood shows a cardinality problem. The PR description "Review decisions" section records this.
  • Panel closure (R4, 4/4): All four raising agents were the round 2 agents, resumed. Luffy accepted: the graduation outcome is now a recorded decision, and the cost class matches the unconditional chatd families. Pen Botter accepted: labels are bounded; ttft_seconds, message_count, prompt_size_bytes already ship unconditionally with provider and model labels; operators can drop series with metric_relabel_configs; the flag condition is a revisit criterion, not a deferral. Ryosuke accepted: the decision the finding asked for is on the record; noted that dogfood runs few models, so the dogfood condition does not exercise the per-model term. Pariston accepted: losing the gate at graduation is intended, cost scales with model count only, and the graduation PR has a stated criterion.

CRF-58 (P4, coderd/x/chatd/chatloop/metrics.go:181) - long help strings re-pad the prometheus.md table

Round log

Round 1

Netero-only (pre-panel gate on CRF-1 P2). Law ran (effective additions 1446 > 1000): Split, Advisory. 1 P2, 6 P3, 2 Nit, 2 Note. Orchestrator verified CRF-1 (grep: no caller of NewStageTracer, NopStageTracer, StartRoot, StartRootAt, RecordAs outside stage.go and stage_test.go), CRF-2 (only reference is the definition), CRF-8 (no _test.go references ExperimentChatStageMetrics), CRF-3 (stage_test.go is package chatloop_test, no export_test.go), CRF-9, CRF-11, CRF-12 against the code at head. Reviewed against 22a6d34..76818c9.

Round 2

Tool v0.12.0 (was v0.9.0); skill re-read. Churn guard: PROCEED (9 addressed, 3 contested; CRF-2 treated as contested because the author kept the constant). Dispute check-in: Netero's round 1 agent resumed with the round 2 prompt; it accepted CRF-1, CRF-2, CRF-4, CRF-11 (each 1/1). CRF-1 closure depends on the stack merging together, which nothing in the repository enforces. Netero also verified the author fixes (CRF-8 by mutation) and raised no new findings (1 Note, 1 OOS). Netero gate (now P0 only): no P0, panel proceeds. Law not re-run (effective additions +101 since round 1).

Round type: full panel. First panel review of this PR, and a restructure since round 1 (new file stage_metrics_internal_test.go, NewStageTracer signature changed). Panel: ging-go, killua, luffy, melody, pen-botter, ryosuke (domain specialists whose triggers matched), pariston, mafuuu (core), bisky (tests changed), gon, leorio (once-per-PR floor), knov (random wildcard from the distilled set). Size target for a focused Go PR is 6-8; exceeded because six specialists plus the Gon/Leorio floor were required. ging-ts not selected: the only .ts change is generated typesGenerated.ts. Takumi, Kurapika, Knuckle, Nami not triggered (no goroutines/locks, no auth or input handling, no SQL, no authored frontend).

Cross-check: 30 new in-scope findings (15 P3, 4 P4, 11 Nit), 1 Note (body), 2 new OOS, 3 dropped. Convergence: span-only anomaly counting (5 reviewers, incomplete fix of CRF-7 class; CRF-7 itself stays fixed for negative_elapsed); experiment as the cost gate (4); TTFT duplicate (3; Pariston verified at pr/29772 that finishTTFT observes both families from one event); span timestamp clock divergence (3, reproduced). Severity decisions: CRF-18 Mafuuu/Bisky P3 over Pariston P4; CRF-20 Knov P3 over Ryosuke P4; CRF-21 Gon P3 over Leorio P4 and three Nits (Gon traced the consequence: callers avoid a deferred End after an explicit End); CRF-22 to CRF-25 Gon P3 over Leorio P4; CRF-40 Knov P4 over Gon Nit. Downgrades from Gon's P2 bloat ratings: keep-argument was that AGENTS.md requires substantive, concise comments; downgraded because the reviewer prompt classes comment wording as Nit and these comments are accurate. CRF-29 went to P3 instead of Nit because its category list is factually incomplete (same class as CRF-22). CRF-33 merges three identical restating comments at stage.go:189, :192, :195 into one finding. Dropped CRF-46, CRF-47 (praise) and CRF-48 (PR description wording). Orchestrator verified: prometheus-collect-db-metrics text at deployment.go:2600-2612, semconv.HTTPStatusCodeKey at tracing/httpmw.go:196, optionValue.ts safe-experiment listing, ReadExperiments append-after-warn at coderd.go:2562-2570, stale_anchor help text unchanged and no #29771 reference in coderd/. Reviewed against 22a6d34..5e32cbb.

Round 3

Tool v0.13.0 (was v0.12.0); skill re-read (change: reviewer prompts point at nested AGENTS.md). Churn guard: PROCEED (24 addressed, 6 contested; CRF-41 treated as contested for its wording part). Netero (new agent, no contested Netero findings): 3 P4 test findings; verified CRF-15, CRF-16, CRF-18, CRF-19, CRF-20 by mutation. Law not re-run (effective additions 1921, +475 since Law's 1446).

Round type: restructure (new file metrics_internal_test.go; exported API changed: StartRoot and the links parameter removed, StageCapacityWait removed, StageModel.Provider added, StageAnomaly typed). Full panel: ging-go, killua, luffy, melody, pen-botter, ryosuke (specialists), pariston, mafuuu (core), bisky (tests), gon, leorio (restructure floor), zoro (random wildcard). Dispute check-in reused round 2 agents for luffy, pen-botter, ryosuke, pariston (CRF-17), gon (CRF-26, 30, 33, 38, 41), bisky (CRF-44); all others were new agents.

Dispute results: CRF-26, 30, 33, 38, 41, 44 closed 1/1. CRF-17 re-raised by Luffy (graduation precedent #26699/#27472 verified by orchestrator: #27472 deleted the gate and emptied ExperimentsSafe); Pen Botter, Ryosuke, Pariston accepted.

Cross-check: 12 new in-scope P4 findings posted inline, 1 P4 in the review body (PR description, no file line), 3 fix-confirmation Notes dropped, 1 OOS. Convergence: help-text length re-pads the docs table (Leorio, Zoro; orchestrator verified 490 to 553 chars and +246 -243 in prometheus.md). Contradiction resolved: Melody cited pr/29772 chatd.go:4902 RecordAs(..., ScopeTurn, ...) as the production queue_wait caller, but that local ref (ca0e2f4) is stale; at the current stack tip 35203c4 there is no production RecordAs caller (Zoro's claim holds) and queue_wait is recorded with Record on a context carrying ScopeTurn and the chat kind (chatd.go:4895-4897). Melody's test finding stands on that corrected evidence; its fix is phrased with Record so it does not conflict with the finding that proposes deleting RecordAs. Correction: the round 2 review body called chat-stage-metrics the first ExperimentsSafe entry; minimum-implicit-member was safe from #26699 to #27472. Corrected in this round's body. Reviewed against 22a6d34..ced30e1.

Posted: #29769 (review) (12 inline, 1 reply). The CRF-17 reply posted, but thread_action: unresolve failed with GraphQL: Resource not accessible by integration (unresolveReviewThread), so thread PRRT_kwDOGkVX1s6lqO7y stays resolved. CRF-61 corrected before posting: AttrHTTPHost was used in #29770 until 5bb414f removed the use and the constant.

Round 4

Tool v0.13.0 (unchanged); skill re-read after context compaction. Churn guard: PROCEED (11 addressed, 1 acknowledged, 2 contested; CRF-63 fixed by a PR description edit with no reply). Netero: the round 3 agent resumed because CRF-52 was contested; no new findings; verified CRF-51 by mutation in both directions and CRF-53 by coverage. Law not re-run (effective additions 1928, +482 since Law's 1446).

Round type: fix round. The only commit since round 3 (b7d8353) addresses prior findings: it removes RecordAs, NopStageTracer, AttrHTTPHost (CRF-59 to CRF-61), renames recordStageAnomaly (CRF-55), and changes tests (CRF-51, 53, 56, 57, 62). It also rewrites the StageModel.Effort doc comment, which no finding requested. Panel: luffy, pen-botter, ryosuke, pariston (CRF-17 raisers), leorio, zoro (CRF-58 raisers), gon (CRF-54 to 56 code touched; comment lines changed), melody (CRF-57 code touched), ging-go (Go files changed), bisky (tests changed), knov (random wildcard from the distilled set excluding zoro). Reused agents: luffy, pen-botter, ryosuke, pariston (round 2 agents), leorio, zoro, netero (round 3 agents). New agents: gon, melody, ging-go, bisky, knov.

Dispute results: CRF-17 closed 4/4 (lifecycle now recorded in the PR description). CRF-52 closed 1/1 (Netero). CRF-58 closed 2/2 (orchestrator verified the longest help string at each stack head: 553, 633, 823, 988).

Cross-check: 4 new P4 and 2 new Nits posted; 1 Nit dropped (CRF-70: Gon P2 on two test section comments; keep-argument was AGENTS.md's substantive-comment rule; dropped because each comment accurately summarizes a multi-line block); 4 Notes dropped as fix confirmations or pattern observations; 1 OOS. Severity: Gon rated CRF-69 and CRF-70 P2; the reviewer prompt classes comment wording as Nit and both comments are accurate, as in round 2. Fix-chain findings: CRF-68 (the CRF-54 wording this review proposed misses StartRootAt) and CRF-72 (the post-End assertion this review proposed for CRF-53 cannot fail, consistent with the CRF-44 closure that the SDK already drops post-End writes). Both are posted as new inline comments rather than replies on the resolved CRF-54 and CRF-53 threads, because thread_action: unresolve fails with this token (round 3) and a reply inside a resolved thread stays collapsed. Interactions: CRF-74 (rename StartRootAt to StartTurnAt) changes the name CRF-68's rewrite uses. CRF-73 verified at the stack tip a62c34a: RecordStageDuration has one non-test caller (StageTracer.observe); the orchestrator added that the three stage family fields are also exported and unused outside metrics.go, so unexporting the method alone does not close the path. Reviewed against 22a6d34..b7d8353.

About deep-review

CRF = Coder Review Finding (P0-P4, Nit, Note)

Reviewer Focus
Bisky tests
Chopper ops/errors
Churn-guard change verification
Ging language modernization
Gon naming
Hisoka edge cases
Killua perf
Kite change integrity
Knov contracts
Knuckle SQL
Komugi flake/determinism
Kurapika security
Law decomposition
Leorio docs
Luffy product
Mafu-san process
Mafuuu contracts
Melody dispatch/pairing
Meruem structural
Nami frontend
Netero mechanical checks
Pariston premise testing
Pen-botter product gaps
Razor verification
Robin duplication
Ryosuke Go arch
Takumi concurrency
Zoro shape

🤖 Managed by Coder Agents.

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The core idea holds up: one call produces both the span and the histogram sample, so trace and metric durations cannot drift apart, and the typed Stage, Scope, and ChatKind constants keep string conversions at the label boundary. The series-cost table in the description is the cardinality accounting a new metric family should come with. Test density is about 50%, and the stage tests assert on real registry and span-recorder output rather than mock returns.

1 P2, 6 P3, 2 Nit, 2 Note.

This is a first-pass review only. These are mechanical findings from Netero; the full review panel has not reviewed this PR yet and will review after these findings are addressed. The P2 (CRF-1) is what holds the panel back: this PR ships a *-enabled experiment and three documented metric families with no production caller, so they never receive a sample unless the stacked PRs also land. CRF-2 (stale_anchor) has the same shape.

Split analysis (advisory, not blocking): at +1446 effective lines the PR was checked for decomposition. The recommendation is to keep the tracer, metric families, and experiment together, and to move the runGenerationStep extraction in generation.go into the stacked PR that adds the deferred stage end (or into a standalone refactor PR ahead of it), so its reason can be checked next to the defer it exists for (see CRF-11). That grouping argues against separating the experiment from the families, which favors CRF-1's first fix (bring the first caller into this PR) over its second (hold the experiment and doc rows back).

From Netero on CRF-5: "an alert on that series sends someone to look at NTP when the real problem is a missing timestamp."

🤖 This review was automatically generated with Coder Agents.

Comment thread coderd/x/chatd/chatloop/stage.go Resolved
Comment thread coderd/x/chatd/chatloop/metrics.go Resolved
Comment thread coderd/x/chatd/chatloop/stage.go Resolved
Comment thread coderd/x/chatd/chatloop/stage.go Resolved
Comment thread coderd/x/chatd/chatloop/stage.go Resolved
Comment thread coderd/x/chatd/chatd.go Resolved
Comment thread coderd/x/chatd/chatloop/metrics.go Resolved
Comment thread coderd/x/chatd/chatloop/stage_test.go Resolved
Comment thread coderd/x/chatd/generation.go Resolved
Comment thread coderd/x/chatd/chatloop/stage.go Resolved
@jscottmiller
jscottmiller force-pushed the scott/chatd-lifecycle/1-stage-tracer branch from 76818c9 to 5e32cbb Compare September 23, 2026 19:36
@jscottmiller
jscottmiller removed this pull request from stack #29773 September 23, 2026 21:05
@linear-code

linear-code Bot commented Sep 24, 2026

Copy link
Copy Markdown

CODAGT-1102

@jscottmiller

Copy link
Copy Markdown
Contributor Author

/coder-agents-review

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR adds the chatd stage tracer, three experiment-gated Prometheus families, and the chat-stage-metrics experiment. This round: 15 P3, 4 P4, 11 Nit.

The round 1 fixes are verified: the injectable clock, the missing_timestamp and future_start reasons, the negative_elapsed ordering, the experiment wiring test (it fails when chatd.go:2981 is forced to true), the gather helpers, and the shared pool label values. The four contested threads (no production callers, stale_anchor, nil-field guards, runGenerationStep extraction) are closed on your replies. Three of those closures assume the stack merges together; nothing in the repository prevents this PR from merging alone, so that needs a human decision.

The reply on the stale_anchor thread says the help text now says the emitter comes with #29771, but no file under coderd/x/chatd mentions #29771 and the help text clause is unchanged; correct the reply or the text.

Out of scope (needs a ticket or explicit acceptance by a human):

  • coderd/coderd.go:2566: ReadExperiments logs "ignoring unknown experiment" and then appends the experiment anyway, so the log says the opposite of what the server does.
  • coderd/x/chatd/chatd.go:278: ptr.Ref calls predate this PR; Go 1.26 go fix rewrites them to new(expr), while ptr.Ref is still the repository pattern.
  • CI Storybook job 107348848639 failed at this head. The only site/ change is the generated typesGenerated.ts; whether this PR caused the failure is unverified.

Notes:

  • chat-stage-metrics is the first ExperimentsSafe entry, so deployments running --experiments=* get the families on upgrade, and every deployment lists the experiment on the Deployment overview page and exports coderd_experiments{experiment="chat-stage-metrics"}.
  • stage.go:141: the time.Now() fallback in Now() is reachable only through WithClock(nil).

🤖 This review was automatically generated with Coder Agents.

Comment thread coderd/x/chatd/chatloop/stage.go Resolved
Comment thread coderd/x/chatd/chatloop/metrics.go Resolved
Comment thread codersdk/deployment.go Resolved
Comment thread coderd/x/chatd/chatloop/stage.go Resolved
Comment thread coderd/x/chatd/chatloop/metrics.go Resolved
Comment thread coderd/x/chatd/chatloop/stage.go Resolved
Comment thread coderd/x/chatd/chatloop/stage_test.go Resolved
Comment thread coderd/x/chatd/chatloop/stage.go Resolved
Comment thread coderd/x/chatd/chatloop/stage.go Resolved
Comment thread coderd/x/chatd/chatloop/stage.go Resolved
@jscottmiller
jscottmiller removed this pull request from stack #29806 September 25, 2026 05:05
@jscottmiller

Copy link
Copy Markdown
Contributor Author

/coder-agents-review

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR adds the chatd stage tracer, its experiment-gated Prometheus families, and the chat-stage-metrics experiment. This round adds 13 P4 findings, 12 inline and one below, and one P3 from the previous review stays open.

The fixes from the previous review are verified, including span-only anomaly filtering, span timestamps from the tracer clock, turn-only model observations, and NewMetrics leaving the stage families off. Six contested threads are closed on your replies. The thread on the experiment as the only switch for the series cost stays open and needs a human decision.

The API bullet in the PR description says End returns the elapsed time, but End(err error) returns nothing; the description becomes the squash commit message, so correct the bullet.

Out of scope (needs a ticket or explicit acceptance by a human):

  • coderd/x/chatd/generation.go:630: recordGenerationRetry picks the larger delay with an if, which go fix with the Go 1.26 modernizers rewrites to the max builtin.

Notes:

  • Correction to the previous review: chat-stage-metrics is the only current ExperimentsSafe entry, not the first one; minimum-implicit-member was safe from #26699 until #27472.

🤖 This review was automatically generated with Coder Agents.

Comment thread coderd/x/chatd/chatloop/stage.go Resolved
Comment thread coderd/x/chatd/chatloop/stage.go Resolved
Comment thread coderd/x/chatd/chatloop/stage.go Resolved
Comment thread coderd/x/chatd/chatloop/metrics.go Resolved
Comment thread coderd/x/chatd/chatloop/stage.go Resolved
Comment thread coderd/x/chatd/chatloop/stage.go Resolved
Comment thread coderd/x/chatd/chatloop/stage.go Resolved
@jscottmiller

Copy link
Copy Markdown
Contributor Author

/coder-agents-review

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR adds the chatd stage tracer, its experiment-gated Prometheus families, and the chat-stage-metrics experiment. This round: 4 P4, 2 Nit.

The fixes from the previous review are verified, including the two-way help text check, the removal of RecordAs, NopStageTracer, and AttrHTTPHost, and the queue_wait test that now takes scope and chat kind from the context. The three contested threads are closed on your replies: the experiment gate on the graduation lifecycle now in the description, the two stage lists, and the help string length.

Out of scope (needs a ticket or explicit acceptance by a human):

  • coderd/x/chatd/chatloop/metrics.go in #29770 to #29772: each later stack PR sets a new longest help string (633, 823, then 988 characters for coderd_chatd_turn_time_seconds_total at a62c34a), so each one re-pads about 245 rows of docs/admin/integrations/prometheus.md.

🤖 This review was automatically generated with Coder Agents.

Comment thread coderd/x/chatd/chatloop/stage.go Resolved
Comment thread coderd/x/chatd/chatloop/stage.go Resolved
Comment thread coderd/x/chatd/chatloop/stage_test.go Resolved
Comment thread coderd/x/chatd/chatloop/metrics.go Resolved
Comment thread coderd/x/chatd/chatloop/stage.go Resolved
@jscottmiller
jscottmiller force-pushed the scott/chatd-lifecycle/1-stage-tracer branch from ba38581 to 539484a Compare September 30, 2026 01:02
Scott Miller added 8 commits September 30, 2026 13:40
Add a StageTracer that emits one span and one stage_duration_seconds
observation per chat lifecycle stage, with typed Stage, Scope, and
ChatKind values, a model_stage_duration_seconds histogram for the
provider-bound stages labeled by provider_type and model, and a
stage_anomalies_total counter for dropped observations. The metric
families are registered when the chat-stage-metrics experiment is
enabled; spans are emitted regardless. Extract the generation step body
into its own function so the step has a boundary for a deferred stage
end.
Count stage anomalies only for observed stages, take live span
timestamps from the tracer clock, leave the stage families off in
NewMetrics, type anomaly reasons as StageAnomaly, and observe
model_stage_duration_seconds only for turn-scoped stages. Cross-reference
ttft_seconds and the time_to_first_token stage in their help text, and
tighten the tracer doc comments.
- Carry the wire provider on StageModel so SetModel and Record stamp it.
- Remove StartRoot and the unused links parameter of StartRootAt.
- Drop time_to_first_token from the model stage family; per-model TTFT
  stays on coderd_chatd_ttft_seconds.
- Remove the capacity_wait stage.
- Define the turn scope as latency attributable to a prompt.
- Add a consistency test over the stage sets and help text.
@jscottmiller
jscottmiller force-pushed the scott/chatd-lifecycle/1-stage-tracer branch from 539484a to 0be4ce3 Compare September 30, 2026 14:02
@jscottmiller
jscottmiller merged commit 7a94420 into main Sep 30, 2026
32 checks passed
@jscottmiller
jscottmiller deleted the scott/chatd-lifecycle/1-stage-tracer branch September 30, 2026 16:54
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 30, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants