Skip to content

feat: add the chat-stage-metrics-full experiment - #29775

Closed
jscottmiller wants to merge 5 commits into
scott/chatd-lifecycle/4-turn-accountingfrom
scott/chatd-stage-metrics-full
Closed

jscottmiller wants to merge 5 commits into
scott/chatd-lifecycle/4-turn-accountingfrom
scott/chatd-stage-metrics-full

Conversation

@jscottmiller

Copy link
Copy Markdown
Contributor

Adds the chat-stage-metrics-full experiment (not in ExperimentsSafe), which widens the chat lifecycle stage metrics from #29769 through #29772 at a higher series count. Without it nothing changes.

With both chat-stage-metrics and chat-stage-metrics-full enabled:

  • Every stage is observed on coderd_chatd_stage_duration_seconds, including generation_step, prepare, thinking, and compaction, which are otherwise span-only.
  • Both stage histograms use a 16-edge ladder that is a superset of the 12-edge one (adds 0.05 20 120 600), so a query written against either reads the same edges.
  • thinking and compaction join coderd_chatd_model_stage_duration_seconds.
  • Three per-turn distribution families are registered, observed once per completed turn:
Family What
turn_time_seconds{category, chat_kind}, turn_time_share{category, chat_kind} per-turn distribution of each category's seconds and share; every category is emitted every turn so shares sum to 1
turn_stage_seconds{stage, chat_kind} total seconds per stage per turn; stages overlap, so this is a profile, not a partition

Per-turn histograms use an 11-edge ladder from 1s to 1h; turn_time_share uses twentieths. Cost per replica per chat_kind is roughly 900 series (15 stages x 19, plus the three distributions over 13 stages and 11 categories) against ~180 without it, and 95 per (chat_kind, model) against 45.

full without the base experiment registers nothing. chatloop.NewMetrics keeps registering the base set only; tests that need the full set use MetricsOptions{StageMetrics: true, FullStageMetrics: true}.

Dashboard rows for these families are not included; they belong with the dashboard in #29735 once it lands.

🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.

Scott Miller added 4 commits September 23, 2026 02:38
Add a StageTracer that emits one span and one stage_duration_seconds
observation per chat lifecycle stage, with typed Stage, Scope, and
ChatKind values, a model_stage_duration_seconds histogram for the
provider-bound stages labeled by provider_type and model, and a
stage_anomalies_total counter for dropped observations. The metric
families are registered when the chat-stage-metrics experiment is
enabled; spans are emitted regardless. Extract the generation step body
into its own function so the step has a boundary for a deferred stage
end.
@github-actions

Copy link
Copy Markdown
Contributor

Docs preview

Check off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here.

@jscottmiller jscottmiller changed the title feat(coderd/x/chatd): add the chat-stage-metrics-full experiment feat: add the chat-stage-metrics-full experiment Sep 23, 2026
Behind the chat-stage-metrics-full experiment, observe every lifecycle
stage on the stage histograms with a 16-edge ladder, add thinking and
compaction to the model histogram, and register the per-turn
distribution families turn_stage_seconds, turn_time_seconds, and
turn_time_share. Without it the families and stage set are unchanged.
@jscottmiller
jscottmiller force-pushed the scott/chatd-stage-metrics-full branch from 3fc5822 to 65fd415 Compare September 23, 2026 03:09
@jscottmiller
jscottmiller force-pushed the scott/chatd-lifecycle/4-turn-accounting branch 14 times, most recently from ff9b980 to dd49b8e Compare September 29, 2026 03:26
@linear-code

linear-code Bot commented Sep 30, 2026

Copy link
Copy Markdown

CODAGT-1102

@github-actions github-actions Bot locked and limited conversation to collaborators Oct 1, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant