Skip to content

docs(examples/monitoring): add chatd lifecycle Grafana dashboard - #28975

Closed
jscottmiller wants to merge 1 commit into
scott/chatd-stages/7-turn-accountingfrom
scott/chatd-stages/8-dashboard
Closed

jscottmiller wants to merge 1 commit into
scott/chatd-stages/7-turn-accountingfrom
scott/chatd-stages/8-dashboard

Conversation

@jscottmiller

@jscottmiller jscottmiller commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

Grafana dashboard for the chatd lifecycle metrics at examples/monitoring/dashboards/grafana/chatd-lifecycle/, with a README documenting every panel.

Rows, sharing one $stat selector (mean, p50, p90, p95, p99):

  • Level stat from the one-hot coderd_chatd_stage_metrics_level (== 1, counts replicas per level).
  • Level 0: turn duration and turns per minute by chat kind, turn outcomes (stacked total equals turns per minute), and the turn time mix by chat kind from turn_time_seconds_total.
  • Turn time partition: seconds per turn by category and unattributed time, from the basic counters.
  • One row per stage tree level with duration per occurrence for the basic stages, each with a collapsed (full level only) row holding seconds per turn, the per-turn category share, and the full-only stages. The label schema is the same at every level, so those panels read "No data" rather than erroring at basic.
  • Model: time to first token, stream duration by model, and provider attempts per minute by provider type and model from coderd_chatd_model_stage_duration_seconds, plus background provider calls.

Variables: $datasource, $stat, $chat_kind, $model; $model applies only to the Model row since no other family carries the label, and there is no organization variable since no chatd metric carries that label. Every division guards its denominator and every histogram_quantile keeps le. Panel descriptions name variables in prose because Grafana interpolates $var into .*.

The README covers the levels with an annotated stage tree; series cost per level against dogfood's measured baseline (~3.4k coderd_* series per replica); alert rules on the round bucket edges (le=~"10(\\.0)?", since Prometheus 3 stores integer edges as 10.0); a drop-only relabel snippet for alert-only deployments and the panels it empties; the provider_type versus legacy provider distinction; when mcp_connect populates; when reasoning_effort appears on spans; the fixed capacity limits and their removal by the agent_runtime_hours entitlement; a TraceQL filter for turn traces; an OTLP endpoint example; and the empty labels on title generation.

An earlier revision had an aggregate flamegraph row; it was removed because it loaded every bucket series over [$__range] on each refresh and Prometheus cannot compute self time for overlapping stages, so the shape was misleading.

Verified live (Grafana 12.1, Prometheus, Tempo) at basic and full before the model histogram and the label changes landed; the reworked Model row has been checked against the metric definitions but not yet re-driven live.


Part of the chatd lifecycle observability stack: 1 extract step · 2 stage tracer · 3 stream stages · 4 provider attempts · 5 turn span · 6 capacity wait · 7 turn accounting · 8 dashboard. Supersedes the experimental branch in #28741.

🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.

@jscottmiller
jscottmiller force-pushed the scott/chatd-stages/8-dashboard branch 3 times, most recently from 7d306ba to b4c4da0 Compare September 9, 2026 23:52
@jscottmiller jscottmiller added the experimental Changes that might not necessarily be merged, until its approved to proceed with. label Sep 11, 2026
@jscottmiller
jscottmiller force-pushed the scott/chatd-stages/8-dashboard branch from b4c4da0 to 098e14d Compare September 11, 2026 02:25
@jscottmiller
jscottmiller force-pushed the scott/chatd-stages/8-dashboard branch from 098e14d to d52393b Compare September 11, 2026 19:17
@jscottmiller jscottmiller removed the experimental Changes that might not necessarily be merged, until its approved to proceed with. label Sep 14, 2026
@jscottmiller
jscottmiller force-pushed the scott/chatd-stages/8-dashboard branch from d52393b to 33ab94d Compare September 14, 2026 20:16
@jscottmiller
jscottmiller force-pushed the scott/chatd-stages/8-dashboard branch from 33ab94d to 099519e Compare September 14, 2026 20:34
@jscottmiller
jscottmiller force-pushed the scott/chatd-stages/8-dashboard branch from 099519e to fe4d8df Compare September 14, 2026 22:06
@jscottmiller
jscottmiller force-pushed the scott/chatd-stages/8-dashboard branch from fe4d8df to aa86059 Compare September 15, 2026 21:39
@jscottmiller
jscottmiller force-pushed the scott/chatd-stages/8-dashboard branch from aa86059 to 0b20a69 Compare September 16, 2026 03:12
@jscottmiller
jscottmiller force-pushed the scott/chatd-stages/8-dashboard branch from 0b20a69 to 8a84f6a Compare September 17, 2026 16:55
@jscottmiller
jscottmiller force-pushed the scott/chatd-stages/8-dashboard branch from 8a84f6a to 8342e49 Compare September 17, 2026 20:31
@jscottmiller
jscottmiller force-pushed the scott/chatd-stages/8-dashboard branch 2 times, most recently from 8a5ff27 to 9307b8d Compare September 21, 2026 00:09
Add a Grafana dashboard that plots the chatd stage histogram and the
turn-end rollups one tree level at a time with a selectable summary
statistic, plus summary panels for the chat pipeline, the per-turn
partition of turn wall time, and a Model row reading the per-model
stage histogram for time to first token, stream duration, and provider
attempts by provider type. The README documents the stage hierarchy by
tree level, the dashboard dimensions, each panel group, and setup.

Rows that need only the basic stage metrics level are expanded by
default; the per-turn distribution panels and the stages observed only
at the full level sit in collapsed rows named for the level. A stat
panel reports the level from coderd_chatd_stage_metrics_level. The
README adds a metric levels section with series cost measured against a
dogfood replica's baseline, example alert rules on the round bucket
edges, and a scrape-time relabel snippet for alert-only deployments.
@jscottmiller
jscottmiller force-pushed the scott/chatd-stages/8-dashboard branch from 9307b8d to 3f619e2 Compare September 21, 2026 22:52
@jscottmiller
jscottmiller deleted the scott/chatd-stages/8-dashboard branch September 22, 2026 15:56
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 22, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant