Repository navigation
docs(examples/monitoring): add chatd lifecycle Grafana dashboard - #28975
Closed
jscottmiller wants to merge 1 commit into
Closed
jscottmiller wants to merge 1 commit into
jscottmiller wants to merge 1 commit into
Conversation
This was referenced Sep 4, 2026
jscottmiller
force-pushed
the
scott/chatd-stages/8-dashboard
branch
3 times, most recently
from
September 9, 2026 23:52
7d306ba to
b4c4da0
Compare
jscottmiller
force-pushed
the
scott/chatd-stages/8-dashboard
branch
from
September 11, 2026 02:25
b4c4da0 to
098e14d
Compare
jscottmiller
force-pushed
the
scott/chatd-stages/8-dashboard
branch
from
September 11, 2026 19:17
098e14d to
d52393b
Compare
jscottmiller
force-pushed
the
scott/chatd-stages/8-dashboard
branch
from
September 14, 2026 20:16
d52393b to
33ab94d
Compare
jscottmiller
force-pushed
the
scott/chatd-stages/8-dashboard
branch
from
September 14, 2026 20:34
33ab94d to
099519e
Compare
jscottmiller
force-pushed
the
scott/chatd-stages/8-dashboard
branch
from
September 14, 2026 22:06
099519e to
fe4d8df
Compare
jscottmiller
force-pushed
the
scott/chatd-stages/8-dashboard
branch
from
September 15, 2026 21:39
fe4d8df to
aa86059
Compare
jscottmiller
force-pushed
the
scott/chatd-stages/8-dashboard
branch
from
September 16, 2026 03:12
aa86059 to
0b20a69
Compare
jscottmiller
force-pushed
the
scott/chatd-stages/8-dashboard
branch
from
September 17, 2026 16:55
0b20a69 to
8a84f6a
Compare
jscottmiller
force-pushed
the
scott/chatd-stages/8-dashboard
branch
from
September 17, 2026 20:31
8a84f6a to
8342e49
Compare
jscottmiller
force-pushed
the
scott/chatd-stages/8-dashboard
branch
2 times, most recently
from
September 21, 2026 00:09
8a5ff27 to
9307b8d
Compare
Add a Grafana dashboard that plots the chatd stage histogram and the turn-end rollups one tree level at a time with a selectable summary statistic, plus summary panels for the chat pipeline, the per-turn partition of turn wall time, and a Model row reading the per-model stage histogram for time to first token, stream duration, and provider attempts by provider type. The README documents the stage hierarchy by tree level, the dashboard dimensions, each panel group, and setup. Rows that need only the basic stage metrics level are expanded by default; the per-turn distribution panels and the stages observed only at the full level sit in collapsed rows named for the level. A stat panel reports the level from coderd_chatd_stage_metrics_level. The README adds a metric levels section with series cost measured against a dogfood replica's baseline, example alert rules on the round bucket edges, and a scrape-time relabel snippet for alert-only deployments.
jscottmiller
force-pushed
the
scott/chatd-stages/8-dashboard
branch
from
September 21, 2026 22:52
9307b8d to
3f619e2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Grafana dashboard for the chatd lifecycle metrics at
examples/monitoring/dashboards/grafana/chatd-lifecycle/, with a README documenting every panel.Rows, sharing one
$statselector (mean, p50, p90, p95, p99):coderd_chatd_stage_metrics_level(== 1, counts replicas per level).turn_time_seconds_total.basiccounters.basicstages, each with a collapsed(full level only)row holding seconds per turn, the per-turn category share, and thefull-only stages. The label schema is the same at every level, so those panels read "No data" rather than erroring atbasic.coderd_chatd_model_stage_duration_seconds, plus background provider calls.Variables:
$datasource,$stat,$chat_kind,$model;$modelapplies only to the Model row since no other family carries the label, and there is no organization variable since no chatd metric carries that label. Every division guards its denominator and everyhistogram_quantilekeepsle. Panel descriptions name variables in prose because Grafana interpolates$varinto.*.The README covers the levels with an annotated stage tree; series cost per level against dogfood's measured baseline (~3.4k
coderd_*series per replica); alert rules on the round bucket edges (le=~"10(\\.0)?", since Prometheus 3 stores integer edges as10.0); a drop-only relabel snippet for alert-only deployments and the panels it empties; theprovider_typeversus legacyproviderdistinction; whenmcp_connectpopulates; whenreasoning_effortappears on spans; the fixed capacity limits and their removal by theagent_runtime_hoursentitlement; a TraceQL filter for turn traces; an OTLP endpoint example; and the empty labels on title generation.An earlier revision had an aggregate flamegraph row; it was removed because it loaded every bucket series over
[$__range]on each refresh and Prometheus cannot compute self time for overlapping stages, so the shape was misleading.Verified live (Grafana 12.1, Prometheus, Tempo) at
basicandfullbefore the model histogram and the label changes landed; the reworked Model row has been checked against the metric definitions but not yet re-driven live.Part of the chatd lifecycle observability stack: 1 extract step · 2 stage tracer · 3 stream stages · 4 provider attempts · 5 turn span · 6 capacity wait · 7 turn accounting · 8 dashboard. Supersedes the experimental branch in #28741.
🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.