Skip to content

docs(examples/monitoring): add chatd lifecycle Grafana dashboard - #29735

Closed
jscottmiller wants to merge 1 commit into
mainfrom
scott/chatd-dashboard
Closed

jscottmiller wants to merge 1 commit into
mainfrom
scott/chatd-dashboard

Conversation

@jscottmiller

@jscottmiller jscottmiller commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Grafana dashboard for the chatd lifecycle metrics at examples/monitoring/dashboards/grafana/chatd-lifecycle/, with a README covering the stage tree, dimensions, caveats, series cost, alert examples, and setup.

Depends on the chatd lifecycle observability stack (#29769 through #29772). The families the panels read (coderd_chatd_stage_duration_seconds, coderd_chatd_model_stage_duration_seconds, and the turn-end counters) land in that stack behind the chat-stage-metrics experiment, so this stays a draft until it merges. Moved out of the stack from #28975 so the dashboard can be reviewed independently of the Go changes.

Rows, sharing one $stat selector (mean, p50, p90, p95, p99):

  • Level 0: turn duration and turns per minute by chat kind, turn outcomes (stacked total equals turns per minute), and the turn time mix by chat kind from turn_time_seconds_total.
  • Turn time partition: seconds per turn by category and unattributed time.
  • One row per stage tree level with duration per occurrence. generation_step, prepare, thinking, and compaction are span-only and not plotted.
  • Model: time to first token, stream duration by model, and provider attempts per minute by provider type and model from coderd_chatd_model_stage_duration_seconds, plus background provider calls.

Variables: $datasource, $stat, $chat_kind, $model; $model applies only to the Model row since no other family carries the label. Every division guards its denominator and every histogram_quantile keeps le.

Rows for the chat-stage-metrics-full families (#29775) are not included here.

Verified live (Grafana 12.1, Prometheus, Tempo) before the model histogram and the label changes landed; the reworked Model row has been checked against the metric definitions but not yet re-driven live.

🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.

Add a Grafana dashboard that plots the chatd stage histogram and the
turn-end rollups one tree level at a time with a selectable summary
statistic, plus summary panels for the chat pipeline, the per-turn
partition of turn wall time, and a Model row reading the per-model
stage histogram for time to first token, stream duration, and provider
attempts by provider type. The README documents the stage hierarchy by
tree level, the dashboard dimensions, each panel group, and setup.

Rows that need only the basic stage metrics level are expanded by
default; the per-turn distribution panels and the stages observed only
at the full level sit in collapsed rows named for the level. A stat
panel reports the level from coderd_chatd_stage_metrics_level. The
README adds a metric levels section with series cost measured against a
dogfood replica's baseline, example alert rules on the round bucket
edges, and a scrape-time relabel snippet for alert-only deployments.
@linear-code

linear-code Bot commented Sep 24, 2026

Copy link
Copy Markdown

CODAGT-1102

@github-actions github-actions Bot locked and limited conversation to collaborators Oct 16, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant