Skip to content

feat: add chatd lifecycle stage tracer - #28969

Closed
jscottmiller wants to merge 1 commit into
scott/chatd-stages/1-extract-stepfrom
scott/chatd-stages/2-stage-tracer
Closed

jscottmiller wants to merge 1 commit into
scott/chatd-stages/1-extract-stepfrom
scott/chatd-stages/2-stage-tracer

Conversation

@jscottmiller

@jscottmiller jscottmiller commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

Superseded by #29769. Closed in favor of the reduced stack #29773.

Adds chatloop.StageTracer, which emits each chat lifecycle stage as an OpenTelemetry span and a coderd_chatd_stage_duration_seconds{stage, scope, chat_kind} observation from one call, so trace and metric durations cannot disagree. The provider-bound stages (time_to_first_token, stream, provider_attempt; thinking and compaction at full) are also observed on coderd_chatd_model_stage_duration_seconds{stage, provider_type, chat_kind, model}, the only family carrying provider_type and model, so series scale with model count only where the model explains the duration. Nothing calls it yet; PRs 3 to 7 instrument the chat loop, transport, and runner.

  • Labels: scope (turn / background), chat_kind (root / subagent, empty without a known chat). organization_name, provider_type, model, and reasoning_effort are span attributes; only provider_type and model are also metric labels, on the model histogram. No family carries an organization label, matching the existing chatd and AI Gateway metrics.
  • provider_type is the configured AI provider type (bedrock, azure, ...), as on the AI Gateway metrics. The pre-existing chatd provider label is the wire protocol (anthropic for Bedrock Claude, openai-compat for Azure), so spans carry both and the two should not be joined by provider.
  • API: Start / StartRoot / StartRootAt for live spans, Record / RecordAs for stages reconstructed from timestamps. End is idempotent; EndWithoutObservation ends the span without a sample. A nil tracer, provider, or metrics discards.
  • coderd_chatd_stage_anomalies_total{reason} counts negative or inverted windows that were dropped and turn anchors clamped as stale_anchor.
  • Buckets are round edges so alert thresholds are exact: 12 at basic (0.1 0.25 0.5 1 2.5 5 10 30 60 300 1800 3600), a 16-edge superset at full adding 0.05 20 120 600.
  • --chat-stage-metrics / CODER_CHAT_STAGE_METRICS (off, basic, full; default basic). basic observes chat_turn, queue_wait, capacity_wait, acquisition, mcp_connect, stream, time_to_first_token, provider_attempt, tool_call, commit, retry_backoff; full observes every stage; off registers no stage family. Spans are emitted at every level. Disabled families are built against a nil registerer so every recorder stays callable. coderd_chatd_stage_metrics_level{level} is one-hot across all three values so a restart at a new level overwrites the old series. The flag rejects unknown values; chatd.Config.StageMetrics and chatloop.MetricsOptions.StageMetrics treat empty or unknown as basic. NewMetrics / NopMetrics register at full so existing callers see every family.
Series cost at basic, against dogfood's measured baseline

A histogram label set is edges + +Inf + _sum + _count: 15 series at basic. Per replica, each chat_kind costs 181 (11 stages x 15, plus the 16 turn partition counters from PR 7) and each (chat_kind, model) costs 45 (3 stages x 15). Both chat kinds on M models: 362 + 90*M, plus one provider_attempt set per chat kind and per model in background scope.

Dogfood main, 3 replicas: ~3.4k coderd_* series per replica (~10.1k total), ~20 of them coderd_chatd_*. Tracing registers nothing on the Prometheus registry.

Models used per replica Added series (3 replicas) vs. baseline Extra TSDB/day (~2 B/sample, 15 s) Extra scrape per replica
1 ~1.4k +13% ~16 MB 57 KB (3 KB gzipped)
5 ~2.4k +24% ~28 MB 103 KB (5 KB gzipped)
10 ~3.8k +37% ~44 MB 160 KB (8 KB gzipped)
35 ~10.5k +104% ~121 MB 446 KB (22 KB gzipped)

Roughly 55% of customer deployments run one model and 89% run four or fewer, so a typical replica adds ~450 series. Series appear lazily and do not grow with organizations. The relabel rules in the dashboard README (#29735) collapse basic to ~64 per chat kind plus 15 per model. full is ~5x the fixed part (285 on the 16-edge ladder plus ~600 for the per-turn distributions per chat kind) and ~2x the per-model part (5 stages x 19).

🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.

@github-actions

github-actions Bot commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

Docs preview

Check off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here.

@jscottmiller
jscottmiller force-pushed the scott/chatd-stages/2-stage-tracer branch from 95c1126 to c2a8273 Compare September 9, 2026 23:19
@jscottmiller jscottmiller added the experimental Changes that might not necessarily be merged, until its approved to proceed with. label Sep 11, 2026
@jscottmiller
jscottmiller force-pushed the scott/chatd-stages/2-stage-tracer branch 2 times, most recently from f5bfc09 to 0643175 Compare September 11, 2026 19:17
@jscottmiller jscottmiller removed the experimental Changes that might not necessarily be merged, until its approved to proceed with. label Sep 14, 2026
@jscottmiller
jscottmiller force-pushed the scott/chatd-stages/2-stage-tracer branch 3 times, most recently from 84b3f71 to 5637ec8 Compare September 15, 2026 21:39
@jscottmiller
jscottmiller force-pushed the scott/chatd-stages/2-stage-tracer branch 2 times, most recently from 61527f5 to f9de6a0 Compare September 17, 2026 16:55
@jscottmiller
jscottmiller force-pushed the scott/chatd-stages/2-stage-tracer branch 3 times, most recently from 996f076 to 126805e Compare September 21, 2026 22:52
@linear-code

linear-code Bot commented Sep 22, 2026

Copy link
Copy Markdown

CODAGT-1102

@jscottmiller
jscottmiller force-pushed the scott/chatd-stages/2-stage-tracer branch from 126805e to 5a0c49b Compare September 23, 2026 01:37
Add StageTracer, which starts one OpenTelemetry span and records one
coderd_chatd_stage_duration_seconds observation per chat lifecycle
stage from the same call, so span and histogram durations cannot
diverge. Stages carry a scope, chat kind, and model identity read from
the context or set on the span, and stages reconstructed from persisted
timestamps are emitted through Record and RecordAs. Windows with
inverted or negative durations are dropped and counted on
coderd_chatd_stage_anomalies_total by reason.

The histogram is labelled by stage, scope, and chat kind. Stages that
are the provider's work on a model (time_to_first_token, stream,
provider_attempt, plus thinking and compaction at full) are also
observed on coderd_chatd_model_stage_duration_seconds, which adds
provider_type (the configured AI provider type, as on the AI Gateway
metrics) and model; the general histogram carries neither. The
organization name is a span attribute only, as are provider_type,
model, and reasoning effort on every stage that has resolved a model.
Buckets are round boundaries so alert thresholds land on bucket edges:
12 from 100ms to 1h at basic, a 16-edge superset from 50ms at full.

Add --chat-stage-metrics (CODER_CHAT_STAGE_METRICS), off|basic|full,
default basic. Basic observes the turn, wait, connect, model-call,
tool, and commit stages; full observes every stage; off registers none
of the stage families. Families disabled by the level are still
constructed against no registerer so every recorder is callable at
every level. A coderd_chatd_stage_metrics_level gauge reports the
configured level.
@jscottmiller
jscottmiller force-pushed the scott/chatd-stages/2-stage-tracer branch from 5a0c49b to a9b73de Compare September 23, 2026 02:14
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 23, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant