Repository navigation
feat: add chatd lifecycle stage tracer - #28969
Closed
jscottmiller wants to merge 1 commit into
Closed
jscottmiller wants to merge 1 commit into
jscottmiller wants to merge 1 commit into
Conversation
Contributor
Docs previewCheck off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here. |
This was referenced Sep 4, 2026
jscottmiller
force-pushed
the
scott/chatd-stages/2-stage-tracer
branch
from
September 9, 2026 23:19
95c1126 to
c2a8273
Compare
jscottmiller
force-pushed
the
scott/chatd-stages/2-stage-tracer
branch
2 times, most recently
from
September 11, 2026 19:17
f5bfc09 to
0643175
Compare
jscottmiller
force-pushed
the
scott/chatd-stages/2-stage-tracer
branch
3 times, most recently
from
September 15, 2026 21:39
84b3f71 to
5637ec8
Compare
jscottmiller
force-pushed
the
scott/chatd-stages/2-stage-tracer
branch
2 times, most recently
from
September 17, 2026 16:55
61527f5 to
f9de6a0
Compare
jscottmiller
force-pushed
the
scott/chatd-stages/2-stage-tracer
branch
3 times, most recently
from
September 21, 2026 22:52
996f076 to
126805e
Compare
jscottmiller
force-pushed
the
scott/chatd-stages/2-stage-tracer
branch
from
September 23, 2026 01:37
126805e to
5a0c49b
Compare
Add StageTracer, which starts one OpenTelemetry span and records one coderd_chatd_stage_duration_seconds observation per chat lifecycle stage from the same call, so span and histogram durations cannot diverge. Stages carry a scope, chat kind, and model identity read from the context or set on the span, and stages reconstructed from persisted timestamps are emitted through Record and RecordAs. Windows with inverted or negative durations are dropped and counted on coderd_chatd_stage_anomalies_total by reason. The histogram is labelled by stage, scope, and chat kind. Stages that are the provider's work on a model (time_to_first_token, stream, provider_attempt, plus thinking and compaction at full) are also observed on coderd_chatd_model_stage_duration_seconds, which adds provider_type (the configured AI provider type, as on the AI Gateway metrics) and model; the general histogram carries neither. The organization name is a span attribute only, as are provider_type, model, and reasoning effort on every stage that has resolved a model. Buckets are round boundaries so alert thresholds land on bucket edges: 12 from 100ms to 1h at basic, a 16-edge superset from 50ms at full. Add --chat-stage-metrics (CODER_CHAT_STAGE_METRICS), off|basic|full, default basic. Basic observes the turn, wait, connect, model-call, tool, and commit stages; full observes every stage; off registers none of the stage families. Families disabled by the level are still constructed against no registerer so every recorder is callable at every level. A coderd_chatd_stage_metrics_level gauge reports the configured level.
jscottmiller
force-pushed
the
scott/chatd-stages/2-stage-tracer
branch
from
September 23, 2026 02:14
5a0c49b to
a9b73de
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Superseded by #29769. Closed in favor of the reduced stack #29773.
Adds
chatloop.StageTracer, which emits each chat lifecycle stage as an OpenTelemetry span and acoderd_chatd_stage_duration_seconds{stage, scope, chat_kind}observation from one call, so trace and metric durations cannot disagree. The provider-bound stages (time_to_first_token,stream,provider_attempt;thinkingandcompactionatfull) are also observed oncoderd_chatd_model_stage_duration_seconds{stage, provider_type, chat_kind, model}, the only family carryingprovider_typeandmodel, so series scale with model count only where the model explains the duration. Nothing calls it yet; PRs 3 to 7 instrument the chat loop, transport, and runner.scope(turn/background),chat_kind(root/subagent, empty without a known chat).organization_name,provider_type,model, andreasoning_effortare span attributes; onlyprovider_typeandmodelare also metric labels, on the model histogram. No family carries an organization label, matching the existing chatd and AI Gateway metrics.provider_typeis the configured AI provider type (bedrock,azure, ...), as on the AI Gateway metrics. The pre-existing chatdproviderlabel is the wire protocol (anthropicfor Bedrock Claude,openai-compatfor Azure), so spans carry both and the two should not be joined by provider.Start/StartRoot/StartRootAtfor live spans,Record/RecordAsfor stages reconstructed from timestamps.Endis idempotent;EndWithoutObservationends the span without a sample. A nil tracer, provider, or metrics discards.coderd_chatd_stage_anomalies_total{reason}counts negative or inverted windows that were dropped and turn anchors clamped asstale_anchor.basic(0.1 0.25 0.5 1 2.5 5 10 30 60 300 1800 3600), a 16-edge superset atfulladding0.05 20 120 600.--chat-stage-metrics/CODER_CHAT_STAGE_METRICS(off,basic,full; defaultbasic).basicobserveschat_turn,queue_wait,capacity_wait,acquisition,mcp_connect,stream,time_to_first_token,provider_attempt,tool_call,commit,retry_backoff;fullobserves every stage;offregisters no stage family. Spans are emitted at every level. Disabled families are built against a nil registerer so every recorder stays callable.coderd_chatd_stage_metrics_level{level}is one-hot across all three values so a restart at a new level overwrites the old series. The flag rejects unknown values;chatd.Config.StageMetricsandchatloop.MetricsOptions.StageMetricstreat empty or unknown asbasic.NewMetrics/NopMetricsregister atfullso existing callers see every family.Series cost at
basic, against dogfood's measured baselineA histogram label set is edges +
+Inf+_sum+_count: 15 series atbasic. Per replica, eachchat_kindcosts 181 (11 stages x 15, plus the 16 turn partition counters from PR 7) and each(chat_kind, model)costs 45 (3 stages x 15). Both chat kinds onMmodels:362 + 90*M, plus oneprovider_attemptset per chat kind and per model in background scope.Dogfood
main, 3 replicas: ~3.4kcoderd_*series per replica (~10.1k total), ~20 of themcoderd_chatd_*. Tracing registers nothing on the Prometheus registry.Roughly 55% of customer deployments run one model and 89% run four or fewer, so a typical replica adds ~450 series. Series appear lazily and do not grow with organizations. The relabel rules in the dashboard README (#29735) collapse
basicto ~64 per chat kind plus 15 per model.fullis ~5x the fixed part (285 on the 16-edge ladder plus ~600 for the per-turn distributions per chat kind) and ~2x the per-model part (5 stages x 19).🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.