Skip to content

feat: trace chatd provider stages and count only output parts as first token - #29770

Merged
jscottmiller merged 7 commits into
scott/chatd-lifecycle/1-stage-tracerfrom
scott/chatd-lifecycle/2-provider-stages
Sep 30, 2026
Merged

jscottmiller merged 7 commits into
scott/chatd-lifecycle/1-stage-tracerfrom
scott/chatd-lifecycle/2-provider-stages

Conversation

@jscottmiller

@jscottmiller jscottmiller commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

TL;DR: Traces the model provider call: a stream stage, a nested time_to_first_token stage, and one provider_attempt per HTTP round trip, labelled with provider type and model. Behavior change: coderd_chatd_ttft_seconds now counts only a real output part as the first token, so error, warning, and finish parts no longer count.

Instruments the provider call.

Stream and time to first token. GenerateAssistant records a stream stage from stream open to close and a nested time_to_first_token stage. The TTFT window closes once, on the first streamed output part: any part other than warnings, finish, or error, including start markers such as text_start. A closed window observes the stage and the pre-existing coderd_chatd_ttft_seconds with the same elapsed time. A failed open, an error part, or a release without an output part ends the TTFT span with the error (a silence timeout is classified as such) and observes nothing. The stream span ends once, with the error GenerateAssistant returns, so content-filter and no-output attempts end errored. Failed stream windows are observed like successful ones. The attempt is released before the stream span ends, so the TTFT window always falls inside its stream. The compaction summary stream passes a nil tracer, so it records neither.

Without #29771, production emits only provider_attempt; #29771 passes the tracer to GenerateAssistant.

Provider attempts. The provider HTTP transport is wrapped in stageSpanRoundTripper, which emits one provider_attempt stage per round trip, so retries each get a span. The span closes on response headers; a status >= 400 marks it errored without changing what the client sees.

resolveModelCall resolves the StageModel (wire provider, configured provider_type, model, effective reasoning effort) and threads it to the transport, so span attributes and labels match the call. provider_type is read from ai_providers.type, so types outside the fantasy set (copilot) keep their configured name. chatd.New builds the tracer with the server clock.

Behavior change to coderd_chatd_ttft_seconds: only an output part counts as a first token. Error parts no longer close the window as a success, and warnings and finish parts are skipped; the help text says so.

StageSpan.End returns the span's elapsed window, which ttft_seconds observes. Callers pass Stages and StageModel through GenerateAssistantOptions; a nil tracer discards (TTFT still records ttft_seconds). Server gains the stages tracer and a TracerProvider option, wired from coderd.

Review decisions
  • Start markers close the TTFT window: they are the first sign the model is producing output, and waiting for a delta would fold hidden reasoning time into TTFT.
  • Failed stream and provider_attempt windows are observed: that time was spent. TTFT is the exception because a first token that never arrived has no duration.
  • http_host was dropped from provider_attempt; every request goes through the local AI Gateway transport, so it was constant.
  • The advisor tool's nested model call is intentionally not instrumented as stream/TTFT.

Supersedes #28970 and #28971.

🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Docs preview

Check off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here.

@jscottmiller jscottmiller changed the title feat(coderd/x/chatd): trace provider stream and attempt stages feat: trace chatd provider stream and attempt stages Sep 23, 2026
@jscottmiller
jscottmiller force-pushed the scott/chatd-lifecycle/2-provider-stages branch from 9e8add5 to aacc28a Compare September 23, 2026 04:25
@jscottmiller
jscottmiller marked this pull request as ready for review September 23, 2026 04:26
@jscottmiller

Copy link
Copy Markdown
Contributor Author

/coder-agents-review

@coder-agents-review

coder-agents-review Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Chat: Review posted | View chat
Requested: 2026-09-25 17:29 UTC by @jscottmiller

Review history
  • R2 (2026-09-24): 13 reviewers, 7 Nit, 1 Note, 2 P2, 16 P3, 1 P4, COMMENT. Review
  • R3 (2026-09-25): 14 reviewers, 12 Nit, 1 Note, 2 P2, 16 P3, 7 P4, COMMENT. Review
  • R4 (2026-09-25): 10 reviewers, 12 Nit, 1 Note, 2 P2, 16 P3, 9 P4, COMMENT. Review

deep-review v0.13.0 | Round 4 | b7d8353..e6f36ac

Last posted: Round 4, 40 findings (2 P2, 16 P3, 9 P4, 12 Nit, 1 Note), COMMENT. Review

Finding inventory

Finding inventory: PR #29770

Findings

# Sev Status Location Summary Round Reviewer Posted
CRF-1 P2 Author fixed (3809e82) chatloop.go:860 Warnings and finish parts close the TTFT window as a success, contradicting the new help text R1 Netero Yes
CRF-2 P2 Author contested; panel closed R2 (1/1 accept) chatloop.go:126 No production caller sets Stages/StageModel, so stream and TTFT stages are never emitted R1 Netero Yes
CRF-3 P3 Author fixed (3809e82) chatloop.go:361 No test fails if the release-before-stream-end ordering in endStream is broken R1 Netero Yes
CRF-4 P3 Author fixed (3809e82) modelcall.go:199 Effort resolved twice; ProviderOptionsForCall parameter now misdescribes its input R1 Netero Yes
CRF-5 P3 Author contested; panel closed R2 (1/1 accept) stage_internal_test.go:128 Stage test fixture and metric helpers duplicate stage_test.go helpers, with a third copy in chatd R1 Netero Yes
CRF-6 Note Author fixed (3809e82); production wiring incomplete, see CRF-7 chatloop.go:830 TTFT histogram and time_to_first_token stage read different clocks R1 Netero Yes
CRF-7 P3 Author fixed (5bb414f) chatd.go:2996 Server.New builds the stage tracer without the server clock, so stages ignore an injected clock R2 Netero Yes
CRF-8 P3 Author contested; panel closed R3 (4/4 accept) model_routing_aibridge.go:157 Failed provider attempts and stream opens are observed into stage histograms with no outcome label R2 Kite, Melody, Ryosuke, Pen Botter Yes
CRF-9 P3 Author fixed (5bb414f) model_routing_aibridge.go:147 http_host on every production provider_attempt span is the constant coder-aibridge R2 Ryosuke P3, Melody P4, Pen Botter P4, Mafuuu P4 Yes
CRF-10 P3 Author contested; panel closed R3 (2/2 accept) model_routing_aibridge.go:132 provider_attempt times the in-process AI Gateway call to its first upstream event; gateway retries share one span R2 Pariston P3, Pen Botter Note Yes
CRF-11 P3 Author fixed (5bb414f) chatloop.go:827 TTFT span records context canceled or errNoFirstToken instead of the stream silence timeout R2 Mafuuu, Takumi Yes
CRF-12 P3 Author contested; panel closed R3 (1/1 accept) chatloop.go:863 text_start, reasoning_start, tool_input_start still close the TTFT window as a first token R2 Pariston Yes
CRF-13 P3 Author fixed (5bb414f) modelcall.go:164 resolvedEffort comment omits the clamp and calls the pre-mapping effort the value the provider uses R2 Mafuuu P3, Ryosuke P3, Gon P2 Yes
CRF-14 P3 Author fixed (5bb414f) compaction.go:528 Compaction comment cites turn accounting that does not exist at this commit R2 Gon P2, Leorio P4, Ryosuke Nit, Melody Note Yes
CRF-15 P3 Author fixed (5bb414f) chatloop.go:791 errNoFirstToken doc says no part streamed, but warnings and finish parts can stream first R2 Leorio P3, Gon P2 Yes
CRF-16 P3 Author fixed (5bb414f) modelcall.go:77 stageModel doc says resolvedProvider is empty for copilot; it is inferred from the model name R2 Leorio P3, Gon P2 Yes
CRF-17 P3 Author fixed (5bb414f, adeadee) modelcall.go:172 No test fails if resolveModelCall stops passing StageModel to the transport R2 Bisky Yes
CRF-18 P3 Author fixed (5bb414f) modelcall_internal_test.go:150 stageModel test uses openai, so it cannot catch ProviderType reverting to resolvedProvider R2 Bisky P3, Kite P4 Yes
CRF-19 P3 Author contested; panel closed R3 (1/1 accept) chatloop.go:830 One TTFT window is observed into three histograms; two duplicate coderd_chatd_ttft_seconds R2 Luffy Yes
CRF-20 P4 Author fixed (5bb414f) metrics.go:166 ttft_seconds help text omits empty-finish streams and does not define content part R2 Leorio Yes
CRF-21 Nit Author fixed (5bb414f) chatloop.go:126 Stages and StageModel fields on GenerateAssistantOptions have no field comments R2 Leorio Yes
CRF-22 Nit Author fixed (5bb414f) chatloop/stage_internal_test.go:167 drainStream uses _ = part instead of for range stream {} R2 Ging-Go Yes
CRF-23 Nit Author fixed (5bb414f) model_routing_aibridge.go:139 stageSpanRoundTripper field named model holds the StageModel R2 Gon Yes
CRF-24 Nit Author contested; panel closed R3 (1/1 accept) chatd/stage_internal_test.go:73 wantStatusCode holds span status, not HTTP status; wantAttribute is ambiguous R2 Gon Yes
CRF-25 Nit Author fixed (5bb414f) chatloop.go:818 finishTTFT and part-switch comments restate the code (818, 864) R2 Gon P2 Yes
CRF-26 Nit Author fixed (5bb414f) chatloop/stage_internal_test.go:23 Test helper docs restate bodies or misstate behavior (23, 42, 61, 103, 148, 166, 212) R2 Gon P2 Yes
CRF-27 Nit Author fixed (5bb414f) chatd/stage_internal_test.go:24 Test helper docs restate bodies (24, 31, 54, 150) R2 Gon P2 Yes
CRF-28 Note Note (body); superseded R3 (requireEndedBefore compares recorder end order, per Netero) chatloop/stage_internal_test.go:468 Ordering assertion compares OTel wall-clock end times and can miss a reorder on the same tick R2 Bisky No
CRF-29 Note Note (body); author fixed (e6f36ac, stream span ends with the returned error) chatloop.go:378 endStream(nil) runs before content-filter and no-output checks, so the stream span reads success for attempts returned as errors R2 Kite No
CRF-30 Note Note (body); author fixed (PR title names the TTFT change before R4) metrics.go:166 coderd_chatd_ttft_seconds shipped in v2.33.0; the PR title does not mention its changed meaning R2 Ryosuke No
CRF-31 Note Note (body); author fixed (PR description states it before R4) PR description Description does not say stream and TTFT stages start only with #29771 R2 Leorio No
CRF-32 OOS Out of scope (body); resolved in base b7d8353 (StageModel.Effort doc now describes the pre-mapping effort, per Ryosuke) stage.go:151 StageModel.Effort doc says the effort is sent to the provider; it is the pre-mapping value R2 Ryosuke No
CRF-33 OOS Out of scope (body) metrics.go:175 Stage histograms have no outcome label, so failed windows cannot be separated for any stage R2 Ryosuke No
CRF-34 OOS Out of scope (body); CI green at adeadee, no longer listed CI Storybook and docs-analysis CI jobs fail on a head that changes no site/ files; cause unverified R2 Netero No
CRF-35 Nit Dropped by orchestrator (Leorio found the comment states behavior the reader needs) model_routing_aibridge.go:155 4xx/5xx comment restates the next two lines R2 Gon P2 No
CRF-36 Nit Dropped by orchestrator (Leorio found the comment states behavior the reader needs) model_routing_aibridge.go:161 Header-close comment makes one point twice R2 Gon P2 No
CRF-37 Nit Dropped by orchestrator (Leorio found the ordering comment names the constraint and consequence) chatloop.go:358 Release-ordering comment is 36 words R2 Gon P2 No
CRF-38 Nit Author fixed (e6f36ac) model_routing_aibridge.go:132 stageSpanRoundTripper doc's last sentence repeats SetModel R2 Gon P2 Yes
CRF-39 Note Dropped by orchestrator (CRF-4 closed; both calls use identical inputs; no new evidence) modelcall.go:166 Effort still resolved twice R2 Gon No
CRF-40 Nit Author fixed (e6f36ac) modelcall_internal_test.go:101 TestResolveModelCallResolvedEffort now mainly tests the stage identity; name says effort R3 Netero Yes
CRF-41 Nit Author fixed (e6f36ac) modelcall.go:84 stageModel() local named model next to r.model; same pattern in two test files (CRF-23 siblings) R3 Gon Yes
CRF-42 Nit Author fixed (e6f36ac) stage_metrics_internal_test.go:41 TestServerStageMetricsFollowExperiment also asserts the tracer clock; name covers only the experiment gate R3 Gon Yes
CRF-43 P4 Author fixed (e6f36ac) model_routing_aibridge.go:149 No test checks that RoundTrip passes the provider_attempt span context to the base transport R3 Bisky Yes
CRF-44 P4 Author fixed (e6f36ac) chatd.go:2996 No test fails if chatd.New stops passing cfg.TracerProvider to the stage tracer R3 Bisky Yes
CRF-45 P4 Author fixed (e6f36ac) chatloop/stage_internal_test.go:258 NilTracerObservesTTFTSeconds asserts an empty recorder that the nil-tracer path never touches R3 Bisky Yes
CRF-46 P4 Author fixed (e6f36ac) stage.go:358 End doc says it returns the recorded duration; finishTTFT relies on the elapsed window even when nothing is observed R3 Leorio Yes
CRF-47 P4 Author contested; panel closed R4 (2/2 accept) modelcall.go:89 Stage identity derived twice (stageModel() before/after build, newModel patches Provider); PR description says resolveModelCall resolves it once R3 Meruem, Leorio Yes
CRF-48 Nit Author fixed (e6f36ac) chatloop.go:823 finishTTFT comment says the silence timeout replaces err; the code replaces err with the classified timeout R3 Leorio Yes
CRF-49 P4 Author fixed (e6f36ac) chatloop.go:363 stream span ended at three call sites; ordering kept by convention; could be owned by guardedStream R3 Meruem Yes
CRF-50 P4 Open chatloop/stage_internal_test.go:238 No test fails if a nil Stages tracer starts observing stage_duration_seconds{stage="time_to_first_token"} (CRF-45 fix removed the only nil-tracer span assertion) R4 Netero Yes
CRF-51 P4 Open chatloop.go:375 stream span records the unwrapped error on open and stream failures but retErr on content-filter/no-output; description says returned error; comment at 361 states the split without a reason R4 Pen Botter P4, Gon P2 (comment at chatloop.go:361) Yes
CRF-52 Nit Dropped by orchestrator (second sentence is the CRF-48 wording this review proposed; first states End's histogram side effect at the call site) chatloop.go:826 finishTTFT comment restates ttftSpan.End(nil) and the if body R4 Gon P2 No
CRF-53 Nit Dropped by orchestrator (assertion comment states the property under test; field type names what seen holds) chatd/stage_internal_test.go:119 stubRoundTripper.seen needs comments at the field and assertion; rename to requestSpanContext R4 Gon P2 No

Contested and acknowledged

CRF-2 (P2, chatloop.go:126) - No production caller sets Stages/StageModel

  • Finding: Neither production GenerateAssistant caller (generation.go, chatadvisor/runner.go) passes Stages or StageModel, so this PR alone emits no stream or time_to_first_token stage. Proposed wiring generation.go here or stating in the description that the next PR wires it, and deciding whether advisor calls are instrumented.
  • Author defense: feat: open a chat_turn span per prompt #29771 passes Stages/StageModel for normal generation. The advisor's nested call intentionally stays uninstrumented and is attributed to its enclosing tool_call; feat: open a chat_turn span per prompt #29771 documents that in ARCHITECTURE.md.
  • Panel closure (R2, 1/1): Netero, the agent that raised it (reused from R1), accepted: without feat: open a chat_turn span per prompt #29771 production emits only provider_attempt stages, and feat: open a chat_turn span per prompt #29771 is the tracked PR that wires generation.go; the advisor model is still built through resolveModelCall, so its HTTP attempts produce provider_attempt spans under the enclosing context. Orchestrator confirmed the stacked branch scott/chatd-lifecycle/3-turn-span (5ffb40e) sets Stages: in generation.go and states the advisor exclusion at coderd/x/chatd/ARCHITECTURE.md:840. The closure depends on feat: open a chat_turn span per prompt #29771 merging with this PR.

CRF-5 (P3, stage_internal_test.go:128) - Duplicated stage test fixtures

  • Finding: chatloop/stage_internal_test.go repeats the fixture and metric helpers of chatloop/stage_test.go, and coderd/x/chatd/stage_internal_test.go has a third copy of the recorder setup. Proposed moving the exported-API test to chatloop_test and sharing helpers.
  • Author defense: The fixtures differ in package and in which clock they inject.
  • Panel closure (R2, 1/1): Netero, the agent that raised it (reused from R1), accepted on the package-boundary half only: sharing across chatloop and chatloop_test needs export_test.go, and the cost of leaving it is test-only (a fixture change made in two places). Netero found the clock half of the defense factually wrong at 3809e82: both fixtures now inject a quartz.Mock through WithClock (stage_test.go:43-45, stage_internal_test.go:138-140).

CRF-8 (P3, model_routing_aibridge.go:157) - Failed windows observed into stage histograms

  • Finding: Failed provider_attempt spans (4xx, 5xx, transport errors) and failed stream windows are observed into stage_duration_seconds and model_stage_duration_seconds, which have no outcome label, so a 429 burst pulls the percentiles down. Proposed EndWithoutObservation on those error paths.
  • Author defense: The failed time was spent and belongs in the stage profile; 5bb414f adds "Failed windows are observed" to both help texts. TTFT is the exception because a first token that never arrived has no duration.
  • Panel closure (R3, 4/4): Kite, Melody, Ryosuke, and Pen Botter, each the agent that raised it in R2 (reused via message), accepted. Shared reasoning: both metrics are declared a stage-time profile, so sum and rate need the failed time; both help texts at adeadee (metrics.go:174, metrics.go:181) state that failed windows are observed; success-only per-model first-token latency remains in coderd_chatd_ttft_seconds. The remaining gap (no outcome label) is CRF-33, out of scope for a human decision.

CRF-10 (P3, model_routing_aibridge.go:132) - provider_attempt spans the gateway call

  • Finding: A provider_attempt span times one call into the in-process AI Gateway until it relays its first upstream event, so gateway key-failover retries share one span. Proposed documenting the stage as the gateway call.
  • Author defense: provider_attempt is one round trip through the gateway as chatd sees it; upstream key rotation belongs to the gateway's own spans.
  • Panel closure (R3, 2/2): Pariston and Pen Botter, the agents that raised it (reused), accepted. Both traced that the in-process gateway uses the same coderAPI.TracerProvider (cli/aibridged.go:50), receives the request context carrying the provider_attempt span through inMemoryRoundTripper, and starts Intercept.ProcessRequest.Upstream spans from it per failover iteration (aibridge/intercept/messages/streaming.go:687), so upstream attempts appear as children of provider_attempt (inferred from code, not run). The type comment wording ("to the model provider") is carried by CRF-38. Bisky's CRF-43 notes that no test pins the context propagation this closure relies on.

CRF-12 (P3, chatloop.go:863) - Start markers close the TTFT window

  • Finding: text_start, reasoning_start, and tool_input_start close the TTFT window, so a start marker followed by an error is a successful first token. Proposed an allowlist of output-bearing parts.
  • Author defense: Start markers are the first streamed output; waiting for a delta would fold hidden reasoning time into TTFT. 5bb414f states the definition in the ttft_seconds help text.
  • Panel closure (R3, 1/1): Pariston, the agent that raised it (reused), accepted: under the definition the help text now states (metrics.go:167), a start marker is a real first-output event, so the observation is correct for that definition; the reasoning-model concern is the author's stated reason.

CRF-19 (P3, chatloop.go:830) - TTFT observed into multiple histograms

  • Finding: One first token was observed into three histograms, two duplicating coderd_chatd_ttft_seconds with coarser buckets. Proposed making the TTFT stage span-only.
  • Author defense: ced30e1 (feat: add chatd lifecycle stage tracer #29769) removed time_to_first_token from model_stage_duration_seconds; adeadee observes ttft_seconds from the stage's elapsed value, so the two remaining series measure one window.
  • Panel closure (R3, 1/1): Luffy, the agent that raised it (reused), accepted: no series duplicates ttft_seconds on its own labels; the remaining stage_duration_seconds{stage="time_to_first_token"} row gives the stage profile a TTFT entry with scope and chat_kind, which ttft_seconds lacks; the ttft_seconds help text names the overlap. The bucket mismatch remains but affects only the stage profile.

CRF-24 (Nit, chatd/stage_internal_test.go:73) - wantStatusCode naming

  • Finding: wantStatusCode holds the OTel span status next to an HTTP status attribute; wantAttribute was ambiguous. Proposed renames.
  • Author defense: The codes.Code type separates it from the HTTP status.
  • Panel closure (R3, 1/1): Gon, the agent that raised it (reused), accepted: e761a9d removed wantAttribute, and every table literal assigns codes.Unset or codes.Error, so the codes. prefix distinguishes the two where the value is set.

CRF-47 (P4, modelcall.go:89) - Stage identity derived twice

  • Finding: stageModel() returns an empty or filled wire Provider depending on whether the model is built yet, and newModel patches the transport's copy separately, so an early caller or an edit to one derivation would give stream/time_to_first_token spans a different provider attribute than provider_attempt spans; the PR description said the identity is resolved once. Proposed: have newModel return the identity it gave the transport, store it once on resolvedModelCall, and drop the Valid() branch.
  • Author defense: TestResolveModelCallStageModel asserts both stageModel() and the provider_attempt span attributes for the same call, and both take the wire provider from model.Provider(), so a divergence fails the test. The PR description no longer claims the identity is resolved once.
  • Panel closure (R4, 2/2): Meruem and Leorio, the R3 agents that raised it (reused via message), accepted. Meruem verified that the test builds one wantStageModel and checks both resolved.stageModel() and the span's provider, provider_type, model, and effort (modelcall_internal_test.go:178-195), so an edit to either derivation fails it; the remaining early-caller case affects only the span provider attribute, not any histogram (model_stage_duration_seconds has no provider label), and the stageModel doc says the wire provider is empty until the model is built. Leorio verified the description no longer says "once" and that its remaining wording is true at the resolveModelCall boundary, since newModel sets the transport's wire provider before any request.

Law analysis

R4 (first Law run for this PR): effective +1231 -274 (15 files) at head e6f36ac; Law counts about 169 hand-written production lines (the rest of the 418 is the generated prometheus.md table and generated_metrics). Verdict: Don't split. Enforcement: Advisory. Law found two separable slices (provider_attempt transport with StageModel and tracer wiring; stream/TTFT stages with the ttft_seconds redefinition) and judged that splitting duplicates the shared stage vocabulary without isolating a higher-risk piece.

Round log

Round 1

Netero-only. 2 P2, 3 P3, 1 Note. Panel skipped per Netero gate (P2 present). Orchestrator reproduced CRF-1 with a throwaway test (Warnings then Error, and Finish then Error, both observed as successful TTFT with span status Unset) and confirmed CRF-2 by grepping production GenerateAssistant callers (generation.go:754, chatadvisor/runner.go:54); the stacked branch scott/chatd-lifecycle/3-turn-span wires only generation.go. Reviewed against 76818c9..aacc28a.

Round 2

Tool v0.12.0. Stack base changed from 76818c9 to 5e32cbb (revised step 1 commit adding WithClock). Churn guard PROCEED: CRF-1, CRF-3, CRF-4, CRF-6 author fixed; CRF-2, CRF-5 contested. Dispute check-in: Netero (the R1 agent, reused via message) accepted both; CRF-2 closure depends on #29771, which the orchestrator fetched (pull/29771/head = 5ffb40e, stacked on 3809e82): it sets Stages: at generation.go:822 and documents the advisor exclusion at ARCHITECTURE.md:840. Netero R2 raised CRF-7 (incomplete CRF-6 fix in production wiring).

Round type: full panel. No prior panel round existed, so the whole diff was new to the panel (not a fix round). Panel (13): specialists ging-go, Ryosuke, Takumi, Melody, Killua, Luffy, Pen Botter (docs row for a changed operator-facing metric); core Pariston, Mafuuu; Bisky (tests changed); Gon and Leorio (once-per-PR floor, first panel round); wildcard Kite (random from distilled set). Size exceeds the focused target (6-8) because seven specialists triggered plus mandatory core, Bisky, Gon, Leorio.

Cross-check: 13 P3, 1 P4, 7 Nit posted; 4 Note (body); 3 out of scope; 5 dropped. Convergence: CRF-8 (4 reviewers, Kite reproduced a 429 observation), CRF-9 (4 reviewers, 3 reproduced the constant host), CRF-11 (Mafuuu and Takumi both reproduced). CRF-9 set to P3 by the tiebreaker (Ryosuke P3 vs three P4). CRF-10 set to P3 over Pen Botter's Note. Severity adjustments with the keep-argument written first: CRF-13, CRF-14, CRF-15, CRF-16 (Gon P2 on comment accuracy): keep-argument is that each comment states behavior the code does not have and no follow-up will correct it; set to P3 because the consequence is a misread by a code reader with no runtime effect, and Gon applied P2 uniformly to all 20 comment items without consequence reasoning. CRF-14 also over Leorio P4/Ryosuke Nit/Melody Note. Gon's verbosity items grouped into CRF-25, CRF-26, CRF-27 as Nits under the AGENTS.md rule that comments be substantive and concise; keep-argument for P2 is the same repo rule, rejected because verbosity has no consequence beyond reading time. Contradictions: Bisky's prior-findings remark that all non-error/warnings/finish parts are model output versus Pariston's reproduced start-marker case (CRF-12); Pariston's evidence (reproduction plus the help text's "content part" wording) kept CRF-12. Pariston's premise that the nil tracer keeps compaction out of turn accounting contradicts Gon/Leorio grep evidence (CRF-14). Leorio praised the comments Gon flagged at model_routing_aibridge.go:155/161 and chatloop.go:358, so those Nits were dropped (CRF-35 to CRF-37). Praise-only notes (Luffy x2, Kite, Leorio model_routing_aibridge.go:155) not recorded as findings. Killua: no findings. Reviewed against 5e32cbb..3809e82.

Round 3

Tool v0.13.0. Stack base changed from 5e32cbb to ced30e1 (revised #29769). New commits: 9026572 (rebased feature), 5bb414f, e761a9d, adeadee. Churn guard PROCEED: 17 addressed, 4 contested (CRF-8, CRF-10, CRF-12, CRF-24); orchestrator also treated CRF-19 as contested because the author kept two of the three series and defended the remainder (churn guard had classified it addressed). CRF-38 reopened: its R2 drop assumed CRF-10 would rewrite the same doc, but CRF-10 was contested. CRF-28 superseded (Netero: requireEndedBefore compares recorder end order). CRF-34 no longer applies (CI green).

Round type: restructure (full panel). StageSpan.End changed its exported signature to return time.Duration (adeadee), stage.go and stage_metrics_internal_test.go entered the PR diff, and adeadee rewrote how the stage identity is derived. Panel (14): specialists ging-go, Ryosuke, Takumi, Melody, Killua, Luffy, Pen Botter; core Pariston, Mafuuu; Bisky; Gon and Leorio (restructure); Kite (contested CRF-8, not counted toward size); wildcard Meruem (random from distilled set minus Mafuuu and Kite). Reused R2 agents via message for every contested persona: Kite, Melody, Ryosuke, Pen Botter, Pariston, Luffy, Gon. New agents for the rest. Netero reused (R1 agent).

Dispute check-in: all five contested findings closed by every persona that raised them (CRF-8 4/4, CRF-10 2/2, CRF-12 1/1, CRF-19 1/1, CRF-24 1/1). Gon re-raised CRF-38 at model_routing_aibridge.go:134; Pen Botter added that the doc's first sentence says the round trip goes "to the model provider" when it goes to the AI Gateway.

Cross-check: new 6 P4 and 3 Nit (CRF-41 to CRF-49), plus Netero's CRF-40 Nit and reopened CRF-38 Nit; 11 posted. Leorio's PR-description P4 folded into Meruem's CRF-47 (same root: stage identity derived in two places; Meruem's fix makes the description true). Meruem's consequence claim that model_stage_duration_seconds series would carry differing provider values is wrong: that histogram's labels are stage, provider_type, chat_kind, model (metrics.go), so the divergence would show only in span provider attributes; posted with the corrected consequence. Praise or verification-only notes (Kite, Luffy chatloop.go:816, Bisky stage_internal_test.go:471, Leorio model_routing_aibridge.go:300 and metrics.go:167) not recorded as findings. No findings: ging-go, Takumi (race runs clean), Killua, Mafuuu, Kite, Melody, Ryosuke, Pen Botter, Pariston, Luffy. Reviewed against ced30e1..adeadee.

Round 4

Tool v0.13.0. Stack base changed from ced30e1 to b7d8353 (revised #29769, drops unused stage tracer API). Round 3 commits rebased (9c79842, 807436b, 5e26b8d, 4b47b43; range-diff shows only base-driven hunk changes). New commit: e6f36ac. Churn guard PROCEED: 10 addressed (CRF-38, CRF-40 to CRF-46, CRF-48, CRF-49), 1 contested (CRF-47). Orchestrator also marked Note (body) items CRF-29 (stream span now errored for content-filter/no-output), CRF-30 (title names the TTFT change), and CRF-31 (description states the #29771 dependency) as author fixed, and CRF-32 resolved in the base (Ryosuke). Inventory validation: dropped CRF-35 to CRF-37 and CRF-39 re-checked; e6f36ac extends the chatloop.go:358 ordering comment (CRF-37) but the drop's reason still holds. CRF-2 dependency re-checked: #29771 (pull/29771/head 282311c) is stacked on e6f36ac and sets Stages: at generation.go:806 and generation.go:1003.

Law ran for the first time (effective additions 1231 > 1000 and no prior Law analysis; round 3 at 1146 met the same condition and Law did not run). Verdict Don't split, advisory. Netero (new agent) raised CRF-50.

Round type: fix round. No files added, renamed, or moved (subagent_internal_test.go is an existing file newly touched); no exported signature changed (GenerateAssistant only named its results); no function rewritten; the title change names the existing TTFT behavior change (CRF-30) rather than a new goal; effective additions grew 85 since round 3. Panel (10): Meruem and Leorio (contested CRF-47, reused R3 agents via message); Gon, Bisky, Pen Botter (raised open findings the new commit touches: CRF-38, CRF-41, CRF-42; CRF-43 to CRF-45; CRF-38 co-raiser); specialists for the changed files ging-go (.go), Ryosuke (Go code), Takumi (the deferred attempt.release cancels the attempt context and disarms the silence timer), Melody (stream span lifecycle pairing); wildcard Knov (random from distilled set minus Meruem). Gon and Leorio also met the floor (comments and doc comments touched).

Dispute check-in: CRF-47 closed 2/2 (Meruem, Leorio). Meruem withdrew its round 3 claim that metric series would diverge.

Cross-check: 2 P4 posted (CRF-50, CRF-51). CRF-51 combines Pen Botter's P4 (description says the stream span records the returned error; lines 351 and 375 record the unwrapped error, pinned by StreamErrorEndsStreamWithError) with Gon's comment item at chatloop.go:361 (the sentence states the split without a reason): one root, the unexplained unwrapped-versus-returned split. Orchestrator verified wrapProviderStreamError prefixes "stream response:" and may join ErrProviderTransportReset, so the span message is a substring of the returned text; the posted consequence is the description mismatch and the unexplained split, not failed trace search. Severity: keep-argument for Gon's P2 is the AGENTS.md MUST rule on substantive comments and Gon's observation that comment quality recurs in fix commits; set to P4 with Pen Botter because the consequence is a misdescription with no runtime effect, matching CRF-46. Contradiction: Knov's note says the span ends with the error GenerateAssistant returns on every path; Melody and Pen Botter trace the unwrapped error on the finish-error path, and the test assertion confirms them. Leorio's note found the chatloop.go:361 comment accurate; Gon's item concerns the missing reason, which Leorio did not address. Dropped CRF-52 (keep-argument: the first sentence repeats StageSpan.End's doc and lines 839-840, against GO.md's "explain why, not what"; dropped because it states End's stage-histogram side effect at the call site, the premise of the CRF-19 closure, and the second sentence is the wording this review proposed for CRF-48). Dropped CRF-53 (keep-argument: seen does not describe its value, so the field and the assertion each need a comment, against GO.md's self-documenting-names rule; dropped because the assertion comment names the property under test, which the proposed name requestSpanContext would not, and the field type *trace.SpanContext says what it holds). Praise or verification-only notes (Leorio chatloop.go:361, Bisky mutation results, Knov chatloop.go:365 and model_routing_aibridge.go:143) not recorded as findings. No findings: Meruem, Leorio, Bisky, ging-go, Ryosuke, Takumi, Melody, Knov. Reviewed against b7d8353..e6f36ac.

About deep-review

CRF = Coder Review Finding (P0-P4, Nit, Note)

Reviewer Focus
Bisky tests
Chopper ops/errors
Churn-guard change verification
Ging language modernization
Gon naming
Hisoka edge cases
Killua perf
Kite change integrity
Knov contracts
Knuckle SQL
Komugi flake/determinism
Kurapika security
Law decomposition
Leorio docs
Luffy product
Mafu-san process
Mafuuu contracts
Melody dispatch/pairing
Meruem structural
Nami frontend
Netero mechanical checks
Pariston premise testing
Pen-botter product gaps
Razor verification
Robin duplication
Ryosuke Go arch
Takumi concurrency
Zoro shape

🤖 Managed by Coder Agents.

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First-pass review only. These are mechanical findings from Netero. The full review panel has not reviewed this PR yet and will review after these findings are addressed.

Closing the TTFT window through one sync.Once is a clean way to keep both histograms success-only, and TestNewModelTracesEachProviderAttempt drives a 429 and then a 200 through the SDK client that newModel builds, so the one-span-per-attempt claim is tested end to end rather than against a stub. TestResolveModelCallResolvedEffort also checks the clamped effort in the provider options, not only on the span.

2 P2, 3 P3, 1 Note. CRF-1 was reproduced with a throwaway test against aacc28a: a warnings part followed by an error part is observed as a successful first token on both histograms. CRF-2 needs a decision on the advisor runner, which no PR in the stack wires.

"The result is correct only because ResolveReasoningEffort is idempotent on its own output." (Netero)

🤖 This review was automatically generated with Coder Agents.

Comment thread coderd/x/chatd/chatloop/chatloop.go Resolved
Comment thread coderd/x/chatd/chatloop/chatloop.go Resolved
Comment thread coderd/x/chatd/chatloop/chatloop.go Resolved
Comment thread coderd/x/chatd/modelcall.go Resolved
Comment thread coderd/x/chatd/chatloop/chatloop.go Resolved
@jscottmiller
jscottmiller force-pushed the scott/chatd-lifecycle/2-provider-stages branch 2 times, most recently from 71d3e86 to 3809e82 Compare September 23, 2026 20:27
@jscottmiller
jscottmiller removed this pull request from stack #29773 September 23, 2026 21:05
@linear-code

linear-code Bot commented Sep 24, 2026

Copy link
Copy Markdown

CODAGT-1102

@jscottmiller

Copy link
Copy Markdown
Contributor Author

/coder-agents-review

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR adds stream, time_to_first_token, and provider_attempt stages around the chatd provider call and narrows what closes the coderd_chatd_ttft_seconds window. This round has 13 P3, 1 P4, and 7 Nit findings.

The warnings and finish handling, the endStream ordering tests, the single effort input to ProviderOptionsForCall, and the shared mock clock in the test fixtures are in 3809e82. The production chatd.New still builds the tracer without the server clock (comment on chatd.go:2996). The reviewer that raised the unwired Stages fields and the duplicated test fixtures accepted both replies; the first depends on #29771 merging with this PR, which sets Stages in generation.go and documents the advisor exclusion in ARCHITECTURE.md.

Out of scope (needs a ticket or explicit acceptance by a human):

  • coderd/x/chatd/chatloop/stage.go:151: the StageModel.Effort doc says the effort is "sent to the provider", but applyReasoningEffort maps or drops it for some models, so every stage that carries reasoning_effort inherits a wrong definition.
  • coderd/x/chatd/chatloop/metrics.go:175: stage_duration_seconds and model_stage_duration_seconds have no outcome label, so no query can separate failed windows from completed ones for any stage that observes failures.
  • CI: the Storybook and Analyze PR for Documentation Updates Needed jobs fail on this head although the PR changes no site/ files; the cause is unverified because the job logs could not be read.

Notes:

  • coderd/x/chatd/chatloop/stage_internal_test.go:468: the ordering assertion compares OTel span end times, which come from the wall clock rather than the fixture's mock clock, so a reorder that ends both spans on the same tick still passes.
  • coderd/x/chatd/chatloop/chatloop.go:378: endStream(nil) runs before the content-filter and no-output checks, so the stream span reads success for attempts that GenerateAssistant then returns as errors.
  • coderd/x/chatd/chatloop/metrics.go:166: coderd_chatd_ttft_seconds has shipped since v2.33.0 and the PR title does not mention its changed meaning, so release notes built from the title will not explain the step in the series.
  • PR description: it does not say that without #29771 production emits only provider_attempt stages.

🤖 This review was automatically generated with Coder Agents.

Comment thread coderd/x/chatd/chatd.go Resolved
Comment thread coderd/x/chatd/chatloop/chatloop.go Resolved
Comment thread coderd/x/chatd/stage_internal_test.go Resolved
Comment thread coderd/x/chatd/chatloop/chatloop.go Resolved
Comment thread coderd/x/chatd/stage_internal_test.go Resolved
@jscottmiller
jscottmiller force-pushed the scott/chatd-lifecycle/2-provider-stages branch from 3809e82 to adeadee Compare September 25, 2026 05:04
@jscottmiller
jscottmiller removed this pull request from stack #29806 September 25, 2026 05:05
@jscottmiller

Copy link
Copy Markdown
Contributor Author

/coder-agents-review

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR adds stream, time_to_first_token, and provider_attempt stages around the chatd provider call and changes what closes the coderd_chatd_ttft_seconds window. This round has 6 P4 and 5 Nit findings.

The round 2 fixes in 5bb414f and adeadee are in place and covered by tests, including the server clock on the tracer, the silence-timeout error on the TTFT span, and the end-to-end identity test with the copilot row. The reviewers that raised failed-window observation, the gateway-level provider_attempt span, start markers closing the TTFT window, the two TTFT series, and the wantStatusCode name accepted the replies. The stageSpanRoundTripper doc comment is raised again because the doc was not rewritten. Accepting the unwired Stages fields still depends on #29771 merging with this PR.

Out of scope (needs a ticket or explicit acceptance by a human):

  • coderd/x/chatd/chatloop/stage.go:149: the StageModel.Effort doc says the effort is "sent to the provider", while the resolvedEffort comment in modelcall.go now says it is the value before provider mapping, so the two comments contradict each other and span readers get the wrong definition.
  • coderd/x/chatd/chatloop/metrics.go:174: stage_duration_seconds and model_stage_duration_seconds have no outcome label, so no query can separate failed stream and provider_attempt windows from completed ones now that both are observed.

Notes:

  • coderd/x/chatd/chatloop/chatloop.go:380: endStream(nil) runs before the content-filter and no-output checks, so the stream span reads success for attempts that GenerateAssistant then returns as errors.
  • coderd/x/chatd/chatloop/metrics.go:167: coderd_chatd_ttft_seconds has shipped since v2.33.0 and the PR title does not mention its changed meaning, so release notes built from the title will not explain the step in the series.
  • PR description: it does not say that without #29771 production emits only provider_attempt stages.

🤖 This review was automatically generated with Coder Agents.

Comment thread coderd/x/chatd/modelcall.go Resolved
Comment thread coderd/x/chatd/chatloop/stage.go Resolved
Comment thread coderd/x/chatd/modelcall.go Resolved
Comment thread coderd/x/chatd/chatloop/chatloop.go Resolved
Comment thread coderd/x/chatd/chatloop/chatloop.go Resolved
@jscottmiller
jscottmiller force-pushed the scott/chatd-lifecycle/2-provider-stages branch from adeadee to e6f36ac Compare September 25, 2026 17:24
@jscottmiller jscottmiller changed the title feat: trace chatd provider stream and attempt stages feat: trace chatd provider stages and count only output parts as first token Sep 25, 2026
@jscottmiller

Copy link
Copy Markdown
Contributor Author

/coder-agents-review

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR adds stream, time_to_first_token, and provider_attempt stages around the chatd provider call and changes what closes the coderd_chatd_ttft_seconds window. This round has 2 P4 findings.

The round 3 fixes in e6f36ac are in place: the stream span now ends from one deferred call after the attempt is released, content-filter and no-output attempts end it errored, and the new tests fail when the span context or the tracer provider is dropped. The title and description now name the TTFT change and the #29771 dependency. The reviewers that raised the two-place stage identity derivation accepted the reply. Accepting the unwired Stages fields still depends on #29771 merging with this PR.

Out of scope (needs a ticket or explicit acceptance by a human):

  • coderd/x/chatd/chatloop/metrics.go:174: stage_duration_seconds and model_stage_duration_seconds have no outcome label, so no query can separate failed stream and provider_attempt windows from completed ones now that both are observed.

🤖 This review was automatically generated with Coder Agents.

Comment thread coderd/x/chatd/chatloop/chatloop.go Resolved
@jscottmiller
jscottmiller force-pushed the scott/chatd-lifecycle/2-provider-stages branch 3 times, most recently from 0510e54 to dc303a5 Compare September 29, 2026 01:35
@jscottmiller
jscottmiller removed this pull request from stack #29877 September 29, 2026 03:26
@jscottmiller
jscottmiller force-pushed the scott/chatd-lifecycle/2-provider-stages branch from dc303a5 to a434ea5 Compare September 30, 2026 01:02
Scott Miller added 7 commits September 30, 2026 13:40
… once

- Stamp the wire provider from StageModel on stream, time_to_first_token,
  and provider_attempt stages instead of setting it by hand.
- Observe coderd_chatd_ttft_seconds with the elapsed time the
  time_to_first_token stage records, so both measure one window.
- Regenerate metric docs.
@jscottmiller
jscottmiller force-pushed the scott/chatd-lifecycle/2-provider-stages branch from a434ea5 to 99a939c Compare September 30, 2026 14:02
@jscottmiller
jscottmiller merged commit e988ddb into main Sep 30, 2026
32 checks passed
@jscottmiller
jscottmiller deleted the scott/chatd-lifecycle/2-provider-stages branch September 30, 2026 16:54
jscottmiller pushed a commit that referenced this pull request Sep 30, 2026
**TL;DR:** Wraps each prompt in a `chat_turn` root span with every
lifecycle stage parented under it. The span records one outcome per turn
(`completed`, `interrupted`, `error`, `abandoned`), and the new
`coderd_chatd_turn_outcomes_total` counter counts those outcomes. Turns
stay correct when tasks are canceled, replaced, or retried.

Each prompt runs inside a `chat_turn` root span owned by the runner
(`runnerTurnSpan`). The generation step, prepare, MCP connect, stream,
TTFT, provider attempt, commit, thinking, tool call, compaction, and
retry backoff stages are parented under it, activating the stages from
#29770. `chat_turn` spans are standalone roots: the turn runs on a
worker, often on another replica, and no trace context is persisted.

**Lifecycle**

- **Start.** Each generation loop iteration calls `Ensure`, which opens
the turn if none is open and returns a `turnToken`. The span is
backdated to the trigger: the latest of the last user prompt, a dynamic
tool's result message after it, and a pending compaction request. An
`acquisition` stage covers trigger to pickup.
- **Stale trigger / takeover.** A trigger at or before the previous
anchor opens the turn at now with no `acquisition` and counts
`stale_anchor`. A runner that took the chat over from an owner with a
stale heartbeat (`TakenOver`, from acquisition) opens its first turn at
now with no `acquisition`, so the dead owner's run is not reported as
pickup delay.
- **Newer prompt.** `Ensure` on the current turn with a newer trigger
closes it and opens a new one. The closed turn is `abandoned` unless the
task that ran it records an outcome before releasing it.
- **Finish.** After `FinishTurn` or `EnterRequiresAction` commits, the
step calls `Complete`; `Settle` closes the span after the step's own
stage ends, so the finishing step is inside the turn. A chat waiting in
`requires_action` is outside any turn; the generation resumed by its
tool results opens a new one.
- **Emission.** Closing a turn fixes its end time; its span is emitted
once every task that joined it (`Ensure` takes the task ID) has called
`Release`, which `runTask` does after the task returns. A finishing task
whose `FinishTurn` promoted a queued prompt therefore still counts its
turn as `completed` even when the promoted task's `Ensure` runs before
its `Complete`. `End` at runner exit emits whatever is left.
- **Failure.** A step that commits `FinishError`, or fails closed,
invalidates the turn as `error` after the commit. Errors the task runner
retries, including task timeouts, keep the same turn open; an expected,
non-retryable exit records no outcome and leaves the turn open until the
interrupt task, a newer prompt, or runner exit closes it. A canceled
generation task records no outcome and cannot obtain a turn token, so a
replaced task never touches its successor's turn.
- **Interrupt.** The interrupt task marks the open turn `interrupted`
before `FinishInterruption` commits and closes it after. Only a stop (by
the user or a parent agent) is `interrupted`; it ends with unset span
status, since error status is reserved for `error` turns. runner
shutdown, loss of ownership, and message edits close the turn as
`abandoned` with unset status.
- **Promotion.** Every promoting transition (`FinishTurn`,
`FinishInterruption`, a send to an errored chat, `PromoteQueued`)
returns the promoted message's queued time in its result, and one
`recordQueueWait` helper records a standalone `queue_wait` to the
promotion. The promoted turn opens at the promotion time.

`chat_turn` is observed on `coderd_chatd_stage_duration_seconds` only
for completed turns, so the histogram is completed-turn latency; every
turn still emits its span.

`coderd_chatd_turn_outcomes_total{outcome, chat_kind}` counts every
closed turn exactly once (`completed`, `interrupted`, `error`,
`abandoned`), matching the span's `turn_outcome`, under the
`chat-stage-metrics` experiment. Turn time accounting is in draft
#29772.

**Stages and identity**

- `tool_call` is a live stage around each local tool call in `chatloop`;
the tool runs on its context, and it ends errored only when the tool
fails to execute. Calls that never run (inactive, unknown, or rejected
by policy) record no stage.
- `mcp_connect` carries connected and failed server counts and ends
errored only when servers were configured and none connected.
- The compaction stage carries the model that runs the summary.
- `withStageIdentity` puts scope and chat kind on the turn context.
Detached background work runs in `background` scope as its own trace
roots.
- The `advisor` tool's nested model call appears only as its enclosing
`tool_call`.

ARCHITECTURE.md gains a "Lifecycle tracing" section.

<details>
<summary>Review decisions</summary>

- Turn identity stays timestamp-based with targeted fixes (clamped
acquisition, rotation on a newer trigger, invalidate after commit,
fail-closed invalidation, settle closing invalidated turns) plus a
takeover flag; keying turns on message IDs would not detect takeover and
cannot express interrupted/abandoned outcomes.
- Outcome classification: the interrupt task owns `interrupted`, and
cancellation or a fence mismatch alone never records an outcome, so
shutdown, ownership loss, and edits are `abandoned`. Only `error` turns
set error span status. Remaining ordering races between a step and an
interrupt are documented as known limitations in ARCHITECTURE.md.
- Turns are emitted after the tasks that ran them return, not when they
close, so an outcome a replaced task records while unwinding reaches its
own turn; the successor task is not delayed. Chosen over an
after-commit, pre-publish hook on `ChatMachine.Update`, which would fix
only the `FinishTurn` ordering.
- `requires_action` ends the turn rather than pausing it, so `chat_turn`
measures chatd time; a prompt with client tools counts as several turns.
- Promoted turns anchor at promotion time and `queue_wait` is always
standalone, so `chat_turn` never includes time spent queued behind
another turn.
- The former `capacity_wait` stage (#29805) was dropped: it was a
per-worker lower bound, excluded from accounting, and needed its own SQL
and bookkeeping. #29805's remaining helpers are folded into this PR.

</details>

Supersedes #28972, #28973, and #29805.

🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 30, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants