Skip to content

feat: report pubsub health from the pubsub backend - #31621

Merged
spikecurtis merged 5 commits into
spike/gru-100-pubsub-healthsdkfrom
spike/gru-100-pubsub-health-report
Nov 13, 2026
Merged

spikecurtis merged 5 commits into
spike/gru-100-pubsub-healthsdkfrom
spike/gru-100-pubsub-health-report

Conversation

@spikecurtis

@spikecurtis spikecurtis commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Fills in the pubsub section of the deployment health report on the server. Stacked on #31620.

  • New pubsub.ReportingPubsub interface: Pubsub plus HealthReport(ctx) healthsdk.PubsubReport. The Pubsub interface itself is unchanged.
  • The PostgreSQL, NATS and in-memory backends implement it. Each reports its backend name (postgres, nats or memory) and its connection state:
    • PostgreSQL: tracks the pq listener's connected state from its event callback. When it's not connected, the report has severity warning and EPS01.
    • NATS: warns with EPS02 if the embedded server isn't Running(), and with EPS01 for each publish or subscribe connection whose Status() isn't CONNECTED.
    • In-memory: always ok.
  • coderd.Options.Pubsub is now a ReportingPubsub. ReplicaSyncPubsub stays a plain Pubsub.
  • coderd/healthcheck has a new Pubsub check, which counts toward healthy and the overall severity like the other sections. formatHealthcheck applies the dismissed setting for it.
  • New health codes EPS01 (not connected) and EPS02 (embedded NATS server not running), documented in docs/admin/monitoring/health-check.md.
  • Test helpers follow the type change: coderdtest.Options.Pubsub and the pubsub returned by dbtestutil.NewDB are now ReportingPubsub, and a few test wrappers embed ReportingPubsub instead of Pubsub.
Decision log
  • HealthReport returns a healthsdk.PubsubReport rather than filling one in place. This matches the healthcheck.Checker methods. ctx is unused for now.
  • The healthcheck layer sets Dismissed, so backends don't need to know about health settings.
  • Problems are reported as warnings, not errors, so they show on the health page without marking the deployment unhealthy. Both backends reconnect on their own.
  • Postgres connection state comes from the listener event callback, not from pq.Listener.Ping, so the health check doesn't do database I/O. Closing the pubsub also marks it disconnected.
  • NATS conn gains Status(), which *natsgo.Conn already implements. The pools are immutable after construction, so HealthReport reads them without locking.
  • The docs say the pubsub section appears in the JSON response, because the health UI doesn't show it until a later PR in this stack.
  • psmock is not regenerated: none of its users pass it to coderd.

Generated by Coder Agents on behalf of @spikecurtis.

@spikecurtis
spikecurtis added this pull request to stack #31622 November 2, 2026 13:38
@linear-code

linear-code Bot commented Nov 2, 2026

Copy link
Copy Markdown

GRU-100

@spikecurtis
spikecurtis force-pushed the spike/gru-100-pubsub-health-report branch 2 times, most recently from 029ab7b to e8610d6 Compare November 5, 2026 14:49
@spikecurtis
spikecurtis marked this pull request as ready for review November 5, 2026 14:58
@spikecurtis
spikecurtis requested a review from cstyan November 5, 2026 14:58
@spikecurtis
spikecurtis force-pushed the spike/gru-100-pubsub-health-report branch from daf3b10 to 6656bfb Compare November 12, 2026 13:09
bpmct added a commit that referenced this pull request Nov 12, 2026
…the wait (#32097)

The doc-check chat keeps running in Coder after the job stops waiting at
40 minutes, and it posts its own comment when it is done. When the local
model is busy, reviews run past the wait and the check goes red even
though nothing failed. 33 of 46 doc-check runs today ended red this way,
including on #32088, #31788, #31621 and #31620.

## Changes

- `continue-on-error` on the agents-chat-action step.
- New step that ends with a notice when the wait timed out and the chat
is still `running` or `pending`, and fails the job for every other
error.

Generated by Coder Agents on behalf of @bpmct.

@cstyan cstyan left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

I think this PR is where we could provide different backend name mappings if we chose to, or in the frontend PR. AFAICT the definition of the BackendMemory value just being short/lower case is to match the existing BackendPostgres/BackendNATS values that are used as metric label values.

@coderagents coderagents Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documentation check (doc-check skill) on the new Pubsub section. 3 findings; see inline comments and the summary below.

## Pubsub

Coder uses a publish-subscribe (pubsub) system to send events between its components and replicas.
The health check reports which pubsub backend the Coder replica uses, either `nats` for the embedded NATS server or `postgres` for PostgreSQL, and whether that backend is connected.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: The Pubsub section never tells the reader that EPS01/EPS02 are always warning severity, never error. The reader has no way to judge urgency without already knowing Coder's severity model, which this new section doesn't explain. (Newcomer)

Evidence: coderd/database/pubsub/pubsub.go:397-413 and coderd/nats/pubsub.go:478-509 set only SeverityWarning for these codes; coderd/healthcheck/healthcheck.go:315-319 only counts a section as failing when Severity > SeverityWarning, so these codes can never make the deployment report unhealthy.

Suggested change
The health check reports which pubsub backend the Coder replica uses, either `nats` for the embedded NATS server or `postgres` for PostgreSQL, and whether that backend is connected.
The health check reports which pubsub backend the Coder replica uses, either `nats` for the embedded NATS server or `postgres` for PostgreSQL, and whether that backend is connected.
Pubsub issues are reported as warnings, not errors, so they don't make the deployment unhealthy.

While the replica is disconnected, it misses events, so features that depend on them, such as workspace build updates, can be delayed.

**Solution:**
Coder tries to reconnect automatically, so this might be a transient issue.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: The pronoun "this" has no clear single referent in this sentence; the reader must guess whether it means the reconnection attempt or the disconnection itself. Per docs/.style/style-guide/voice-and-tone.md#give-each-pronoun-one-clear-referent, name the thing a bare "this" refers to instead of letting the reader infer it. (Copy Editor)

Suggested change
Coder tries to reconnect automatically, so this might be a transient issue.
Coder tries to reconnect automatically, so the disconnection might be transient.

@coderagents

coderagents Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

Documentation Check

Blocking

  • docs/admin/monitoring/health-check.md:11 - The intro sentence ("The deployment health page is broken up into the following sections:") frames every ## heading, including the new ## Pubsub section, as part of the UI dashboard page. Pubsub is the only section that doesn't render there (site/src/pages/HealthPage/HealthLayout.tsx:45-52 lists derp, access_url, websocket, database, workspace_proxy, provisioner_daemons, no pubsub). The Pubsub section does flag itself as JSON-only at line 333, but a reader following the page-level intro first expects to find it on the dashboard. Fix: adjust the intro, e.g. "The deployment health page and its JSON response are broken up into the following sections:" (Cartographer)
  • docs/admin/monitoring/health-check.md:332 - Missing: the section never states EPS01/EPS02 are always warning severity and never make the deployment unhealthy. See inline comment. (Newcomer)
  • docs/admin/monitoring/health-check.md:346 - Rule: bare pronoun "this" lacks a clear referent. See inline comment. (Copy Editor)

Related open PRs

Findings for agents
- id: coverage-1
  persona: coverage
  class: wrong
  blocking: true
  file: docs/admin/monitoring/health-check.md
  line: 11
  quote: "The deployment health page is broken up into the following sections:"
  evidence:
    - site/src/pages/HealthPage/HealthLayout.tsx:45-52
    - docs/admin/monitoring/health-check.md:329-333
  rule: null
  problem: The intro claims the UI dashboard page is broken into "the following sections," and treats the new Pubsub heading as one of those siblings, but Pubsub never renders on the dashboard.
  fix: "The deployment health page and its JSON response are broken up into the following sections:"
  verified: true
- id: learner-1
  persona: learner
  class: missing
  blocking: true
  file: docs/admin/monitoring/health-check.md
  line: 331
  quote: "Coder uses a publish-subscribe (pubsub) system to send events between its components and replicas."
  evidence:
    - coderd/database/pubsub/pubsub.go:397-413
    - coderd/nats/pubsub.go:478-509
    - coderd/healthcheck/healthcheck.go:315-319
    - coderd/debug.go:138-139
  rule: null
  problem: The section never states EPS01/EPS02 are always warning severity and never make the deployment unhealthy.
  fix: "Pubsub issues are reported as warnings, not errors, so they don't make the deployment unhealthy."
  verified: true
- id: language-1
  persona: language
  class: rule
  blocking: true
  file: docs/admin/monitoring/health-check.md
  line: 346
  quote: "Coder tries to reconnect automatically, so this might be a transient issue."
  evidence: []
  rule: docs/.style/style-guide/voice-and-tone.md#give-each-pronoun-one-clear-referent
  problem: Bare "this" has no single clear antecedent noun.
  fix: "Coder tries to reconnect automatically, so the disconnection might be transient."
  verified: true
- id: learner-2
  persona: learner
  class: suggestion
  blocking: false
  file: docs/admin/monitoring/health-check.md
  line: 332
  quote: "The health check reports which pubsub backend the Coder replica uses, either `nats` for the embedded NATS server or `postgres` for PostgreSQL, and whether that backend is connected."
  evidence:
    - coderd/database/pubsub/metrics.go
    - coderd/database/pubsub/pubsub_memory.go
  rule: null
  problem: Omits the test-only `memory` backend value.
  fix: No change needed.
  verified: false
  rejected_reason: "Evidence confirms memory is never reachable through the production health-check path; the docs are accurate for every real reader. Rejected at checks 2 and 5."

Automated review via Coder Agents. Reply to a finding's thread with the reason if the finding is wrong.

@spikecurtis
spikecurtis force-pushed the spike/gru-100-pubsub-health-report branch from 6656bfb to 2769ec2 Compare November 13, 2026 15:43
@spikecurtis
spikecurtis temporarily deployed to docs-preview November 13, 2026 15:47 — with GitHub Actions Inactive
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Docs preview

Links open this pull request's preview at https://pr-31621.docs.cdr-preview.dev, which updates with each push.

Check off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here.

Preview sites for each built commit (2)

Every built commit keeps its own preview, newest first. Previews aren't indexed by search engines and are deleted when this pull request is merged or closed.

Adds pubsub.ReportingPubsub, which extends Pubsub with a HealthReport
method. The PostgreSQL, NATS and in-memory backends implement it and
report their backend name. coderd.Options.Pubsub now takes a
ReportingPubsub so the health check can include a pubsub section, and
the section can be dismissed like the others.
PGPubsub tracks whether its listener is connected to PostgreSQL and
warns when it isn't. The NATS pubsub warns when its embedded server is
not running or any publish or subscribe connection is not connected.
Adds health codes EPS01 and EPS02 with docs.
Restores import grouping and an unused receiver in coderdtest, drops a
now-unneeded variable in the prebuilds test, and gives the NATS
unhealthy test a real (unstarted) embedded server now that
HealthReport no longer guards against a nil server.
@spikecurtis
spikecurtis force-pushed the spike/gru-100-pubsub-health-report branch from 2769ec2 to 7c2990c Compare November 13, 2026 15:53
@spikecurtis
spikecurtis temporarily deployed to docs-preview November 13, 2026 15:57 — with GitHub Actions Inactive
@spikecurtis
spikecurtis merged commit 4f6a564 into main Nov 13, 2026
42 of 48 checks passed
@spikecurtis
spikecurtis deleted the spike/gru-100-pubsub-health-report branch November 13, 2026 16:20
@spikecurtis
spikecurtis temporarily deployed to docs-preview November 13, 2026 16:20 — with GitHub Actions Inactive
spikecurtis added a commit that referenced this pull request Nov 13, 2026
Stacked on #31621.

Adds a **Pubsub** section to the deployment health page
(`/health/pubsub`). It shows:

- the pubsub severity dot and the mute-warnings button,
- the report error, if one is set,
- each warning, with a link to its docs (EPS01, EPS02),
- the backend in use (`postgres` or `nats`).

The section is added to the health sidebar. Storybook stories cover the
default, warning (NATS), and error states.

<details>
<summary>Plan and decisions</summary>

- The page follows the same pattern as `DatabasePage`: it reads the
report from the outlet context and renders warnings as dismissible
Alerts with `HealthMessageDocsLink`.
- Warning keys use `code-message` instead of `code`. The NATS backend
can report the same code (EPS01) once for each publish and subscribe
connection, so a key on `code` alone could repeat.
- The page shows `pubsub.error` as an error Alert, because the server
sets `error` when the report cannot be produced.
- If the backend is empty, the page shows "Unknown".
- The sidebar entry comes from the existing `sections` map in
`HealthLayout`, so the severity dot and the dismissed state work the
same way as in other sections.
- Stories: `Pubsub` (default MockHealth), `WithWarnings` (NATS with
EPS02 and two EPS01 warnings), `WithError`.

</details>

Generated by Coder Agents on behalf of @spikecurtis.

This branch was previously deployed

1 inactive deployment
docs-preview — 7c2990c9 Deployed Nov 13, 2026 by spikecurtis via cleanup #40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants