Repository navigation
feat: report pubsub health from the pubsub backend - #31621
spikecurtis merged 5 commits into
Conversation
029ab7b to
e8610d6
Compare
daf3b10 to
6656bfb
Compare
…the wait (#32097) The doc-check chat keeps running in Coder after the job stops waiting at 40 minutes, and it posts its own comment when it is done. When the local model is busy, reviews run past the wait and the check goes red even though nothing failed. 33 of 46 doc-check runs today ended red this way, including on #32088, #31788, #31621 and #31620. ## Changes - `continue-on-error` on the agents-chat-action step. - New step that ends with a notice when the wait timed out and the chat is still `running` or `pending`, and fails the job for every other error. Generated by Coder Agents on behalf of @bpmct.
cstyan
left a comment
There was a problem hiding this comment.
LGTM 👍
I think this PR is where we could provide different backend name mappings if we chose to, or in the frontend PR. AFAICT the definition of the BackendMemory value just being short/lower case is to match the existing BackendPostgres/BackendNATS values that are used as metric label values.
| ## Pubsub | ||
|
|
||
| Coder uses a publish-subscribe (pubsub) system to send events between its components and replicas. | ||
| The health check reports which pubsub backend the Coder replica uses, either `nats` for the embedded NATS server or `postgres` for PostgreSQL, and whether that backend is connected. |
There was a problem hiding this comment.
Blocking: The Pubsub section never tells the reader that EPS01/EPS02 are always warning severity, never error. The reader has no way to judge urgency without already knowing Coder's severity model, which this new section doesn't explain. (Newcomer)
Evidence: coderd/database/pubsub/pubsub.go:397-413 and coderd/nats/pubsub.go:478-509 set only SeverityWarning for these codes; coderd/healthcheck/healthcheck.go:315-319 only counts a section as failing when Severity > SeverityWarning, so these codes can never make the deployment report unhealthy.
| The health check reports which pubsub backend the Coder replica uses, either `nats` for the embedded NATS server or `postgres` for PostgreSQL, and whether that backend is connected. | |
| The health check reports which pubsub backend the Coder replica uses, either `nats` for the embedded NATS server or `postgres` for PostgreSQL, and whether that backend is connected. | |
| Pubsub issues are reported as warnings, not errors, so they don't make the deployment unhealthy. |
| While the replica is disconnected, it misses events, so features that depend on them, such as workspace build updates, can be delayed. | ||
|
|
||
| **Solution:** | ||
| Coder tries to reconnect automatically, so this might be a transient issue. |
There was a problem hiding this comment.
Blocking: The pronoun "this" has no clear single referent in this sentence; the reader must guess whether it means the reconnection attempt or the disconnection itself. Per docs/.style/style-guide/voice-and-tone.md#give-each-pronoun-one-clear-referent, name the thing a bare "this" refers to instead of letting the reader infer it. (Copy Editor)
| Coder tries to reconnect automatically, so this might be a transient issue. | |
| Coder tries to reconnect automatically, so the disconnection might be transient. |
Documentation CheckBlocking
Related open PRs
Findings for agents- id: coverage-1
persona: coverage
class: wrong
blocking: true
file: docs/admin/monitoring/health-check.md
line: 11
quote: "The deployment health page is broken up into the following sections:"
evidence:
- site/src/pages/HealthPage/HealthLayout.tsx:45-52
- docs/admin/monitoring/health-check.md:329-333
rule: null
problem: The intro claims the UI dashboard page is broken into "the following sections," and treats the new Pubsub heading as one of those siblings, but Pubsub never renders on the dashboard.
fix: "The deployment health page and its JSON response are broken up into the following sections:"
verified: true
- id: learner-1
persona: learner
class: missing
blocking: true
file: docs/admin/monitoring/health-check.md
line: 331
quote: "Coder uses a publish-subscribe (pubsub) system to send events between its components and replicas."
evidence:
- coderd/database/pubsub/pubsub.go:397-413
- coderd/nats/pubsub.go:478-509
- coderd/healthcheck/healthcheck.go:315-319
- coderd/debug.go:138-139
rule: null
problem: The section never states EPS01/EPS02 are always warning severity and never make the deployment unhealthy.
fix: "Pubsub issues are reported as warnings, not errors, so they don't make the deployment unhealthy."
verified: true
- id: language-1
persona: language
class: rule
blocking: true
file: docs/admin/monitoring/health-check.md
line: 346
quote: "Coder tries to reconnect automatically, so this might be a transient issue."
evidence: []
rule: docs/.style/style-guide/voice-and-tone.md#give-each-pronoun-one-clear-referent
problem: Bare "this" has no single clear antecedent noun.
fix: "Coder tries to reconnect automatically, so the disconnection might be transient."
verified: true
- id: learner-2
persona: learner
class: suggestion
blocking: false
file: docs/admin/monitoring/health-check.md
line: 332
quote: "The health check reports which pubsub backend the Coder replica uses, either `nats` for the embedded NATS server or `postgres` for PostgreSQL, and whether that backend is connected."
evidence:
- coderd/database/pubsub/metrics.go
- coderd/database/pubsub/pubsub_memory.go
rule: null
problem: Omits the test-only `memory` backend value.
fix: No change needed.
verified: false
rejected_reason: "Evidence confirms memory is never reachable through the production health-check path; the docs are accurate for every real reader. Rejected at checks 2 and 5."Automated review via Coder Agents. Reply to a finding's thread with the reason if the finding is wrong. |
6656bfb to
2769ec2
Compare
Docs previewLinks open this pull request's preview at https://pr-31621.docs.cdr-preview.dev, which updates with each push. Check off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here. Preview sites for each built commit (2)Every built commit keeps its own preview, newest first. Previews aren't indexed by search engines and are deleted when this pull request is merged or closed. |
Adds pubsub.ReportingPubsub, which extends Pubsub with a HealthReport method. The PostgreSQL, NATS and in-memory backends implement it and report their backend name. coderd.Options.Pubsub now takes a ReportingPubsub so the health check can include a pubsub section, and the section can be dismissed like the others.
PGPubsub tracks whether its listener is connected to PostgreSQL and warns when it isn't. The NATS pubsub warns when its embedded server is not running or any publish or subscribe connection is not connected. Adds health codes EPS01 and EPS02 with docs.
Restores import grouping and an unused receiver in coderdtest, drops a now-unneeded variable in the prebuilds test, and gives the NATS unhealthy test a real (unstarted) embedded server now that HealthReport no longer guards against a nil server.
2769ec2 to
7c2990c
Compare
Stacked on #31621. Adds a **Pubsub** section to the deployment health page (`/health/pubsub`). It shows: - the pubsub severity dot and the mute-warnings button, - the report error, if one is set, - each warning, with a link to its docs (EPS01, EPS02), - the backend in use (`postgres` or `nats`). The section is added to the health sidebar. Storybook stories cover the default, warning (NATS), and error states. <details> <summary>Plan and decisions</summary> - The page follows the same pattern as `DatabasePage`: it reads the report from the outlet context and renders warnings as dismissible Alerts with `HealthMessageDocsLink`. - Warning keys use `code-message` instead of `code`. The NATS backend can report the same code (EPS01) once for each publish and subscribe connection, so a key on `code` alone could repeat. - The page shows `pubsub.error` as an error Alert, because the server sets `error` when the report cannot be produced. - If the backend is empty, the page shows "Unknown". - The sidebar entry comes from the existing `sections` map in `HealthLayout`, so the severity dot and the dismissed state work the same way as in other sections. - Stories: `Pubsub` (default MockHealth), `WithWarnings` (NATS with EPS02 and two EPS01 warnings), `WithError`. </details> Generated by Coder Agents on behalf of @spikecurtis.
Fills in the
pubsubsection of the deployment health report on the server. Stacked on #31620.pubsub.ReportingPubsubinterface:PubsubplusHealthReport(ctx) healthsdk.PubsubReport. ThePubsubinterface itself is unchanged.postgres,natsormemory) and its connection state:pqlistener's connected state from its event callback. When it's not connected, the report has severitywarningandEPS01.EPS02if the embedded server isn'tRunning(), and withEPS01for each publish or subscribe connection whoseStatus()isn'tCONNECTED.ok.coderd.Options.Pubsubis now aReportingPubsub.ReplicaSyncPubsubstays a plainPubsub.coderd/healthcheckhas a newPubsubcheck, which counts towardhealthyand the overall severity like the other sections.formatHealthcheckapplies the dismissed setting for it.EPS01(not connected) andEPS02(embedded NATS server not running), documented indocs/admin/monitoring/health-check.md.coderdtest.Options.Pubsuband the pubsub returned bydbtestutil.NewDBare nowReportingPubsub, and a few test wrappers embedReportingPubsubinstead ofPubsub.Decision log
HealthReportreturns ahealthsdk.PubsubReportrather than filling one in place. This matches thehealthcheck.Checkermethods.ctxis unused for now.Dismissed, so backends don't need to know about health settings.pq.Listener.Ping, so the health check doesn't do database I/O. Closing the pubsub also marks it disconnected.conngainsStatus(), which*natsgo.Connalready implements. The pools are immutable after construction, soHealthReportreads them without locking.Generated by Coder Agents on behalf of @spikecurtis.