Skip to content

docs: add provisioner metrics and a metrics signpost page - #31804

Closed
coder[bot] wants to merge 2 commits into
mainfrom
docs/provisioner-metrics
Closed

coder[bot] wants to merge 2 commits into
mainfrom
docs/provisioner-metrics

Conversation

@coder

@coder coder Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Admins who start from the External provisioners docs had no way to find out which Prometheus metrics exist for provisioners, or how to turn them on for standalone provisioner daemons.
This PR documents both, and turns the deployment metrics page into a signpost to the monitoring docs for each area.

Changes

  • docs/install/operate/provisioners/index.md: expands the "Prometheus metrics" section. The heading and its #prometheus-metrics anchor are unchanged.
    • It explains which metrics each provisioner daemon reports and which the control plane reports. For built-in provisioners, the control plane also reports the daemon metrics.
    • Two tables list each metric's purpose and link to the generated Available metrics table for types and labels.
    • It explains how to read the queue wait metric. Jobs whose tags no provisioner matches don't appear in it until a provisioner acquires them.
    • It shows how to turn on the daemon's Prometheus endpoint with --prometheus-enable / CODER_PROMETHEUS_ENABLE and --prometheus-address / CODER_PROMETHEUS_ADDRESS. It corrects the stated default address from localhost:2112 to 127.0.0.1:2112.
    • It covers the coder-provisioner Helm chart: the fixed 0.0.0.0:2112 address, the prometheus-http container port, and the missing Service. It adds a Pod discovery example with coder.podAnnotations and a PodMonitor alternative.
  • docs/install/operate/provisioners/manage-provisioner-jobs.md: links from the troubleshooting section to the metrics section.
  • docs/admin/monitoring/metrics.md: becomes a signpost page.
    • It keeps the control plane setup links and the #setting-up-metrics anchor.
    • It adds a table of monitoring pages per area (provisioners, workspace proxies, prebuilt workspaces, AI Gateway, Agent Relay), and notes which areas serve metrics on their own endpoint.
    • It links to the full metrics list.

No new pages, so docs/manifest.json is unchanged.

Testing

  • make lint/emdash, go run ./scripts/docshtmlcheck, markdownlint, and markdown-table-formatter are clean on the changed files.
  • Vale reports 0 errors and 0 warnings. Its 4 suggestions are on lines this PR doesn't change.
  • The pre-commit hook (pre-commit-light, including lint/docs-redirects, lint/markdown, and lint/style-claims) passed.
  • A script resolved every relative link and anchor in the changed files.
Verified against
  • provisionerd/provisionerd.go (NewMetrics): names of the five provisioner daemon metrics.
  • provisionerd/runner/runner.go: where the job and build metrics are recorded.
  • cli/server.go:1273-1296: the control plane registers the daemon metrics only for built-in provisioners, and sets num_daemons to the built-in count. cli/server.go:385-389 registers the provisioner server metrics.
  • coderd/provisionerdserver/provisionerdserver.go:1011-1019: coderd_provisioner_job_queue_wait_seconds is recorded only when a provisioner daemon acquires a job. coderd/coderd.go:2602 and enterprise/coderd/provisionerdaemons.go:365 pass the same metrics to built-in and external provisioners.
  • coderd/jobreaper/detector.go:26-28: a pending job that nothing acquires is failed after 30 minutes, without a queue wait observation.
  • coderd/wsbuilder/metrics.go and coderd/wsbuilder/wsbuilder.go:357-373: coderd_workspace_builds_enqueued_total.
  • enterprise/cli/provisionerdaemonstart.go:204-218, 371-383: the flags, env vars, and 127.0.0.1:2112 default. A standalone daemon sets num_daemons to 1 and registers the Go, process, and promhttp collectors.
  • cli/agent.go:603-611: the metrics handler serves every path, so the default /metrics scrape path works.
  • helm/provisioner/templates/_coder.tpl:40-41, 81-96: the fixed address, and the prometheus-http port that appears only when CODER_PROMETHEUS_ENABLE is "true". The chart has no Service template.
  • helm/libcoder/templates/_coder.yaml:24-33, helm/libcoder/templates/_helpers.tpl:20-22, and helm/provisioner/tests/testdata/labels_annotations.golden: coder.podAnnotations and coder.podLabels land on the Pod template, which carries the app.kubernetes.io/name and app.kubernetes.io/instance labels.
  • enterprise/cli/proxyserver.go:243-250: workspace proxies serve metrics on their own listener.
  • Tests that pin these claims: coderd/workspaces_test.go (queue wait), helm/provisioner/tests/testdata/default_values.golden (fixed address), and enterprise/cli/testdata/coder_provisioner_start_--help.golden (CLI default).
Decision log
  • Kept the ## Prometheus metrics heading. Renaming it gained nothing and would break external links to #prometheus-metrics. The AI Gateway and Agent Relay pages that the signpost table links to use the same heading.
  • Kept <a id="setting-up-metrics"></a> on metrics.md. External links to that anchor keep working, and scripts/docshtmlcheck passes with it.
  • Dropped the Type and Labels columns. scripts/metricsdocgen fills one marked block in prometheus.md, so it can't generate these tables. Types and labels drift the most, so the page now points to the generated table for them. Extending the generator, or adding a check that metrics named in docs/ exist in scripts/metricsdocgen/generated_metrics, would need a separate PR.
  • Rewrote the queue wait guidance. The earlier text said a rising value can mean no provisioner matches a job's tags. The code records the value only when a provisioner acquires a job, so unmatched jobs never reach the metric. The page now links to the existing troubleshooting steps for finding them, instead of restating a command.
  • Reworded the signpost intro. The workspace proxies page explains how to turn on metrics but doesn't list them, so the table describes each linked page as covering monitoring, not listing metrics.
  • "Separate endpoint" column. Checked in code for workspace proxies. Agent Relay has no server code in this repo, so its "Yes" comes from docs/agent-relay/monitoring/index.md.
Related gaps not edited
  • docs/admin/integrations/prometheus.md:14 and docs/admin/networking/workspace-proxies.md:221 still say localhost:2112. The server and proxy default is 127.0.0.1:2112 (codersdk/deployment.go:2658).
  • docs/workspaces/speed-up-templates.md:42-44 implies that control plane metrics include coderd_provisionerd_job_timings_seconds. That holds only for built-in provisioners.
  • docs/admin/networking/workspace-proxies.md#observability explains how to turn on the endpoint but doesn't list any proxy metrics.
  • docs/admin/networking/ (DERP and tailnet, for example coder_derp_server_* and coder_servertailnet_*) has no metrics section.
  • docs/admin/monitoring/health-check.md links to the database metrics flag but doesn't list the coderd_db_* metrics.
  • docs/admin/monitoring/notifications/ doesn't list the coderd_notifications_* metrics.
  • docs/admin/users/ (coderd_api_active_users_duration_hour, license metrics) and docs/admin/security/audit-logs.md have no metrics sections.
  • docs/templates/extending-templates/prebuilt-workspaces.md#available-metrics lists metrics without types or labels, and doesn't say how to turn them on.

Disclosure: these changes were generated by Coder Agents.

Closes DOCS-892

@coder
coder Bot requested a review from nickvigilante November 5, 2026 20:19
@linear-code

linear-code Bot commented Nov 5, 2026

Copy link
Copy Markdown

DOCS-892

@github-actions github-actions Bot added the community Pull Requests and issues created by the community. label Nov 5, 2026
Comment thread docs/admin/monitoring/metrics.md Resolved
Comment on lines +349 to +355
| Metric | Type | Labels | Purpose |
|-------------------------------------------------------|-----------|------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------|
| `coderd_provisionerd_jobs_current` | gauge | `provisioner` | Provisioner jobs running now. |
| `coderd_provisionerd_num_daemons` | gauge | | Provisioner daemons in the process. Always `1` for an external provisioner daemon. |
| `coderd_provisionerd_job_timings_seconds` | histogram | `provisioner`, `status` | Duration of each provisioner job. |
| `coderd_provisionerd_workspace_build_timings_seconds` | histogram | `status`, `template_name`, `template_version`, `workspace_transition` | Duration of each workspace build. |
| `coderd_workspace_builds_total` | counter | `status`, `template_name`, `template_version`, `workspace_name`, `workspace_owner`, `workspace_transition` | Workspaces started, updated, or deleted. |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to generate this list automatically? This can rot quickly.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not without changing the generator. scripts/metricsdocgen fills exactly one marked block in docs/admin/integrations/prometheus.md, so it can't write this table.

cf7267a reduces the rot: it drops the Type and Labels columns, which drift the most, and points readers to the generated Available metrics table for them. The metric names and purposes are still maintained by hand.

Two options for a follow-up PR, if you want one:

  • Extend metricsdocgen to fill filtered blocks on other pages.
  • Add a check that fails when a metric named in docs/ is missing from scripts/metricsdocgen/generated_metrics.

I've left this thread open so you can decide.

🤖 Reply generated with Coder Agents on behalf of @nickvigilante.

Comment on lines +361 to +364
| Metric | Type | Labels | Purpose |
|---------------------------------------------|-----------|--------------------------------------------------------------|----------------------------------------------------------------|
| `coderd_provisioner_job_queue_wait_seconds` | histogram | `build_reason`, `job_type`, `provisioner_type`, `transition` | Time from job creation until a provisioner daemon acquires it. |
| `coderd_workspace_builds_enqueued_total` | counter | `build_reason`, `provisioner_type`, `status`, `transition` | Attempts to add a workspace build to the job queue. |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same issue here.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same change in cf7267a: the control plane table no longer has Type and Labels columns. The follow-up options in the daemon table thread apply here too.

🤖 Reply generated with Coder Agents on behalf of @nickvigilante.

The queue wait metric is recorded only when a provisioner daemon
acquires a job, so it can't show jobs whose tags no provisioner
matches. Say so, and point readers to the existing troubleshooting
steps for finding those jobs.

Restore the "Prometheus metrics" heading and the setting-up-metrics
anchor so existing links keep working. The heading also matches the
AI Gateway and Agent Relay pages that the signpost table links to.

Drop the Type and Labels columns, which duplicate the generated
Available metrics table and would drift from it. Reword the signpost
intro, because the workspace proxies page doesn't list metrics. Add a
Pod discovery example for the Helm chart, and apply contraction and
"for example" punctuation fixes from the docs style guide.

Refs DOCS-892
@nickvigilante
nickvigilante temporarily deployed to main November 10, 2026 00:16 — with GitHub Actions Inactive
@nickvigilante
nickvigilante temporarily deployed to main November 10, 2026 00:18 — with GitHub Actions Inactive
@nickvigilante

Copy link
Copy Markdown
Collaborator

Closing and reopening under a human author, as required for this repository.

@nickvigilante
nickvigilante temporarily deployed to main November 10, 2026 15:52 — with GitHub Actions Inactive
@nickvigilante
nickvigilante temporarily deployed to main November 10, 2026 15:52 — with GitHub Actions Inactive
@nickvigilante
nickvigilante temporarily deployed to main November 10, 2026 15:52 — with GitHub Actions Inactive
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 10, 2026

This branch was previously deployed

1 inactive deployment
main — cf7267a9 Deployed Nov 10, 2026 by nickvigilante via community-label #253419
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

community Pull Requests and issues created by the community.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants