Repository navigation
docs: add provisioner metrics and a metrics signpost page - #31804
coder[bot] wants to merge 2 commits into
Conversation
| | Metric | Type | Labels | Purpose | | ||
| |-------------------------------------------------------|-----------|------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------| | ||
| | `coderd_provisionerd_jobs_current` | gauge | `provisioner` | Provisioner jobs running now. | | ||
| | `coderd_provisionerd_num_daemons` | gauge | | Provisioner daemons in the process. Always `1` for an external provisioner daemon. | | ||
| | `coderd_provisionerd_job_timings_seconds` | histogram | `provisioner`, `status` | Duration of each provisioner job. | | ||
| | `coderd_provisionerd_workspace_build_timings_seconds` | histogram | `status`, `template_name`, `template_version`, `workspace_transition` | Duration of each workspace build. | | ||
| | `coderd_workspace_builds_total` | counter | `status`, `template_name`, `template_version`, `workspace_name`, `workspace_owner`, `workspace_transition` | Workspaces started, updated, or deleted. | |
There was a problem hiding this comment.
Is there a way to generate this list automatically? This can rot quickly.
There was a problem hiding this comment.
Not without changing the generator. scripts/metricsdocgen fills exactly one marked block in docs/admin/integrations/prometheus.md, so it can't write this table.
cf7267a reduces the rot: it drops the Type and Labels columns, which drift the most, and points readers to the generated Available metrics table for them. The metric names and purposes are still maintained by hand.
Two options for a follow-up PR, if you want one:
- Extend
metricsdocgento fill filtered blocks on other pages. - Add a check that fails when a metric named in
docs/is missing fromscripts/metricsdocgen/generated_metrics.
I've left this thread open so you can decide.
🤖 Reply generated with Coder Agents on behalf of @nickvigilante.
| | Metric | Type | Labels | Purpose | | ||
| |---------------------------------------------|-----------|--------------------------------------------------------------|----------------------------------------------------------------| | ||
| | `coderd_provisioner_job_queue_wait_seconds` | histogram | `build_reason`, `job_type`, `provisioner_type`, `transition` | Time from job creation until a provisioner daemon acquires it. | | ||
| | `coderd_workspace_builds_enqueued_total` | counter | `build_reason`, `provisioner_type`, `status`, `transition` | Attempts to add a workspace build to the job queue. | |
There was a problem hiding this comment.
Same change in cf7267a: the control plane table no longer has Type and Labels columns. The follow-up options in the daemon table thread apply here too.
🤖 Reply generated with Coder Agents on behalf of @nickvigilante.
The queue wait metric is recorded only when a provisioner daemon acquires a job, so it can't show jobs whose tags no provisioner matches. Say so, and point readers to the existing troubleshooting steps for finding those jobs. Restore the "Prometheus metrics" heading and the setting-up-metrics anchor so existing links keep working. The heading also matches the AI Gateway and Agent Relay pages that the signpost table links to. Drop the Type and Labels columns, which duplicate the generated Available metrics table and would drift from it. Reword the signpost intro, because the workspace proxies page doesn't list metrics. Add a Pod discovery example for the Helm chart, and apply contraction and "for example" punctuation fixes from the docs style guide. Refs DOCS-892
|
Closing and reopening under a human author, as required for this repository. |
Summary
Admins who start from the External provisioners docs had no way to find out which Prometheus metrics exist for provisioners, or how to turn them on for standalone provisioner daemons.
This PR documents both, and turns the deployment metrics page into a signpost to the monitoring docs for each area.
Changes
docs/install/operate/provisioners/index.md: expands the "Prometheus metrics" section. The heading and its#prometheus-metricsanchor are unchanged.--prometheus-enable/CODER_PROMETHEUS_ENABLEand--prometheus-address/CODER_PROMETHEUS_ADDRESS. It corrects the stated default address fromlocalhost:2112to127.0.0.1:2112.coder-provisionerHelm chart: the fixed0.0.0.0:2112address, theprometheus-httpcontainer port, and the missing Service. It adds a Pod discovery example withcoder.podAnnotationsand aPodMonitoralternative.docs/install/operate/provisioners/manage-provisioner-jobs.md: links from the troubleshooting section to the metrics section.docs/admin/monitoring/metrics.md: becomes a signpost page.#setting-up-metricsanchor.No new pages, so
docs/manifest.jsonis unchanged.Testing
make lint/emdash,go run ./scripts/docshtmlcheck, markdownlint, andmarkdown-table-formatterare clean on the changed files.pre-commit-light, includinglint/docs-redirects,lint/markdown, andlint/style-claims) passed.Verified against
provisionerd/provisionerd.go(NewMetrics): names of the five provisioner daemon metrics.provisionerd/runner/runner.go: where the job and build metrics are recorded.cli/server.go:1273-1296: the control plane registers the daemon metrics only for built-in provisioners, and setsnum_daemonsto the built-in count.cli/server.go:385-389registers the provisioner server metrics.coderd/provisionerdserver/provisionerdserver.go:1011-1019:coderd_provisioner_job_queue_wait_secondsis recorded only when a provisioner daemon acquires a job.coderd/coderd.go:2602andenterprise/coderd/provisionerdaemons.go:365pass the same metrics to built-in and external provisioners.coderd/jobreaper/detector.go:26-28: a pending job that nothing acquires is failed after 30 minutes, without a queue wait observation.coderd/wsbuilder/metrics.goandcoderd/wsbuilder/wsbuilder.go:357-373:coderd_workspace_builds_enqueued_total.enterprise/cli/provisionerdaemonstart.go:204-218, 371-383: the flags, env vars, and127.0.0.1:2112default. A standalone daemon setsnum_daemonsto 1 and registers the Go, process, andpromhttpcollectors.cli/agent.go:603-611: the metrics handler serves every path, so the default/metricsscrape path works.helm/provisioner/templates/_coder.tpl:40-41, 81-96: the fixed address, and theprometheus-httpport that appears only whenCODER_PROMETHEUS_ENABLEis"true". The chart has no Service template.helm/libcoder/templates/_coder.yaml:24-33,helm/libcoder/templates/_helpers.tpl:20-22, andhelm/provisioner/tests/testdata/labels_annotations.golden:coder.podAnnotationsandcoder.podLabelsland on the Pod template, which carries theapp.kubernetes.io/nameandapp.kubernetes.io/instancelabels.enterprise/cli/proxyserver.go:243-250: workspace proxies serve metrics on their own listener.coderd/workspaces_test.go(queue wait),helm/provisioner/tests/testdata/default_values.golden(fixed address), andenterprise/cli/testdata/coder_provisioner_start_--help.golden(CLI default).Decision log
## Prometheus metricsheading. Renaming it gained nothing and would break external links to#prometheus-metrics. The AI Gateway and Agent Relay pages that the signpost table links to use the same heading.<a id="setting-up-metrics"></a>onmetrics.md. External links to that anchor keep working, andscripts/docshtmlcheckpasses with it.scripts/metricsdocgenfills one marked block inprometheus.md, so it can't generate these tables. Types and labels drift the most, so the page now points to the generated table for them. Extending the generator, or adding a check that metrics named indocs/exist inscripts/metricsdocgen/generated_metrics, would need a separate PR.docs/agent-relay/monitoring/index.md.Related gaps not edited
docs/admin/integrations/prometheus.md:14anddocs/admin/networking/workspace-proxies.md:221still saylocalhost:2112. The server and proxy default is127.0.0.1:2112(codersdk/deployment.go:2658).docs/workspaces/speed-up-templates.md:42-44implies that control plane metrics includecoderd_provisionerd_job_timings_seconds. That holds only for built-in provisioners.docs/admin/networking/workspace-proxies.md#observabilityexplains how to turn on the endpoint but doesn't list any proxy metrics.docs/admin/networking/(DERP and tailnet, for examplecoder_derp_server_*andcoder_servertailnet_*) has no metrics section.docs/admin/monitoring/health-check.mdlinks to the database metrics flag but doesn't list thecoderd_db_*metrics.docs/admin/monitoring/notifications/doesn't list thecoderd_notifications_*metrics.docs/admin/users/(coderd_api_active_users_duration_hour, license metrics) anddocs/admin/security/audit-logs.mdhave no metrics sections.docs/templates/extending-templates/prebuilt-workspaces.md#available-metricslists metrics without types or labels, and doesn't say how to turn them on.Disclosure: these changes were generated by Coder Agents.
Closes DOCS-892