Skip to content

docs: Prometheus metrics + generator - #5179

Merged
mtojek merged 20 commits into
coder:mainfrom
mtojek:3520-doc-prom-metrics
Nov 30, 2022
Merged

mtojek merged 20 commits into
coder:mainfrom
mtojek:3520-doc-prom-metrics

Conversation

@mtojek

@mtojek mtojek commented Nov 28, 2022 •

Copy link
Copy Markdown
Member

Fixes: #3520

Preview: https://github.com/mtojek/coder/blob/3520-doc-prom-metrics/docs/admin/prometheus.md

This PR adds another doc page, that describes the Prometheus endpoint. I didn't list or describe all metrics on purpose, so we won't end up with outdated content. I added the doc generator, which converts the Prometheus dump into the Markdown table.

@mtojek mtojek self-assigned this Nov 28, 2022
@mtojek
mtojek requested a review from a team November 28, 2022 17:04
@mtojek
mtojek marked this pull request as ready for review November 28, 2022 17:14
Comment thread docs/admin/prometheus.md Resolved
Comment thread docs/admin/prometheus.md Resolved
Comment thread docs/admin/prometheus.md Resolved
Comment thread docs/admin/prometheus.md Resolved
Comment thread docs/admin/prometheus.md Resolved
Comment thread docs/admin/prometheus.md Resolved
Comment thread docs/admin/prometheus.md Resolved
mtojek and others added 6 commits November 28, 2022 18:32
Co-authored-by: Dean Sheather <dean@deansheather.com>
Co-authored-by: Dean Sheather <dean@deansheather.com>
Co-authored-by: Dean Sheather <dean@deansheather.com>
Co-authored-by: Dean Sheather <dean@deansheather.com>
Co-authored-by: Dean Sheather <dean@deansheather.com>
@mtojek mtojek mentioned this pull request Nov 28, 2022
3 tasks done
Comment thread docs/admin/prometheus.md Resolved
Comment thread docs/admin/prometheus.md Resolved
@mtojek
mtojek marked this pull request as draft November 29, 2022 07:56
@mtojek
mtojek marked this pull request as ready for review November 29, 2022 14:30
@mtojek
mtojek requested a review from a team November 29, 2022 15:00
@mtojek mtojek changed the title docs: Prometheus metrics docs: Prometheus metrics + generator Nov 29, 2022
Comment thread Makefile Resolved
Comment thread Makefile Resolved
Comment thread scripts/metricsdocgen/metrics Resolved
coderd_api_concurrent_websockets 1
# HELP coderd_api_request_latencies_ms Latency distribution of requests in milliseconds
# TYPE coderd_api_request_latencies_ms histogram
coderd_api_request_latencies_ms_bucket{method="GET",path="",le="1"} 10

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wonder if we could get a cleaner dump here, the histogram seems to take up a lot of space, but it's just repeating different routes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Frankly speaking, the idea was to just dump it and forget about it. The format is not intended to be reviewed by humans. The generator makes sure that it can be cleaned and processed into the Markdown table.

Regarding the dumping process, I considered other options like booting up the coderd and fetching Prometheus metrics from http://localhost:2112, but you would have to run it in a container as process metrics are only available in Linux. Eventually, I decided to leave it to the developer to just use curl and update the file.

If you have any ideas on how to improve it, I'm happy to change the concept.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if instead of storing the dump in the repo you made the script directly connect to the server and download the prom metrics? Would probably need to be moved out of make gen as it would be prone to failure if not configured properly.

@mtojek mtojek Nov 30, 2022 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I said before, in this case, the coder server must be started in the container, so it can also run the process stats collector (Linux only). I'm looking into fully reproducible options.

Most likely, the script would have to simulate some actions, start the workspace, provision something, etc. to collect as many metrics as possible.

Long story short, this method isn't perfect too.

EDIT:

Unless you're thinking about detaching the script from make gen totally, but I guess that the idea of having a container might be worth considering here as well.

@mtojek

mtojek commented Nov 30, 2022

Copy link
Copy Markdown
Member Author

I've just re-requested reviews. Kindly please to take another look and if it's ok, please 👍 .

@mtojek
mtojek merged commit 38bdae7 into coder:main Nov 30, 2022
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 30, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs: observability & prometheus

5 participants