Skip to content

Commit eecca9a

Browse files
authored
docs(secretstore): improve secret store backend documentation (#22099)
- Rewrite overview descriptions to lead with user benefit instead of jargon - Add realistic collector config examples (MySQL DSN + Elasticsearch/PostgreSQL/HTTP check) with explicit descriptions explaining the secret substitution mechanics - Move standalone reference examples into format.description as inline examples - Document that secretstore configs support ${env:...}, ${file:...}, ${cmd:...} resolvers - Add ${env:...} alternative examples for Azure client_secret and Vault token - Add "Choosing a Resolver" decision guide to SECRETS.md - Add "Multiple Secretstores" and "Mixing Resolver Types" sections to SECRETS.md - Add Docker Secrets and Kubernetes Secrets notes to the Files section - Add config directory creation instructions for /etc/netdata/go.d/ss/ - Add IAM/RBAC minimum permission hints per backend - Fix GCP limitations (was describing default behavior, not actual limitations) - Fix Vault examples: remove unnecessary tls_skip_verify default, add Enterprise namespace example - Add token_file re-read behavior note for Vault - Add GKE Workload Identity note for GCP metadata mode - Add region "no auto-detection" note for AWS - Fix schema bug: minLength -> minItems for array validation in secretstore.json - Change default code fence language from text to yaml in collector_configs template
1 parent 137e28a commit eecca9a

13 files changed

Lines changed: 394 additions & 148 deletions

File tree

integrations/gen_doc_secrets_page.py

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727
],
2828
"jump_to": [
2929
{"label": "Resolver Quick Reference", "anchor": "resolver-quick-reference"},
30+
{"label": "Choosing a Resolver", "anchor": "choosing-a-resolver"},
3031
{"label": "Environment Variables", "anchor": "environment-variables"},
3132
{"label": "Files", "anchor": "files"},
3233
{"label": "Commands", "anchor": "commands"},
@@ -61,6 +62,12 @@
6162
"notes": "Configure the secretstore first, then reference it from collector configs.",
6263
},
6364
],
65+
"choosing_a_resolver": [
66+
"Use `${env:...}` or `${file:...}` for simple setups where secrets are already available locally on the Netdata host.",
67+
"Use `${cmd:...}` when you need dynamic secret retrieval via a trusted local command, such as 1Password CLI or a custom script.",
68+
"Use `${store:...}` when your organization manages secrets centrally in a cloud provider or Vault and you want Netdata to pull from that source directly.",
69+
"You can use different resolver types across different collectors, different jobs within the same collector, or even within the same configuration value. See [Mixing resolver types](#mixing-resolver-types).",
70+
],
6471
"sections": {
6572
"env": {
6673
"heading": "## Environment Variables",
@@ -87,6 +94,8 @@
8794
"The file path must be absolute.",
8895
"Netdata trims leading and trailing whitespace from the file contents.",
8996
"The file must exist on the Netdata host and be readable by the `netdata` user.",
97+
"**Docker Secrets**: Docker mounts secrets as files under `/run/secrets/` inside the container. Use `${file:/run/secrets/<secret-name>}` to read them.",
98+
"**Kubernetes Secrets**: If you mount Kubernetes Secrets as volume files in the Netdata pod, reference them with `${file:/path/to/mounted/secret}`.",
9099
],
91100
},
92101
"cmd": {
@@ -130,6 +139,32 @@
130139
],
131140
"file_intro": "Each secretstore backend has its own file under `/etc/netdata/go.d/ss/`:",
132141
"file_note": "File-based secretstores are loaded at agent startup. If you edit these files, restart the Netdata Agent to apply the changes.",
142+
"file_directory": (
143+
"If the `/etc/netdata/go.d/ss/` directory does not exist, create it:\n\n"
144+
"```bash\n"
145+
"sudo mkdir -p /etc/netdata/go.d/ss\n"
146+
"sudo chown netdata:netdata /etc/netdata/go.d/ss\n"
147+
"sudo chmod 0750 /etc/netdata/go.d/ss\n"
148+
"```\n\n"
149+
"Secretstore configuration files may contain sensitive values such as tokens or client secrets. "
150+
"Restrict directory and file permissions to the `netdata` user."
151+
),
152+
"mixing": (
153+
"You can mix different resolver types in the same configuration value or the same config file. "
154+
"For example, you might read the username from an environment variable and the password from a secretstore:\n\n"
155+
"```yaml\n"
156+
"jobs:\n"
157+
" - name: mysql_prod\n"
158+
' dsn: "${env:MYSQL_USER}:${store:vault:vault_prod:secret/data/netdata/mysql#password}@tcp(127.0.0.1:3306)/"\n'
159+
"```\n\n"
160+
"Different jobs within the same collector config file can also use different resolver types."
161+
),
162+
"multiple_stores": (
163+
"Each secretstore config file can contain multiple `jobs` entries, each with a unique store name. "
164+
"You can use different secretstore backends simultaneously. "
165+
"For example, you might configure a Vault store for database credentials and an AWS Secrets Manager store for API keys, "
166+
"then reference each one using its `${store:<kind>:<name>:<operand>}` syntax in the relevant collector configs."
167+
),
133168
},
134169
"secretstores": {
135170
"heading": "## Supported Secretstore Backends",
@@ -145,6 +180,7 @@
145180
"security_notes": [
146181
"Prefer secret references over plain-text credentials in collector configs.",
147182
"Prefer platform-native identity modes for production when a backend supports them, such as instance roles, managed identities, or metadata-based credentials.",
183+
"Secretstore configuration values (such as tokens and client secrets) also support `${env:...}`, `${file:...}`, and `${cmd:...}` resolvers. Use them to avoid storing backend credentials in plain text. Note that `${store:...}` references are not supported inside secretstore configurations.",
148184
"Keep local secret material readable only by the `netdata` user, including token files, service account files, and any files used with `${file:...}`.",
149185
"Use `${cmd:...}` only with trusted local commands and absolute paths.",
150186
],
@@ -288,6 +324,7 @@ def build_page_context() -> Dict[str, Any]:
288324
f'[{jump["label"]}](#{jump["anchor"]})' for jump in SECRETS_PAGE["jump_to"]
289325
),
290326
"quick_reference": SECRETS_PAGE["quick_reference"],
327+
"choosing_a_resolver": SECRETS_PAGE["choosing_a_resolver"],
291328
"sections": [
292329
SECRETS_PAGE["sections"]["env"],
293330
SECRETS_PAGE["sections"]["file"],

integrations/schemas/secretstore.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -167,7 +167,7 @@
167167
"properties": {
168168
"list": {
169169
"type": "array",
170-
"minLength": 1,
170+
"minItems": 1,
171171
"items": {
172172
"$ref": "#/$defs/collector_configs_example"
173173
}

integrations/templates/collector_configs.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@
2020

2121
[[ example.description ]]
2222

23-
```[[ example.language or 'text' ]]
23+
```[[ example.language or 'yaml' ]]
2424
[[ example.content ]]
2525
```
2626
[% endfor %]

integrations/templates/secrets.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,12 @@
1717
| [[ item.resolver ]] | [[ item.syntax ]] | [[ item.best_for ]] | [[ item.notes ]] |
1818
[% endfor %]
1919

20+
## Choosing a Resolver
21+
22+
[% for item in page.choosing_a_resolver %]
23+
- [[ item ]]
24+
[% endfor %]
25+
2026
[% for section in page.sections %]
2127
[[ section.heading ]]
2228

@@ -75,6 +81,16 @@ Each file contains a `jobs` array. The backend kind is determined by the filenam
7581

7682
:::
7783

84+
[[ page.store.file_directory ]]
85+
86+
### Multiple Secretstores
87+
88+
[[ page.store.multiple_stores ]]
89+
90+
### Mixing Resolver Types
91+
92+
[[ page.store.mixing ]]
93+
7894
[[ page.secretstores.heading ]]
7995

8096
[[ page.secretstores.intro ]]

src/collectors/SECRETS.md

Lines changed: 37 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ Netdata lets you reference secret values in collector configs instead of storing
66

77
### Jump To
88

9-
[Resolver Quick Reference](#resolver-quick-reference)[Environment Variables](#environment-variables)[Files](#files)[Commands](#commands)[Secretstores](#secretstores)[Supported Secretstore Backends](#supported-secretstore-backends)[How It Works](#how-it-works)[Troubleshooting](#troubleshooting)
9+
[Resolver Quick Reference](#resolver-quick-reference)[Choosing a Resolver](#choosing-a-resolver)[Environment Variables](#environment-variables)[Files](#files)[Commands](#commands)[Secretstores](#secretstores)[Supported Secretstore Backends](#supported-secretstore-backends)[How It Works](#how-it-works)[Troubleshooting](#troubleshooting)
1010

1111

1212
## Resolver Quick Reference
@@ -18,6 +18,13 @@ Netdata lets you reference secret values in collector configs instead of storing
1818
| Command | `${cmd:/absolute/path/to/command args}` | Secrets returned by a trusted local command | The command path must be absolute. Netdata uses a 10-second timeout. |
1919
| Secretstore | `${store:<kind>:<name>:<operand>}` | Secrets stored in remote backends such as Vault, AWS, Azure, or GCP | Configure the secretstore first, then reference it from collector configs. |
2020

21+
## Choosing a Resolver
22+
23+
- Use `${env:...}` or `${file:...}` for simple setups where secrets are already available locally on the Netdata host.
24+
- Use `${cmd:...}` when you need dynamic secret retrieval via a trusted local command, such as 1Password CLI or a custom script.
25+
- Use `${store:...}` when your organization manages secrets centrally in a cloud provider or Vault and you want Netdata to pull from that source directly.
26+
- You can use different resolver types across different collectors, different jobs within the same collector, or even within the same configuration value. See [Mixing resolver types](#mixing-resolver-types).
27+
2128
## Environment Variables
2229

2330
Use `${env:VARIABLE_NAME}` to read a secret from the Netdata process environment.
@@ -44,6 +51,8 @@ jobs:
4451
- The file path must be absolute.
4552
- Netdata trims leading and trailing whitespace from the file contents.
4653
- The file must exist on the Netdata host and be readable by the `netdata` user.
54+
- **Docker Secrets**: Docker mounts secrets as files under `/run/secrets/` inside the container. Use `${file:/run/secrets/<secret-name>}` to read them.
55+
- **Kubernetes Secrets**: If you mount Kubernetes Secrets as volume files in the Netdata pod, reference them with `${file:/path/to/mounted/secret}`.
4756

4857
## Commands
4958

@@ -114,6 +123,32 @@ File-based secretstores are loaded at agent startup. If you edit these files, re
114123

115124
:::
116125

126+
If the `/etc/netdata/go.d/ss/` directory does not exist, create it:
127+
128+
```bash
129+
sudo mkdir -p /etc/netdata/go.d/ss
130+
sudo chown netdata:netdata /etc/netdata/go.d/ss
131+
sudo chmod 0750 /etc/netdata/go.d/ss
132+
```
133+
134+
Secretstore configuration files may contain sensitive values such as tokens or client secrets. Restrict directory and file permissions to the `netdata` user.
135+
136+
### Multiple Secretstores
137+
138+
Each secretstore config file can contain multiple `jobs` entries, each with a unique store name. You can use different secretstore backends simultaneously. For example, you might configure a Vault store for database credentials and an AWS Secrets Manager store for API keys, then reference each one using its `${store:<kind>:<name>:<operand>}` syntax in the relevant collector configs.
139+
140+
### Mixing Resolver Types
141+
142+
You can mix different resolver types in the same configuration value or the same config file. For example, you might read the username from an environment variable and the password from a secretstore:
143+
144+
```yaml
145+
jobs:
146+
- name: mysql_prod
147+
dsn: "${env:MYSQL_USER}:${store:vault:vault_prod:secret/data/netdata/mysql#password}@tcp(127.0.0.1:3306)/"
148+
```
149+
150+
Different jobs within the same collector config file can also use different resolver types.
151+
117152
## Supported Secretstore Backends
118153

119154
Use the backend README for provider-specific authentication, operand rules, configuration examples, and troubleshooting.
@@ -137,6 +172,7 @@ Use the backend README for provider-specific authentication, operand rules, conf
137172

138173
- Prefer secret references over plain-text credentials in collector configs.
139174
- Prefer platform-native identity modes for production when a backend supports them, such as instance roles, managed identities, or metadata-based credentials.
175+
- Secretstore configuration values (such as tokens and client secrets) also support `${env:...}`, `${file:...}`, and `${cmd:...}` resolvers. Use them to avoid storing backend credentials in plain text. Note that `${store:...}` references are not supported inside secretstore configurations.
140176
- Keep local secret material readable only by the `netdata` user, including token files, service account files, and any files used with `${file:...}`.
141177
- Use `${cmd:...}` only with trusted local commands and absolute paths.
142178

src/go/plugin/agent/secrets/secretstore/backends/aws/integrations/aws-sm.md

Lines changed: 29 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,9 @@ Kind: aws-sm
2020

2121
## Overview
2222

23-
Use AWS Secrets Manager as a secretstore backend when you want Netdata collectors to read secrets from AWS at runtime instead of storing them in plain text in collector configuration files.
23+
Netdata can pull collector credentials directly from AWS Secrets Manager at runtime, so you never store passwords or tokens in plain-text configuration files.
2424

25-
This page covers AWS Secrets Manager specific setup. For the shared resolver workflow and syntax, see [Secrets Management](https://github.com/netdata/netdata/blob/master/src/collectors/SECRETS.md).
25+
This page covers AWS Secrets Manager specific setup. For the full resolver overview and syntax reference, including simpler alternatives like `${env:...}`, `${file:...}`, and `${cmd:...}`, see [Secrets Management](https://github.com/netdata/netdata/blob/master/src/collectors/SECRETS.md).
2626

2727

2828
### Limitations
@@ -54,7 +54,7 @@ For production on AWS, prefer `ecs` or `imds` over `env` so credentials are supp
5454

5555
#### Allow access to Secrets Manager
5656

57-
The AWS identity used by this secretstore must be allowed to read the secrets you reference in collector configs in the configured `region`.
57+
The AWS identity used by this secretstore must have the `secretsmanager:GetSecretValue` permission on the secrets you reference in collector configs. Scope the IAM policy to only the secret ARNs Netdata needs.
5858

5959

6060
#### Plan for file-based changes
@@ -75,7 +75,7 @@ The following options can be defined for this secretstore backend.
7575
| Option | Description | Default | Required |
7676
|:-----|:------------|:--------|:---------:|
7777
| [auth_mode](#option-auth-mode) | How Netdata obtains AWS credentials. | env | yes |
78-
| region | AWS region used for Secrets Manager requests. | | yes |
78+
| region | AWS region used for Secrets Manager requests. There is no automatic region detection — you must always set this explicitly. | | yes |
7979

8080
<a id="option-auth-mode"></a>
8181
##### auth_mode
@@ -148,13 +148,13 @@ jobs:
148148

149149
## Use in collector configs
150150

151-
Reference AWS Secrets Manager secrets from collector configs with the `aws-sm` secretstore kind.
151+
Use the `${store:aws-sm:...}` syntax to reference AWS Secrets Manager secrets in any string field of a collector configuration file.
152152

153153

154154
The operand is `secret-name` or `secret-name#key`.
155155

156-
- Use `secret-name` to return the whole `SecretString`.
157-
- Use `secret-name#key` to read one top-level field from a JSON `SecretString`.
156+
- Use `secret-name` to return the whole `SecretString`, for example: `${store:aws-sm:aws_prod:netdata/mysql/password}`.
157+
- Use `secret-name#key` to read one top-level field from a JSON `SecretString`, for example: `${store:aws-sm:aws_prod:netdata/mysql#password}`.
158158
- If you use `#key`, Netdata parses the secret value as JSON. Secret resolution fails if the value is not valid JSON or if the key does not exist.
159159
- Nested paths such as `parent.child` are not interpreted as nested JSON lookups.
160160

@@ -168,28 +168,37 @@ ${store:aws-sm:<store-name>:<secret-name[#key]>}
168168
- `<secret-name[#key]>`: The AWS Secrets Manager secret name, optionally followed by `#key` to read one field from a JSON `SecretString`.
169169

170170
### Examples
171-
#### Whole secret value
171+
#### MySQL collector with password from AWS Secrets Manager
172172

173-
Return the full `SecretString` stored under the `netdata/mysql/password` secret.
173+
This example configures a MySQL collector job in `/etc/netdata/go.d/mysql.conf`.
174+
The password in the DSN connection string is not stored in plain text. Instead,
175+
`${store:aws-sm:aws_prod:netdata/mysql#password}` tells Netdata to fetch the secret
176+
named `netdata/mysql` from the `aws_prod` store, extract the `password` field from
177+
its JSON value, and substitute it into the DSN at runtime.
174178

175-
```text
176-
${store:aws-sm:aws_prod:netdata/mysql/password}
177-
```
178-
#### JSON field from SecretString
179179

180-
Read the `password` field from a JSON `SecretString`.
180+
```yaml
181+
# /etc/netdata/go.d/mysql.conf
182+
jobs:
183+
- name: mysql_prod
184+
dsn: "netdata:${store:aws-sm:aws_prod:netdata/mysql#password}@tcp(127.0.0.1:3306)/"
181185

182-
```text
183-
${store:aws-sm:aws_prod:netdata/mysql#password}
184186
```
185-
#### Collector config example
187+
#### Elasticsearch collector with HTTP basic auth
188+
189+
This example configures an Elasticsearch collector job in `/etc/netdata/go.d/elasticsearch.conf`.
190+
The `password` field uses a secret reference instead of a plain-text password. Netdata fetches
191+
the secret named `netdata/elasticsearch/password` from the `aws_prod` store and substitutes
192+
its full value into the `password` field at runtime.
186193

187-
Use an AWS-stored password in a collector DSN.
188194

189195
```yaml
196+
# /etc/netdata/go.d/elasticsearch.conf
190197
jobs:
191-
- name: mysql_prod
192-
dsn: "netdata:${store:aws-sm:aws_prod:netdata/mysql#password}@tcp(127.0.0.1:3306)/"
198+
- name: es_prod
199+
url: https://elasticsearch.example.com:9200
200+
username: netdata
201+
password: "${store:aws-sm:aws_prod:netdata/elasticsearch/password}"
193202

194203
```
195204

0 commit comments

Comments
 (0)