fix/observability: Prometheus read only root volume - #1000

Open issues

Found in review on 2026-11-13 (review thread), at b197759. The chart change is one correct line; this PR is a draft until the image it needs is on main, because nothing but the sentence under Solution was holding the merge back.

1. The prometheus image on main does not have the fix yet. sourcegraph/sourcegraph#18737 is still open, and 8.1.4873, the tag main pins, still starts Alertmanager with --storage.path=/alertmanager. The 8.2.0 release image does too. Reproduced on EKS with this branch and that image: the pod stays Ready with 0 restarts, Alertmanager drops from three receivers to only src-noop-receiver, and the site configuration shows failed to write Alertmanager configuration: open /sg_config_prometheus/alertmanager.yml: read-only file system. Nothing pages, so a merge today would silently stop alert notifications on every default install of the insiders chart. Fix: merge after main pins an image with the fix
2. Merge conflict with main in charts/sourcegraph/CHANGELOG.md
3. The changelog entry no longer says how to opt out. An earlier commit told users to set readOnlyRootFilesystem back to false when running an older prometheus image with this chart; that sentence is gone. Anyone who pins an older image tag or mirrors images late hits issue 1. Fix: put the sentence back, and name the first version whose image has the fix
4. Small ones:
   • no Cloud override setting readOnlyRootFilesystem: false was found in the cloud or controller repositories, which the description offers as the alternative for Cloud
   • no test asserts the new default

Tested in this review on the EKS test instance m5, by upgrading from main to this branch merged onto main, with the prometheus image main pins:

• The upgrade replaced only the prometheus pod, and the manifest differs from main by the one readOnlyRootFilesystem line
• On main, /alertmanager and /tmp are writable and Alertmanager has three receivers. On this branch only the /prometheus volume is writable, and the result is issue 1, which matches the control under Test plan
• After going back to main, the three receivers returned
• Not tested: the image from sourcegraph/sourcegraph#18737 with this chart. That is the test under Test plan, and this review did not repeat it

Problem

Some customers need a read-only root filesystem on every container, and still need Alertmanager to work

The chart turned readOnlyRootFilesystem off for prometheus, because Alertmanager wrote to the root filesystem

Customers who forced it on lost the alert notifications they set in observability.alerts

Solution

Set prometheus.containerSecurityContext.readOnlyRootFilesystem: true

• sourcegraph/sourcegraph#18737 moves Alertmanager's files onto the /prometheus data volume
• runAsUser, runAsGroup, and fsGroup are not changed

Do not merge until the chart's prometheus image includes sourcegraph/sourcegraph#18737:

• With an older image, Alertmanager cannot apply observability.alerts
• Cloud deploys non-release instances with the latest insiders chart and the instance's own image tag
  • Those instances need an image with the fix first, or readOnlyRootFilesystem: false in Cloud's global override

Test plan

• helm lint charts/sourcegraph and helm unittest charts/sourcegraph pass
• helm template renders readOnlyRootFilesystem: true on the prometheus container
• Upgraded an EKS instance from the released 8.1.1335, with this chart change and the image Buildkite built from sourcegraph/sourcegraph#18737
  • helm diff shows only the prometheus image and readOnlyRootFilesystem change
  • The pod runs read-only on its existing EBS volume, with no read-only or permission errors
  • The receivers from observability.alerts come back, and a change to them while the pod runs reaches Alertmanager
  • Deleting the pod takes 1.8 seconds, well inside the 120 second grace period
  • Control: this chart change with the released prometheus:8.1.1335 fails with open /sg_config_prometheus/alertmanager.yml: read-only file system, and keeps only the no-op receiver
• The rest of the results are in sourcegraph/sourcegraph#18737

Linked PRs

Merge in this order, because the two deploy PRs need a prometheus image with the fix:

1. Image fix: sourcegraph/sourcegraph#18737
2. Helm: fix/observability: Prometheus read only root volume #1000
3. Docker Compose: fix/observability: Prometheus read only root volume deploy-sourcegraph-docker#1479

Supersedes #858; the other earlier attempts are listed in sourcegraph/sourcegraph#18737

Amp threads

• Design, implementation, and the EKS upgrade test
• Runtime test in Docker and kind, with commands and output
• Earlier Helm-only workaround

Changelog

• prometheus now runs with a read-only root filesystem