sourcegraph: read-only root filesystem for blobstore, cadvisor, otel, and pgsql-exporter - #1003

Open issues

Found in review on 2026-11-13 (review thread), at a6fa5d5. The change passed its cluster test (see "Test plan"); this PR is a draft until the first is fixed.

1. Conflicts with main in README.md. Fix: rebase and regenerate it with ./scripts/helm-docs.sh
2. A release that already mounts its own volume at /tmp on blobstore collides with the new tmpdir volume. The API rejects a new Deployment with two mounts at /tmp (mountPath: must be unique), or with a second volume named tmpdir. Checked with kubectl create --dry-run=server; a real helm upgrade of such a release was not run. Fix: say so in the changelog
3. No test asserts the new defaults or the tmpdir volume
4. Small ones:
   • the changelog does not name the keys that revert otel-agent, otel-collector, and the exporter sidecar (openTelemetry.agent, openTelemetry.gateway, postgresExporter)
   • "every container except prometheus" leaves out the helm test pod, which sets no security context

Problem

Some customers need a read-only root filesystem on every container

These containers didn't set readOnlyRootFilesystem: true:

• blobstore, cadvisor, otel-agent, and otel-collector
• The pgsql-exporter sidecar of pgsql, codeintel-db, and codeinsights-db
  • The templates read postgresExporter.containerSecurityContext, but values.yaml never defined it

Solution

Set readOnlyRootFilesystem: true on each of them:

• blobstore also gets an emptyDir at /tmp
• cadvisor stays privileged
• values.yaml defines postgresExporter.containerSecurityContext, which also sets allowPrivilegeEscalation: false
  • It sets no runAsUser or runAsGroup, so the sidecar keeps running as its database pod's user
• No image change is needed

Not changed:

• prometheus, which is fix/observability: Prometheus read only root volume #1000, and waits on an image change

Test plan

• helm lint charts/sourcegraph and helm unittest charts/sourcegraph pass
• helm template with default values renders readOnlyRootFilesystem: true on every container except prometheus, and the tmpdir volume on blobstore

Helm

On the EKS test instance m5, on 2026-11-13, by upgrading from main to this branch merged onto main, with cadvisor.enabled: true:

• All pods Ready. blobstore, otel-collector, the three databases, and each cadvisor and otel-agent pod were replaced once
• Of the running containers, only prometheus and the k8s executor (a separate chart) lack a read-only root filesystem
• No read-only or permission errors in the logs of the changed containers
• blobstore: /tmp is writable, the rest of the root filesystem is not, and a search job completed and wrote its results to blobstore
• The three exporter sidecars report pg_up 1 with no scrape errors, running as their database's user
• Each cadvisor pod queried serves container metrics, and otel-agent and otel-collector start cleanly
• Search works

Not run:

• OpenShift, and the manual testing process

Docker Compose

• CI in docker-compose: Run services with a read-only root filesystem deploy-sourcegraph-docker#1470 starts the same images with read_only: true
  • That is a smoke test of a different deployment

Linked PRs

• fix/observability: Prometheus read only root volume #1000: the same for prometheus
• docker-compose: Run services with a read-only root filesystem deploy-sourcegraph-docker#1470: the same images with read_only: true in Docker Compose
• Create global container and pod security context configs #818: conflicts with this PR in values.yaml, so whichever merges second needs a rebase
• Add standalone pgsql-exporter for external databases #969: adds a standalone pgsql-exporter Deployment, which picks up this default

Amp threads

• Design and implementation

Changelog

• blobstore, cadvisor, otel-agent, otel-collector, and the pgsql-exporter sidecars now run with a read-only root filesystem
  • Set <serviceName>.containerSecurityContext.readOnlyRootFilesystem: false to revert
• Added postgresExporter.containerSecurityContext, which the templates already read but values.yaml did not define