Fix broken database backup / restore commands - #2156

The new-environment and existing-environment restore procedures were
the same steps written twice. On Kubernetes they are now identical,
since the existing-environment steps no longer delete the database
volumes. On Docker Compose only the first step differed, so it becomes
a conditional step: stop the deployment and remove the volumes when
restoring into an existing one.

Also drop the stale `docker-compose ps` sample output from the restore
steps; it listed services that no longer exist.

Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1d197-7a9c-71fc-9998-6e55df9c47d0

The steps used `$pgsql_POD_NAME`, `$codeintel-db_POD_NAME`, and
`$codeinsights-db_POD_NAME`, which the page never defines. The last two
are not valid shell variables either: `$codeintel-db_POD_NAME` expands
`$codeintel` and leaves `-db_POD_NAME`. The databases are StatefulSets
with one replica in both the Helm chart and deploy-sourcegraph-k8s, so
their pods are always `pgsql-0`, `codeintel-db-0`, and
`codeinsights-db-0`.

`kubectl cp` also drops the `$NAMESPACE/` prefix, so every command
takes `-n $NAMESPACE` the same way, as the note says.

Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1d197-7a9c-71fc-9998-6e55df9c47d0

Scaling it to zero stops its connections just as well, and the final
step of applying the manifests scales it back up. Deleting it leaves
Helm and Kustomize users to recreate it.

Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1d197-7a9c-71fc-9998-6e55df9c47d0

The docs site doesn't wrap code blocks, so these commands scrolled sideways: up to 253 characters, in a block that fits 75 at desktop width. Each line is now at most 67 characters. The commands pass the same arguments as before.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1d140-e6de-7173-8b4f-187543b7833c
Co-authored-by: Amp <amp@ampcode.com>

Backup:
- Drop scaling down the frontend. pg_dump takes a consistent snapshot
  of each database while services stay connected, and scaling down only
  the frontend left the worker and others connected anyway. The backup
  also never scaled the frontend back up.
- Step A asks for a baseline of the running pods and their replicas.
- The amcheck step says it blocks writes to each table while it checks
  that table's indexes, since services now keep running.

Restore:
- Run pg_dump and psql directly with kubectl exec, without bash -c, so
  no quotes nest.
- Deleting the dump files from the volumes is its own step, after all
  three restores succeed.
- Restart the services with helm upgrade, which sets the replicas that
  the scale-down step zeroed back to the values; Kustomize users keep
  the applying manifests link.

Run as written on m6 (Helm 8.2.0, Helm 4.3.1), with the backup taken
while every service ran: all exit codes 0, no ERROR lines, the three
re-dumps identical to the backup, GraphQL data matches the seeded data,
and every workload back at 1/1.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1d197-7a9c-71fc-9998-6e55df9c47d0
Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1d197-7a9c-71fc-9998-6e55df9c47d0

Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1d197-7a9c-71fc-9998-6e55df9c47d0

"Access the database" showed `pgsql-76a4bfcd64-rt4cn`, a pod name from
when pgsql was a Deployment; it is a StatefulSet, so its pod is always
`pgsql-0`. The restore's wait step now names the database pods too.

Drop the "Migrate to Kustomize" and "Upgrade" sections, which only
linked elsewhere; Kustomize is being deprecated.

Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1d197-7a9c-71fc-9998-6e55df9c47d0

The amcheck SQL had no command to run it. Save it as amcheck.sql and
pipe it into psql on each database pod, with ON_ERROR_STOP=1 so a
failed check exits 3 instead of 0, and stdout discarded so only errors
show. The restore points at the backup's commands instead of repeating
the query.

- `create extension if not exists`: amcheck already exists in all
  three databases, so the plain form failed every time.
- `c.relkind = 'i'`: on codeintel-db the check failed with "only B-Tree
  indexes are supported as targets for verification" on the
  partitioned index codeintel_documents_pkey.

Step A now asks to verify the pods, especially the three databases.

Tested on m6: exit 0 on all three databases; without the relkind
filter, codeintel-db prints the error and exits 3.

Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1d197-7a9c-71fc-9998-6e55df9c47d0

Drop the backup step that stopped the deployment and started only the
databases. pg_dump takes a consistent snapshot of each database while
services stay connected, and the backup never started the deployment
again, so following it left Sourcegraph down. This matches the
Kubernetes page.

Use the `docker compose` plugin instead of the old standalone
`docker-compose` command in the examples.

Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1d197-7a9c-71fc-9998-6e55df9c47d0

AGENTS.md asks for sequentially numbered ordered Markdown lists. The
Kubernetes page lettered its steps, the only page in the repo to do so,
and the Docker Compose page escaped its numbers (`1\.`) so they were not
lists. Both now use `1.`, `2.`, `3.`.

The Kubernetes restore also ends with the Docker Compose page's last
step: browse to Sourcegraph and check the configuration came back.

Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1d197-7a9c-71fc-9998-6e55df9c47d0

The sample was docker-compose v1 output from an old release, listing
jaeger and query-runner, which 8.2.0 no longer runs. Replace it with
output captured from a fresh 8.2.0 stack. The command prints only names
and status, since plain `docker compose ps` prints pinned image digests
and runs past 200 characters.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1d197-7a9c-71fc-9998-6e55df9c47d0
Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1d197-7a9c-71fc-9998-6e55df9c47d0

The same two fixes as the Kubernetes operations page:
- `create extension if not exists amcheck`: the plain form errors on
  databases that already have amcheck.
- `c.relkind = 'i'`: without it the check stops on codeintel-db with
  "only B-Tree indexes are supported as targets for verification" at
  the partitioned index codeintel_documents_pkey, which reads like
  corruption and leaves the remaining indexes unchecked.

Tested on m6 (8.2.0): both pages' queries exit 0 with no errors on all
three databases.

Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1d197-7a9c-71fc-9998-6e55df9c47d0

docker exec runs them directly, as the Kubernetes page does with
kubectl exec, so no quotes nest.

Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1d197-7a9c-71fc-9998-6e55df9c47d0

It covers a bug in PostgreSQL 14.0 to 14.3. Sourcegraph now requires
PostgreSQL 16, so no supported deployment can hit it. Remove the two
links to it, and redirect its old URLs to "How to rebuild corrupt
Postgres indexes".

Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1d197-7a9c-71fc-9998-6e55df9c47d0

…ns page

It only restated that the frontend migrates on startup and linked to the
migrator docs.

Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1d197-7a9c-71fc-9998-6e55df9c47d0

Drop the trailing comments, which only repeated the container name and
pushed the lines past the width of a code block.

Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1d197-7a9c-71fc-9998-6e55df9c47d0

The restore drops and recreates each database, and
db-only-migrate.docker-compose.yaml starts only the three databases, so
nothing else is connected to block the dump's DROP DATABASE. Deleting
the volumes is unnecessary, as on the Kubernetes page; stopping the
deployment is still needed so nothing writes during the restore.

Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1d197-7a9c-71fc-9998-6e55df9c47d0

The docker compose ps sample now comes from an 8.2.0 stack started with
the Jaeger and executor overrides, so it lists `jaeger` and `executor`.
Plain `docker compose ps` lists them too.

Plain `docker compose down` and `up -d` ignore those override files:
jaeger and executor kept running through a test restore, and the final
`up -d` recreated the frontend without `JAEGER_SERVER_URL`. The restore
now says to pass the same `-f` options as when starting Sourcegraph.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1d197-7a9c-71fc-9998-6e55df9c47d0
Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1d197-7a9c-71fc-9998-6e55df9c47d0