Repository navigation
Conversation
ConnectToPostgres now passes a logger named "migrations" to the migration function, so coder server reports migration progress, lock waits and heartbeats while it starts. Commands that connect without migrating are unchanged. Part of ENT-237.
|
All contributors have signed the CLA ✍️ ✅ |
…rstaeff/ent-237-migration-logging-server-integration
Documentation CheckBlocking
New documentation needed
Design notes
Related open PRs
Findings for agents- id: coverage-1
persona: coverage
class: missing
blocking: true
file: docs/install/operate/upgrade/best-practices.md
line: 80
quote: "# Check pod logs for migration progress"
evidence:
- cli/server.go:2779
- cli/server.go:3247
- coderd/database/migrations/migrate.go:155
- coderd/database/migrations/reporter.go:22,150,155,201,242,262,267,296,419
rule: docs/.style/content-guidelines.md#documentation-lands-with-the-change
problem: The change makes coder server log database migration progress under a migrations logger on every startup, a default-visible user-facing behavior, but no docs page describes the logger, its lines, or the 10-second heartbeat, because the docs that covered it were reverted in commit b7bd019.
fix: Restore and keep a section in docs/install/operate/upgrade/best-practices.md (near the existing "Check pod logs for migration progress" line 80) documenting that coder server logs migration progress under a migrations logger, what each line means, and the kubectl logs -l app.kubernetes.io/name=coder -n <namespace> | grep 'migrations:' filter.
verified: trueAutomated review via Coder Agents. Reply to a finding's thread with the reason if the finding is wrong. |
Name the migrations logger and its main lines in the upgrade best practices, now that coder server logs migration progress on startup.
|
|
||
| - `database migration required` shows how many migrations are pending. | ||
| - `applying update N of M` marks each migration as it starts. | ||
| - `waiting for database migration lock` means another session holds the migration lock, such as another replica that is applying the migrations. |
There was a problem hiding this comment.
Blocking: The prose introduces "the migration lock" for a concept the rest of the page already calls "database lock" (lines 28, 53, 123, 125, 135), violating one-term-per-concept. Keep the literal log string `waiting for database migration lock` in backticks, but rename the prose to "database lock". (Continuity Editor)
Evidence: docs/.style/style-guide/word-choice.md#one-term-per-concept; best-practices.md:28,53,123,125,135 use "database lock(s)", line 90 uses "the migration lock".
There was a problem hiding this comment.
These are two different locks, so I kept both terms and defined the new one instead of renaming it.
- The "database locks" elsewhere on this page are PostgreSQL locks on tables, held by old pods' open connections. They block a migration's SQL, and they are what
blocked_byreports. - "The migration lock" is the lock Coder uses so that only one replica applies migrations at a time. It is what
waiting for database migration lockis about.
Calling it a database lock would suggest that a replica waiting for the migration lock is blocked by old pods' table locks. Fixed in 1620f5f: the line now says what the migration lock is, and the blocked_by sentence uses "database lock" for table locks, matching the rest of the page.
There was a problem hiding this comment.
You're right. The migration lock is Coder's transaction-scoped advisory lock (pg_advisory_xact_lock, txnmigrator.go:55) that serializes migrations so only one replica applies them, while the "database lock" in the HA/recovery context refers to relation/table locks old connections hold that block the migration's DDL. These are genuinely different locks, so keeping "the migration lock" (and defining it on line 90) alongside "database lock" (line 92) is correct. Resolved with 1620f5ff.
|
|
||
| Look for events indicating `Liveness probe failed` or `Container coder failed liveness probe, will be restarted`. | ||
|
|
||
| When Coder applies database migrations at startup, it logs their progress under the `migrations` logger: |
There was a problem hiding this comment.
Add a concrete example log line so an operator can recognize the full migration-progress output in pod logs. The page lists message substrings but never shows a complete line with its fields (from_version, target_version, pending, position, total, version, name, elapsed_ms, state, wait_event, blocked_by). (Newcomer)
Example: migrations: still applying update 3 of 7 version=... name=... position=3 total=7 elapsed_ms=... state=waiting_on_lock blocked_by=6652
Evidence: coderd/database/migrations/reporter.go:242,262,296,491-498.
| - `database migration required` shows how many migrations are pending. | ||
| - `applying update N of M` marks each migration as it starts. | ||
| - `waiting for database migration lock` means another session holds the migration lock, such as another replica that is applying the migrations. | ||
| - `still applying update N of M` and `still waiting for database migration lock` repeat every 10 seconds while a migration or a wait lasts. |
There was a problem hiding this comment.
The documented 10-second heartbeat is a copy of defaultHeartbeatInterval (reporter.go:22), but no test pins it. Recommend a test that asserts newReporter without WithHeartbeatInterval yields defaultHeartbeatInterval (10s), so the documented value cannot silently drift. (Fact Checker)
Evidence: coderd/database/migrations/reporter.go:22,90-91,352; every test passes WithHeartbeatInterval(time.Second), so the default is unpinned.
| - `database migration required` shows how many migrations are pending. | ||
| - `applying update N of M` marks each migration as it starts. | ||
| - `waiting for database migration lock` means another session holds the migration lock, such as another replica that is applying the migrations. | ||
| - `still applying update N of M` and `still waiting for database migration lock` repeat every 10 seconds while a migration or a wait lasts. |
There was a problem hiding this comment.
Add the diagnostic takeaway this section exists for: a steady 10-second "still applying update" heartbeat means the pod is alive and progressing, so a restart during migrations indicates a liveness-probe kill, not a stalled migration. (Newcomer)
Evidence: coderd/database/migrations/reporter.go:153-156,268-272 (the heartbeat is the liveness signal).
…rstaeff/ent-237-migration-logging-server-integration
The migration lock is Coder's lock that lets one replica migrate at a time. The page's database locks are table locks held by other sessions, which blocked_by reports. Also say what steady heartbeats mean during a liveness probe restart.
| - `still applying update N of M` and `still waiting for database migration lock` repeat every 10 seconds while a migration or a wait lasts. | ||
| When a migration waits on a database lock, `blocked_by` lists the process IDs of the sessions that hold it. | ||
|
|
||
| As long as heartbeat lines keep appearing, Coder is alive and still working on the upgrade, so a restart at that point comes from the liveness probe, not from a stalled upgrade. |
There was a problem hiding this comment.
The inference "not from a stalled upgrade" overstates what the heartbeats support. Heartbeats also keep appearing during a database-lock wait (lines 91-92, reporter.go:153-156,268-271), so steady heartbeats confirm Coder was active when the probe restarted it but do not rule out a lock-blocked (stalled) upgrade. (Cartographer)
Suggest: "As long as heartbeat lines keep appearing, Coder is still running, so a restart at that point comes from the liveness probe rather than a crash; the upgrade may still be waiting on a lock."
| - `still applying update N of M` and `still waiting for database migration lock` repeat every 10 seconds while a migration or a wait lasts. | ||
| When a migration waits on a database lock, `blocked_by` lists the process IDs of the sessions that hold it. | ||
|
|
||
| As long as heartbeat lines keep appearing, Coder is alive and still working on the upgrade, so a restart at that point comes from the liveness probe, not from a stalled upgrade. |
There was a problem hiding this comment.
The term "heartbeat lines" is introduced here but is not tied to the still applying update N of M / still waiting for database migration lock lines described on line 91, so a new reader must infer what a "heartbeat line" is. (Newcomer)
Suggest rewording to reference the repeating lines by name, e.g. "As long as the repeating still applying / still waiting lines keep appearing every 10 seconds, ..."
…ress-heartbeat' into kirstaeff/ent-237-migration-logging-server-integration
…rstaeff/ent-237-migration-logging-server-integration
coder servernow logs database migration progress while it starts:ConnectToPostgrespasses a logger namedmigrationsto the migration function. Commands that connect without migrating are unchanged.Output from two instances upgrading a v2.37.0 database (schema 585, 54 pending migrations) on a disposable PostgreSQL, while another session holds a lock on
templatesfor 30 seconds. Instance A migrates and its heartbeat names the blocking session (PID 171); instance B waits for the migration lock and names A's connection (PID 173,coder-migrate/...). Steps 2 to 54 are omitted.The final outcome line ("committed" or "rolled back") follows in ENT-236. Until then, a failed migration still returns the existing error.
Closes ENT-237
This PR was written with Claude Code. I reviewed every change and ran the upgrade above myself; both instances started and the schema ended at version 639.