Skip to content

feat(cli): log database migration progress on server startup - #31790

Open
kirstaeff wants to merge 8 commits into
kirstaeff/ent-235-migration-progress-heartbeatfrom
kirstaeff/ent-237-migration-logging-server-integration
Open

kirstaeff wants to merge 8 commits into
kirstaeff/ent-235-migration-progress-heartbeatfrom
kirstaeff/ent-237-migration-logging-server-integration

Conversation

@kirstaeff

@kirstaeff kirstaeff commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

coder server now logs database migration progress while it starts: ConnectToPostgres passes a logger named migrations to the migration function. Commands that connect without migrating are unchanged.

Output from two instances upgrading a v2.37.0 database (schema 585, 54 pending migrations) on a disposable PostgreSQL, while another session holds a lock on templates for 30 seconds. Instance A migrates and its heartbeat names the blocking session (PID 171); instance B waits for the migration lock and names A's connection (PID 173, coder-migrate/...). Steps 2 to 54 are omitted.

[A] [info]  migrations: database migration required  from_version=585  target_version=639  pending=54
[A] [info]  migrations: applying update 1 of 54  version=586  name=template_allow_workspace_renames  position=1  total=54
[B] [info]  migrations: waiting for database migration lock  phase=setup
[A] [info]  migrations: still applying update 1 of 54  elapsed_ms=10006  state=waiting_on_lock  wait_event=relation  blocked_by=171  version=586  name=template_allow_workspace_renames  position=1  total=54
[B] [info]  migrations: still waiting for database migration lock  elapsed_ms=10007  state=waiting_on_lock  wait_event=advisory  blocked_by=173
[A] [info]  migrations: still applying update 1 of 54  elapsed_ms=20001  state=waiting_on_lock  wait_event=relation  blocked_by=171  version=586  name=template_allow_workspace_renames  position=1  total=54
[B] [info]  migrations: still waiting for database migration lock  elapsed_ms=20002  state=waiting_on_lock  wait_event=advisory  blocked_by=173
[A] [info]  migrations: database migration applied, pending commit  version=586  name=template_allow_workspace_renames  position=1  total=54  duration_ms=28973
[B] [info]  migrations: acquired database migration lock  phase=setup  waited_ms=26128

The final outcome line ("committed" or "rolled back") follows in ENT-236. Until then, a failed migration still returns the existing error.

Closes ENT-237

This PR was written with Claude Code. I reviewed every change and ran the upgrade above myself; both instances started and the schema ended at version 639.

ConnectToPostgres now passes a logger named "migrations" to the
migration function, so coder server reports migration progress, lock
waits and heartbeats while it starts. Commands that connect without
migrating are unchanged.

Part of ENT-237.
@kirstaeff
kirstaeff added this pull request to stack #31791 November 5, 2026 15:35
@linear-code

linear-code Bot commented Nov 5, 2026

Copy link
Copy Markdown

ENT-237

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

…rstaeff/ent-237-migration-logging-server-integration
@kirstaeff
kirstaeff marked this pull request as ready for review November 5, 2026 17:30
@coderagents

coderagents Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Documentation Check

Blocking

  • docs/install/operate/upgrade/best-practices.md - Name the migrations logger so operators can filter migration progress lines (reverted in b7bd019)
  • docs/install/operate/upgrade/best-practices.md - Resolve the "migration lock" vs "database lock" term drift (reverted in b7bd019)
  • docs/install/operate/upgrade/best-practices.md (open) - The change makes coder server log database migration progress under a migrations logger on every startup, a default-visible user-facing behavior, but the docs that described it were removed in b7bd019. No page explains the migrations logger, its lines (database migration required, applying update N of M, waiting for database migration lock, still applying update N of M, still waiting for database migration lock, database migration applied, pending commit, and the Debug database schema is up to date), or the 10-second heartbeat.

New documentation needed

  • None.

Design notes

  • None.

Related open PRs

Findings for agents
- id: coverage-1
  persona: coverage
  class: missing
  blocking: true
  file: docs/install/operate/upgrade/best-practices.md
  line: 80
  quote: "# Check pod logs for migration progress"
  evidence:
    - cli/server.go:2779
    - cli/server.go:3247
    - coderd/database/migrations/migrate.go:155
    - coderd/database/migrations/reporter.go:22,150,155,201,242,262,267,296,419
  rule: docs/.style/content-guidelines.md#documentation-lands-with-the-change
  problem: The change makes coder server log database migration progress under a migrations logger on every startup, a default-visible user-facing behavior, but no docs page describes the logger, its lines, or the 10-second heartbeat, because the docs that covered it were reverted in commit b7bd019.
  fix: Restore and keep a section in docs/install/operate/upgrade/best-practices.md (near the existing "Check pod logs for migration progress" line 80) documenting that coder server logs migration progress under a migrations logger, what each line means, and the kubectl logs -l app.kubernetes.io/name=coder -n <namespace> | grep 'migrations:' filter.
  verified: true

Automated review via Coder Agents. Reply to a finding's thread with the reason if the finding is wrong.

Name the migrations logger and its main lines in the upgrade best
practices, now that coder server logs migration progress on startup.

- `database migration required` shows how many migrations are pending.
- `applying update N of M` marks each migration as it starts.
- `waiting for database migration lock` means another session holds the migration lock, such as another replica that is applying the migrations.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: The prose introduces "the migration lock" for a concept the rest of the page already calls "database lock" (lines 28, 53, 123, 125, 135), violating one-term-per-concept. Keep the literal log string `waiting for database migration lock` in backticks, but rename the prose to "database lock". (Continuity Editor)

Evidence: docs/.style/style-guide/word-choice.md#one-term-per-concept; best-practices.md:28,53,123,125,135 use "database lock(s)", line 90 uses "the migration lock".

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are two different locks, so I kept both terms and defined the new one instead of renaming it.

  • The "database locks" elsewhere on this page are PostgreSQL locks on tables, held by old pods' open connections. They block a migration's SQL, and they are what blocked_by reports.
  • "The migration lock" is the lock Coder uses so that only one replica applies migrations at a time. It is what waiting for database migration lock is about.

Calling it a database lock would suggest that a replica waiting for the migration lock is blocked by old pods' table locks. Fixed in 1620f5f: the line now says what the migration lock is, and the blocked_by sentence uses "database lock" for table locks, matching the rest of the page.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right. The migration lock is Coder's transaction-scoped advisory lock (pg_advisory_xact_lock, txnmigrator.go:55) that serializes migrations so only one replica applies them, while the "database lock" in the HA/recovery context refers to relation/table locks old connections hold that block the migration's DDL. These are genuinely different locks, so keeping "the migration lock" (and defining it on line 90) alongside "database lock" (line 92) is correct. Resolved with 1620f5ff.


Look for events indicating `Liveness probe failed` or `Container coder failed liveness probe, will be restarted`.

When Coder applies database migrations at startup, it logs their progress under the `migrations` logger:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a concrete example log line so an operator can recognize the full migration-progress output in pod logs. The page lists message substrings but never shows a complete line with its fields (from_version, target_version, pending, position, total, version, name, elapsed_ms, state, wait_event, blocked_by). (Newcomer)

Example: migrations: still applying update 3 of 7 version=... name=... position=3 total=7 elapsed_ms=... state=waiting_on_lock blocked_by=6652

Evidence: coderd/database/migrations/reporter.go:242,262,296,491-498.

- `database migration required` shows how many migrations are pending.
- `applying update N of M` marks each migration as it starts.
- `waiting for database migration lock` means another session holds the migration lock, such as another replica that is applying the migrations.
- `still applying update N of M` and `still waiting for database migration lock` repeat every 10&nbsp;seconds while a migration or a wait lasts.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The documented 10-second heartbeat is a copy of defaultHeartbeatInterval (reporter.go:22), but no test pins it. Recommend a test that asserts newReporter without WithHeartbeatInterval yields defaultHeartbeatInterval (10s), so the documented value cannot silently drift. (Fact Checker)

Evidence: coderd/database/migrations/reporter.go:22,90-91,352; every test passes WithHeartbeatInterval(time.Second), so the default is unpinned.

- `database migration required` shows how many migrations are pending.
- `applying update N of M` marks each migration as it starts.
- `waiting for database migration lock` means another session holds the migration lock, such as another replica that is applying the migrations.
- `still applying update N of M` and `still waiting for database migration lock` repeat every 10&nbsp;seconds while a migration or a wait lasts.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add the diagnostic takeaway this section exists for: a steady 10-second "still applying update" heartbeat means the pod is alive and progressing, so a restart during migrations indicates a liveness-probe kill, not a stalled migration. (Newcomer)

Evidence: coderd/database/migrations/reporter.go:153-156,268-272 (the heartbeat is the liveness signal).

…rstaeff/ent-237-migration-logging-server-integration
The migration lock is Coder's lock that lets one replica migrate at a
time. The page's database locks are table locks held by other sessions,
which blocked_by reports. Also say what steady heartbeats mean during a
liveness probe restart.
- `still applying update N of M` and `still waiting for database migration lock` repeat every 10&nbsp;seconds while a migration or a wait lasts.
When a migration waits on a database lock, `blocked_by` lists the process IDs of the sessions that hold it.

As long as heartbeat lines keep appearing, Coder is alive and still working on the upgrade, so a restart at that point comes from the liveness probe, not from a stalled upgrade.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The inference "not from a stalled upgrade" overstates what the heartbeats support. Heartbeats also keep appearing during a database-lock wait (lines 91-92, reporter.go:153-156,268-271), so steady heartbeats confirm Coder was active when the probe restarted it but do not rule out a lock-blocked (stalled) upgrade. (Cartographer)

Suggest: "As long as heartbeat lines keep appearing, Coder is still running, so a restart at that point comes from the liveness probe rather than a crash; the upgrade may still be waiting on a lock."

- `still applying update N of M` and `still waiting for database migration lock` repeat every 10&nbsp;seconds while a migration or a wait lasts.
When a migration waits on a database lock, `blocked_by` lists the process IDs of the sessions that hold it.

As long as heartbeat lines keep appearing, Coder is alive and still working on the upgrade, so a restart at that point comes from the liveness probe, not from a stalled upgrade.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The term "heartbeat lines" is introduced here but is not tied to the still applying update N of M / still waiting for database migration lock lines described on line 91, so a new reader must infer what a "heartbeat line" is. (Newcomer)

Suggest rewording to reference the repeating lines by name, e.g. "As long as the repeating still applying / still waiting lines keep appearing every 10 seconds, ..."

Reverts 66acd2f and 1620f5f. The upgrade guide changes for
migration progress logs will follow in a separate pull request.
…ress-heartbeat' into kirstaeff/ent-237-migration-logging-server-integration
…rstaeff/ent-237-migration-logging-server-integration

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants