Skip to content

feat: log migration progress and heartbeat - #31789

Open
kirstaeff wants to merge 15 commits into
mainfrom
kirstaeff/ent-235-migration-progress-heartbeat
Open

kirstaeff wants to merge 15 commits into
mainfrom
kirstaeff/ent-235-migration-progress-heartbeat

Conversation

@kirstaeff

@kirstaeff kirstaeff commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Database migrations run during server startup without any output, so administrators can't tell which migration is running, whether it is waiting on a lock, or how far the upgrade has got.

This adds a reporter to the migration driver. It logs the pending migrations, each step with its position and duration, the migration lock, and a heartbeat every 10 seconds while a lock wait or a step lasts, with the state of the migration connection from pg_stat_activity and the blocking sessions when it waits on a lock. Every run executes the same statements: the migration transaction is named coder-migrate/<version> and takes the lock with one blocking pg_advisory_xact_lock. The logger only decides what is written. Heartbeat queries run on a separate connection and never delay the migration.

Up now takes options: WithFS replaces UpWithFS, and WithLogger sets the logger. Without WithLogger, the reporter logs to a zero slog.Logger, which discards every line. Down, EnsureClean and Stepper use the same driver, so their transactions are named coder-migrate/<version> too. ConnectToPostgres takes the new Up signature. Server startup turns reporting on in the next PR of this stack, which shows the output from a real upgrade.

Reporter tests compare the log output with golden files in coderd/database/migrations/testdata/reporter, regenerated with make gen/golden-files.

Differences from RFC 1:

  • A separate reporter holds the progress state, the heartbeat and the PostgreSQL queries, and the driver only calls it. There is no Progress type, UpWithLogger or runUp.
  • Adds a "database migration required" line, and names the migration connection coder-migrate/<version> so other instances and administrators can recognize it.
  • Lines with nothing to act on are logged at debug level: acquiring and acquired the migration lock, "database schema is up to date", and the per-step lines of a fresh install. A normal restart adds no line at info, and a real lock wait shows up through the heartbeat. Heartbeats stay at info.
  • Messages are constant, for example "applying database migration", with the position in position and total fields instead of "applying update x of y".

Closes ENT-235

This PR was written with Claude Code. I reviewed every change, and verified it with the tests in this PR against PostgreSQL and with real upgrades of a v2.37.0 database on a disposable PostgreSQL (output in the next PR of this stack). The changes for the first review round were made with Coder Agents.

Database migrations run during server startup without any progress
output. Add an optional reporter to the migration driver that logs
which migration is running, its position among the pending ones, how
long each one took, waits for the migration lock, and a heartbeat every
10 seconds read from pg_stat_activity, naming the blocking session when
the migration waits on a lock.

Up now takes options: WithFS replaces UpWithFS and WithLogger enables
reporting. Without WithLogger, migrations run the same queries and
return the same errors as before. ConnectToPostgres takes the new Up
signature; server startup does not pass a logger yet.

Part of ENT-235.
@kirstaeff
kirstaeff added this pull request to stack #31791 November 5, 2026 15:35
@linear-code

linear-code Bot commented Nov 5, 2026

Copy link
Copy Markdown

ENT-235

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@kirstaeff kirstaeff changed the title kirstaeff/ent 235 migration progress heartbeat feat: log migration progress and heartbeat Nov 5, 2026
@kirstaeff

Copy link
Copy Markdown
Collaborator Author

I have read the CLA Document and I hereby sign the CLA

cdrci2 added a commit to coder/cla that referenced this pull request Nov 5, 2026
Most server starts have nothing to migrate. Logging that at info added
a line to every start, which TestServer/SpammyLogs guards against.

@spikecurtis spikecurtis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't reviewed all the tests, but I suspect they'll change anyway if you take my suggestions.

Comment thread coderd/database/migrations/reporter.go Resolved
Comment thread coderd/database/migrations/reporter.go Resolved
Comment thread coderd/database/migrations/reporter.go Resolved
Comment thread coderd/database/migrations/reporter.go Resolved
Comment thread coderd/database/migrations/reporter.go Resolved
Comment thread coderd/database/migrations/reporter.go Resolved
@spikecurtis

Copy link
Copy Markdown
Collaborator

I had another thought about the tests. All the inline assertions about the logs are very verbose and difficult to read.

Maybe what we do instead is the golden file pattern. You'll need to create a test-only sink that emits the logs without test-time variable information, like the timestamp, but still outputs a human-readable format. Then we store the expected output in a golden file we check in and do a straight byte comparison to assert the outputs. We have a make target for other tests that regenerate the golden files, so its easy to make changes to the logging code.

Without WithLogger the reporter logs to a zero slog.Logger, so every
caller runs the same code path and no method needs a nil check.
…e blocking call

Lock now runs the same statements on every run: name the connection,
identify the backend, then a single pg_advisory_xact_lock. The try-lock
only decided whether to log a wait; the heartbeat already reports that.
Acquiring and acquired are logged at debug so a normal start stays quiet.
…der a run context

Every heartbeat derives its context from a run context that close
cancels, so a heartbeat started before or after close is stopped and no
closed flag is needed. A tick re-checks its context and emits under r.mu,
and stop cancels under r.mu, so no line is written after stop returns
without ever waiting for an in-flight pg_stat_activity query.
…tructured fields

Messages no longer embed the step position, so lines of one kind can be
filtered. blocked_by is logged as the PID array, and a NULL wait type is
treated as running.
The reporter tests write logs through a sink without timestamps, with
backend PIDs replaced by names, and compare the output with golden files
in coderd/database/migrations/testdata/reporter. make gen/golden-files
regenerates them.
…cal to Lock

lockWaitStarted returns the heartbeat's stop function and start time, so
Lock stops the heartbeat it started and the reporter no longer stores it.
Also shorten the reporter and WithLogger doc comments.
@kirstaeff
kirstaeff temporarily deployed to main November 13, 2026 11:07 — with GitHub Actions Inactive
@kirstaeff
kirstaeff temporarily deployed to main November 13, 2026 11:11 — with GitHub Actions Inactive

@spikecurtis spikecurtis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Golden files for the tests worked out really well, I think.

I still think the concurrency in the reporter is hard to follow and verify, comment inline.

Comment thread coderd/database/migrations/reporter.go Resolved
…ves on done

FreshInstall duplicated StepBlockedByTableLock; WaitsForMigrationLock
and HeartbeatCheckFails already cover a run from an empty schema.
RequireReceive fails the test if done is ever closed, where TryReceive
would return a nil error.

@spikecurtis spikecurtis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some small nits inline, but I don't need to review again before merging.

Comment thread coderd/database/migrations/reporter.go Resolved
Comment thread coderd/database/migrations/reporter.go Resolved
Comment thread coderd/database/migrations/reporter.go Resolved

This branch was previously deployed

1 inactive deployment
main — 3adf285c Deployed Nov 18, 2026 by kirstaeff via cla #259754
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants