Repository navigation
Conversation
Database migrations run during server startup without any progress output. Add an optional reporter to the migration driver that logs which migration is running, its position among the pending ones, how long each one took, waits for the migration lock, and a heartbeat every 10 seconds read from pg_stat_activity, naming the blocking session when the migration waits on a lock. Up now takes options: WithFS replaces UpWithFS and WithLogger enables reporting. Without WithLogger, migrations run the same queries and return the same errors as before. ConnectToPostgres takes the new Up signature; server startup does not pass a logger yet. Part of ENT-235.
|
All contributors have signed the CLA ✍️ ✅ |
|
I have read the CLA Document and I hereby sign the CLA |
Most server starts have nothing to migrate. Logging that at info added a line to every start, which TestServer/SpammyLogs guards against.
spikecurtis
left a comment
There was a problem hiding this comment.
I haven't reviewed all the tests, but I suspect they'll change anyway if you take my suggestions.
|
I had another thought about the tests. All the inline assertions about the logs are very verbose and difficult to read. Maybe what we do instead is the golden file pattern. You'll need to create a test-only sink that emits the logs without test-time variable information, like the timestamp, but still outputs a human-readable format. Then we store the expected output in a golden file we check in and do a straight byte comparison to assert the outputs. We have a make target for other tests that regenerate the golden files, so its easy to make changes to the logging code. |
Without WithLogger the reporter logs to a zero slog.Logger, so every caller runs the same code path and no method needs a nil check.
…e blocking call Lock now runs the same statements on every run: name the connection, identify the backend, then a single pg_advisory_xact_lock. The try-lock only decided whether to log a wait; the heartbeat already reports that. Acquiring and acquired are logged at debug so a normal start stays quiet.
…der a run context Every heartbeat derives its context from a run context that close cancels, so a heartbeat started before or after close is stopped and no closed flag is needed. A tick re-checks its context and emits under r.mu, and stop cancels under r.mu, so no line is written after stop returns without ever waiting for an in-flight pg_stat_activity query.
…tructured fields Messages no longer embed the step position, so lines of one kind can be filtered. blocked_by is logged as the PID array, and a NULL wait type is treated as running.
The reporter tests write logs through a sink without timestamps, with backend PIDs replaced by names, and compare the output with golden files in coderd/database/migrations/testdata/reporter. make gen/golden-files regenerates them.
…cal to Lock lockWaitStarted returns the heartbeat's stop function and start time, so Lock stops the heartbeat it started and the reporter no longer stores it. Also shorten the reporter and WithLogger doc comments.
spikecurtis
left a comment
There was a problem hiding this comment.
Golden files for the tests worked out really well, I think.
I still think the concurrency in the reporter is hard to follow and verify, comment inline.
…ves on done FreshInstall duplicated StepBlockedByTableLock; WaitsForMigrationLock and HeartbeatCheckFails already cover a run from an empty schema. RequireReceive fails the test if done is ever closed, where TryReceive would return a nil error.
… and drop the reporter mutex
…artbeat and simplify the reporter
…r_internal_test.go
…ration-progress-heartbeat
spikecurtis
left a comment
There was a problem hiding this comment.
Some small nits inline, but I don't need to review again before merging.
… store the heartbeat context
Database migrations run during server startup without any output, so administrators can't tell which migration is running, whether it is waiting on a lock, or how far the upgrade has got.
This adds a reporter to the migration driver. It logs the pending migrations, each step with its position and duration, the migration lock, and a heartbeat every 10 seconds while a lock wait or a step lasts, with the state of the migration connection from
pg_stat_activityand the blocking sessions when it waits on a lock. Every run executes the same statements: the migration transaction is namedcoder-migrate/<version>and takes the lock with one blockingpg_advisory_xact_lock. The logger only decides what is written. Heartbeat queries run on a separate connection and never delay the migration.Upnow takes options:WithFSreplacesUpWithFS, andWithLoggersets the logger. WithoutWithLogger, the reporter logs to a zeroslog.Logger, which discards every line.Down,EnsureCleanandStepperuse the same driver, so their transactions are namedcoder-migrate/<version>too.ConnectToPostgrestakes the newUpsignature. Server startup turns reporting on in the next PR of this stack, which shows the output from a real upgrade.Reporter tests compare the log output with golden files in
coderd/database/migrations/testdata/reporter, regenerated withmake gen/golden-files.Differences from RFC 1:
Progresstype,UpWithLoggerorrunUp.coder-migrate/<version>so other instances and administrators can recognize it.positionandtotalfields instead of "applying update x of y".Closes ENT-235
This PR was written with Claude Code. I reviewed every change, and verified it with the tests in this PR against PostgreSQL and with real upgrades of a v2.37.0 database on a disposable PostgreSQL (output in the next PR of this stack). The changes for the first review round were made with Coder Agents.