Repository navigation
Conversation
This comment has been minimized.
This comment has been minimized.
|
Chat: Review posted | View chat Review historydeep-review v0.13.0 | Round 3 | Last posted: Round 3, 19 findings (2 P1, 2 P3, 5 P4, 10 Nit), APPROVE. Review Finding inventoryFinding inventory: PR #31167Findings
Contested and acknowledgedCRF-3 (P1, 000609_chat_automations.up.sql:125) - v2.38 upgrade cancelled at 000609 when a waiter's single deadlock check already ran
CRF-4 (P1, 000615_remove_chat_automations.up.sql:32) - v2.38 upgrade cancelled at 000615 when a users-then-chats reader waited past deadlock_timeout
Round logRound 1Netero: 1 P3, 1 Nit. No P0, panel proceeds. Round 2Churn guard PROCEED: 9 author fixed, CRF-3 and CRF-4 contested. Netero: no findings; CRF-1 and CRF-2 fixes verified. Round 3Churn guard PROCEED: CRF-18 to CRF-26 all author fixed (a165b0b and PR description). Author replied that the R2 out-of-scope items will not be ticketed from this PR; they stay Out of scope (body) and need a human decision. Author agreed the stated reason in CRF-27 was wrong and gave a new one (the case adds no coverage: reintroducing a retry block already fails the existing test). Author added the preview image range to the description (CRF-29). About deep-reviewCRF = Coder Review Finding (P0-P4, Nit, Note)
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
58f9a29 to
253e751
Compare
|
Answers to the remaining asks in the R1 review, so none is left open. This replaces the last sentence of my CRF-3 reply.
|
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
coder server runs all pending migrations in one transaction. On a release upgrade, 000607 and 000608 lock chats before 000609 runs, so a chat message insert holds chat_messages while it waits for chats, and 000609 needs chat_messages. Postgres checks a waiting request for a deadlock once, after deadlock_timeout. The lock-retry block in 000609 pauses between short lock waits, so a check that lands in a pause finds no deadlock, the insert keeps its lock, and the upgrade fails after 2 minutes. Without the block, Postgres cancels the insert and the upgrade continues. Remove the block from 000609 and 000611, which returns them to their content before #30297, and from both directions of 000615. No release contains these migrations. TestMigrationChatTablesConcurrentTraffic tested what the blocks gave when their migration takes the first chat lock of an upgrade. Replace it with a test of the v2.38 upgrade during chat message inserts.
Give each insert its own deadlock_timeout instead of sleeping between inserts. The checks still land at different points of a lock-retry cycle, and all of them fire after 000609 starts waiting and well before its own check, so Postgres cancels inserts, not the migration.
Run the upgrade test for 000609 (chat message inserts) and 000615 (transactions that read users and then chats). Give the requests a 2s deadlock_timeout and the migration a 10s one, so the requests' checks always find the migration waiting regardless of runner speed, and require every request to be cancelled as the deadlock victim. The doc comment now states that the test covers only requests that start waiting shortly before the migration needs their lock.
Start all chat requests before waiting for them to block, so their deadlock checks are spaced by their deadlock_timeout values alone. A retry loop that pauses 100ms then cannot avoid all of them, instead of depending on how quickly each request was observed blocking.
The doc comment said Postgres cancels the migration "with or without lock retries", but a migration that retries 100ms lock waits never reaches its own deadlock check; it retries until its deadline. Keep the expected outcome and coverage limit in the doc comment and the timing mechanism in one place next to the constants, stated as the relation the constants must satisfy. Name the v2.38 version and the request count, and compare the SQLSTATE by name as txnmigrator.go does.
a165b0b to
1caf00e
Compare
Migrations 000609, 000611 and 000615 start with a lock-retry block (added in #30297 and #30412) that is meant to stop live chat traffic from deadlocking with the migration. On a release upgrade it cannot do that, and it can turn a failure into a 2-minute hang or cause a failure that would otherwise not happen. This PR removes the blocks. A release upgrade that runs 000607 to 000615 while chats are in use is unreliable with or without them, so it needs chat traffic stopped, for example by scaling the deployment to zero first.
coder serverruns all pending migrations in one transaction, so every migration's locks are held until the transaction commits. On a release upgrade (from v2.38 or earlier), 000607 and 000608 lockchatsbefore any block runs. A chat request that writeschat_messagesorchat_queued_messages, or that readsusers, holds a lock that a later migration needs and then waits forchats. The block can release only the locks it took, so it can avoid this deadlock only when its migration takes the first chat lock of the upgrade. Postgres checks a waiting request for a deadlock once, afterdeadlock_timeout(1 second by default). The block pauses 100ms between 100ms lock attempts, so a check that lands in a pause finds no deadlock, the request keeps its lock, and the block retries until it fails the upgrade after 2 minutes, with the request stuck until then.Without the block, the outcome depends on timing. If the request's check fires after the migration starts waiting for it, Postgres cancels the request and the upgrade continues. If the request had already waited longer than
deadlock_timeout, its check found no deadlock, so Postgres cancels the migration instead and the upgrade fails. Main does not finish those cases either: its block was still retrying after a minute, and it gives up at its 2-minute deadline.000609 and 000611 return to their content before #30297, and 000615 loses the block in both directions. A block's migration takes the first chat lock of an upgrade only when upgrading from a build of main at 608 through 614: dogfood, where the #30297 deploy failed, or a
coder-previewimage built between 2026-10-02 and 2026-10-11. No release stops in that range, and dogfood is already past 615.Editing these merged migrations leaves every schema unchanged: no release contains them (every v2.38 release, through v2.38.3, ends at 605), databases that already ran them do not run them again, and
dump.sqlregenerates without a diff.TestMigrationChatTablesConcurrentTrafficonly covered upgrades where these migrations take the first chat lock, so it is removed.TestMigrationUpgradeFromReleaseDuringChatTrafficruns the v2.38 upgrade through 000609 with chat message inserts waiting and through 000615 withusers-then-chatstransactions waiting, each started shortly before the migration needs its lock. It passes on this branch; with main's migrations, the migration is still retrying at the 25-second timeout. Requests that waited longer are not covered.Measurements
Each run starts chat requests while the upgrade transaction is open, on the Postgres 13 test container with the default
deadlock_timeout. "Request canceled" means Postgres canceled the chat request as the deadlock victim. "Stuck" means the migration was still retrying when the run was stopped, after 10 seconds unless noted.users, thenchatsusers, thenchats, plus a chat message insertTestMigrationChatTablesConcurrentTraffic