fix(tests): close three integration-test races that flaked CI by waleedlatif1 · Pull Request #9964 · simstudioai/sim

waleedlatif1 · 2026-11-13T00:24:23Z

Summary

lane.integration.ts "finishes an occurrence whose Sim process died mid-run": waits until the first leg's text is persisted to the stream buffer before killing the process. Killing on worker.open alone raced the persist, so the takeover reattached with no text and the summary lost its first leg (the unattended suite already waits this way)
0037_seal_account_tokens.integration.ts: the cancelled migration run's rejection is handled the moment it starts, not after the pg_cancel_backend loop, so a cancel that lands first no longer fails the shard as an unhandled rejection while every test passes
search-source-setup.integration.ts: same fix for the refused provider expansion awaited after the concurrent grant
CI: the integration Postgres now logs lock waits (1s) and statements over 20s. One remaining flake (dispatcher.integration.ts) hangs on a lock that the current logs can't attribute; the next occurrence will name the waiter, the lock and the blocking statement

Found by ranking every failed integration job in ci.yml over the last two weeks; each of these failed on unrelated branches with passing reruns.

Type of Change

Bug fix (tests)

Testing

Each race injected deterministically (local-only delay): original test red with the same failure CI showed, fixed test green with and without the delay
bun run test:integration lane.integration.ts search-source-setup.integration.ts; the 0037 file against its own Postgres container
biome on the touched files
The CI settings command run against a scratch Postgres 17: all three settings applied

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing (new tests pass the test-audit authoring gate)
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

…killing its process The takeover test aborted the turn as soon as the fake worker had written the first leg, before the dying process had recorded that text in its stream buffer. The re-attach then carried receivedTextChars 0 and the occurrence's summary lost the first leg. Wait for the text event in the buffer first, as the unattended-turn suite already does.

… polling for its backend The operator-cancel test started migrate() and only attached a handler after its pg_cancel_backend polling loop. When the cancellation reached the run's connection before the loop broke, Node saw an unhandled rejection and Vitest failed the whole db shard even though every test passed. Attach the rejection assertion when the run starts and await it after the loop.

… concurrent grant The concurrent-grant test started ensureWorkspaceAccountsGroup and awaited the grant transaction before attaching a handler to the refusal. When the refusal resolved first, Node reported an unhandled rejection and Vitest failed the shard although the test passed. Attach the rejection assertion when the call starts and await it after the grant commits.

vercel · 2026-11-13T00:24:26Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Nov 13, 2026 12:35am UTC

cubic-dev-ai

No issues found across 3 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Turn on auto-fix | Re-trigger cubic}

greptile-apps · 2026-11-13T00:26:29Z

Confidence Score: 5/5

[Low impact] The PR appears safe to merge; no actionable issues remain.

Summary

Fixes three integration-test races and adds PostgreSQL logs for slow statements and lock waits.

The scheduled-job test waits for saved text before killing the worker.
The search setup test watches for refusal as soon as it starts.
The migration test catches cancellation as soon as the run starts.
CI Postgres logs lock waits and long-running statements.

No new actionable issues were found in the changes since the previous review. The full PR’s changed lines also passed the repository-rule check. Tests were not run during this review.

_{Reviews (2) · Last reviewed commit: "chore(ci): log lock waits and slow state..." · Reviewed by Greptile}

…gres

waleedlatif1 · 2026-11-13T00:37:59Z

@greptile

waleedlatif1 · 2026-11-13T00:38:00Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-11-13T00:38:05Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

2 issues found across 4 files

Confidence score: 4/5

In .github/workflows/checks.yml, a short statement can leave a transaction holding a lock without its SQL being logged, so the PID alone may not reveal what caused the wait. Consider capturing the active statement as well.
The Postgres diagnostics in .github/workflows/checks.yml are written to the service container log, but the job does not expose that log; a hang may still show only the test timeout. Print or upload the service log.

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".github/workflows/checks.yml">

<violation number="1" location=".github/workflows/checks.yml:155">
P3: `log_lock_waits` identifies the holder by PID, while this threshold logs only completed statements taking at least 20 seconds. A short statement can leave a transaction holding the lock without its SQL appearing; capture the holder's `pg_stat_activity.query` when the wait occurs.</violation>

<violation number="2" location=".github/workflows/checks.yml:155">
P3: The new Postgres settings write their lock-wait and slow-statement output to the service container's log, but nothing in this job prints or uploads that log. The next hang will still show only the test timeout, so the stated diagnostic goal is not reached. Add a step with `if: always()` that runs `docker logs` on the Postgres service container and uploads the output as an artifact.</violation>
</file>

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

cubic-dev-ai · 2026-11-13T00:41:48Z

+          # A test that hangs on a lock leaves only its timeout behind; these name the waiter, the
+          # lock, and the blocking statement in the Postgres log. Service containers take no -c flags,
+          # and ALTER SYSTEM runs outside a transaction, so each setting is its own statement.
+          bun -e 'import postgres from "postgres"; const sql = postgres(process.env.DATABASE_URL); for (const setting of ["log_lock_waits = on", "deadlock_timeout = 1000", "log_min_duration_statement = 20000"]) await sql.unsafe(`ALTER SYSTEM SET ${setting}`); await sql`SELECT pg_reload_conf()`; await sql.end()'


P3: log_lock_waits identifies the holder by PID, while this threshold logs only completed statements taking at least 20 seconds. A short statement can leave a transaction holding the lock without its SQL appearing; capture the holder's pg_stat_activity.query when the wait occurs.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/checks.yml, line 155: <comment>`log_lock_waits` identifies the holder by PID, while this threshold logs only completed statements taking at least 20 seconds. A short statement can leave a transaction holding the lock without its SQL appearing; capture the holder's `pg_stat_activity.query` when the wait occurs.</comment> <file context> @@ -149,6 +149,10 @@ jobs: + # A test that hangs on a lock leaves only its timeout behind; these name the waiter, the + # lock, and the blocking statement in the Postgres log. Service containers take no -c flags, + # and ALTER SYSTEM runs outside a transaction, so each setting is its own statement. + bun -e 'import postgres from "postgres"; const sql = postgres(process.env.DATABASE_URL); for (const setting of ["log_lock_waits = on", "deadlock_timeout = 1000", "log_min_duration_statement = 20000"]) await sql.unsafe(`ALTER SYSTEM SET ${setting}`); await sql`SELECT pg_reload_conf()`; await sql.end()' bun run db:${{ matrix.provision }} </file context>

cubic-dev-ai · 2026-11-13T00:41:48Z

+          # A test that hangs on a lock leaves only its timeout behind; these name the waiter, the
+          # lock, and the blocking statement in the Postgres log. Service containers take no -c flags,
+          # and ALTER SYSTEM runs outside a transaction, so each setting is its own statement.
+          bun -e 'import postgres from "postgres"; const sql = postgres(process.env.DATABASE_URL); for (const setting of ["log_lock_waits = on", "deadlock_timeout = 1000", "log_min_duration_statement = 20000"]) await sql.unsafe(`ALTER SYSTEM SET ${setting}`); await sql`SELECT pg_reload_conf()`; await sql.end()'


P3: The new Postgres settings write their lock-wait and slow-statement output to the service container's log, but nothing in this job prints or uploads that log. The next hang will still show only the test timeout, so the stated diagnostic goal is not reached. Add a step with if: always() that runs docker logs on the Postgres service container and uploads the output as an artifact.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/checks.yml, line 155: <comment>The new Postgres settings write their lock-wait and slow-statement output to the service container's log, but nothing in this job prints or uploads that log. The next hang will still show only the test timeout, so the stated diagnostic goal is not reached. Add a step with `if: always()` that runs `docker logs` on the Postgres service container and uploads the output as an artifact.</comment> <file context> @@ -149,6 +149,10 @@ jobs: + # A test that hangs on a lock leaves only its timeout behind; these name the waiter, the + # lock, and the blocking statement in the Postgres log. Service containers take no -c flags, + # and ALTER SYSTEM runs outside a transaction, so each setting is its own statement. + bun -e 'import postgres from "postgres"; const sql = postgres(process.env.DATABASE_URL); for (const setting of ["log_lock_waits = on", "deadlock_timeout = 1000", "log_min_duration_statement = 20000"]) await sql.unsafe(`ALTER SYSTEM SET ${setting}`); await sql`SELECT pg_reload_conf()`; await sql.end()' bun run db:${{ matrix.provision }} </file context>

waleedlatif1 added 3 commits November 12, 2026 16:23

vercel Bot temporarily deployed to Preview November 13, 2026 00:24 Inactive

cubic-dev-ai Bot reviewed Nov 13, 2026

View reviewed changes

chore(ci): log lock waits and slow statements in the integration Post…

f6442fb

…gres

waleedlatif1 merged commit 005cb97 into staging Nov 13, 2026
74 of 76 checks passed

waleedlatif1 deleted the fix/integration-flakes branch November 13, 2026 00:31

vercel Bot deployed to Preview November 13, 2026 00:35

View deployment

cubic-dev-ai Bot reviewed Nov 13, 2026

View reviewed changes

Navigation Menu

Repository navigation

fix(tests): close three integration-test races that flaked CI - #9964

fix(tests): close three integration-test races that flaked CI#9964
waleedlatif1 merged 4 commits into
stagingfrom
fix/integration-flakes

waleedlatif1 commented Nov 13, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Nov 13, 2026 •

edited

Loading

Uh oh!

cubic-dev-ai Bot left a comment

Uh oh!

greptile-apps Bot commented Nov 13, 2026 •

edited

Loading

Uh oh!

Uh oh!

waleedlatif1 commented Nov 13, 2026

Uh oh!

waleedlatif1 commented Nov 13, 2026

Uh oh!

cubic-dev-ai Bot commented Nov 13, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Uh oh!

cubic-dev-ai Bot Nov 13, 2026 •

edited

Loading

Uh oh!

cubic-dev-ai Bot Nov 13, 2026 •

edited

Loading

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

waleedlatif1 commented Nov 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Nov 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot commented Nov 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

waleedlatif1 commented Nov 13, 2026

Uh oh!

waleedlatif1 commented Nov 13, 2026

Uh oh!

cubic-dev-ai Bot commented Nov 13, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Nov 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Nov 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

waleedlatif1 commented Nov 13, 2026 •

edited

Loading

vercel Bot commented Nov 13, 2026 •

edited

Loading

greptile-apps Bot commented Nov 13, 2026 •

edited

Loading

cubic-dev-ai Bot Nov 13, 2026 •

edited

Loading

cubic-dev-ai Bot Nov 13, 2026 •

edited

Loading