Skip to content

fix(tests): close three integration-test races that flaked CI #9964

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
waleedlatif1 merged 4 commits into from
Nov 13, 2026
+25 −8
Merged

fix(tests): close three integration-test races that flaked CI #9964

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7c829c1
fix(tests): wait for the job lane's first leg text to persist before …
waleedlatif1 Nov 13, 2026
3810acc
fix(tests): handle the cancelled token-sealing run's rejection before…
waleedlatif1 Nov 13, 2026
2be3fb3
fix(tests): handle the refused provider expansion before awaiting the…
waleedlatif1 Nov 13, 2026
f6442fb
chore(ci): log lock waits and slow statements in the integration Post…
waleedlatif1 Nov 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/checks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,10 @@ jobs:
working-directory: packages/db
run: |
bun -e 'import postgres from "postgres"; const sql = postgres(process.env.DATABASE_URL); for (const extension of ["vector", "btree_gin", "pg_trgm"]) await sql`CREATE EXTENSION IF NOT EXISTS ${sql(extension)}`; await sql.end()'
# A test that hangs on a lock leaves only its timeout behind; these name the waiter, the
# lock, and the blocking statement in the Postgres log. Service containers take no -c flags,
# and ALTER SYSTEM runs outside a transaction, so each setting is its own statement.
bun -e 'import postgres from "postgres"; const sql = postgres(process.env.DATABASE_URL); for (const setting of ["log_lock_waits = on", "deadlock_timeout = 1000", "log_min_duration_statement = 20000"]) await sql.unsafe(`ALTER SYSTEM SET ${setting}`); await sql`SELECT pg_reload_conf()`; await sql.end()'

@cubic-dev-ai cubic-dev-ai Bot Nov 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: log_lock_waits identifies the holder by PID, while this threshold logs only completed statements taking at least 20 seconds. A short statement can leave a transaction holding the lock without its SQL appearing; capture the holder's pg_stat_activity.query when the wait occurs.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/checks.yml, line 155:

<comment>`log_lock_waits` identifies the holder by PID, while this threshold logs only completed statements taking at least 20 seconds. A short statement can leave a transaction holding the lock without its SQL appearing; capture the holder's `pg_stat_activity.query` when the wait occurs.</comment>

<file context>
@@ -149,6 +149,10 @@ jobs:
+          # A test that hangs on a lock leaves only its timeout behind; these name the waiter, the
+          # lock, and the blocking statement in the Postgres log. Service containers take no -c flags,
+          # and ALTER SYSTEM runs outside a transaction, so each setting is its own statement.
+          bun -e 'import postgres from "postgres"; const sql = postgres(process.env.DATABASE_URL); for (const setting of ["log_lock_waits = on", "deadlock_timeout = 1000", "log_min_duration_statement = 20000"]) await sql.unsafe(`ALTER SYSTEM SET ${setting}`); await sql`SELECT pg_reload_conf()`; await sql.end()'
           bun run db:${{ matrix.provision }}
 
</file context>
Fix with cubic

@cubic-dev-ai cubic-dev-ai Bot Nov 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The new Postgres settings write their lock-wait and slow-statement output to the service container's log, but nothing in this job prints or uploads that log. The next hang will still show only the test timeout, so the stated diagnostic goal is not reached. Add a step with if: always() that runs docker logs on the Postgres service container and uploads the output as an artifact.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/checks.yml, line 155:

<comment>The new Postgres settings write their lock-wait and slow-statement output to the service container's log, but nothing in this job prints or uploads that log. The next hang will still show only the test timeout, so the stated diagnostic goal is not reached. Add a step with `if: always()` that runs `docker logs` on the Postgres service container and uploads the output as an artifact.</comment>

<file context>
@@ -149,6 +149,10 @@ jobs:
+          # A test that hangs on a lock leaves only its timeout behind; these name the waiter, the
+          # lock, and the blocking statement in the Postgres log. Service containers take no -c flags,
+          # and ALTER SYSTEM runs outside a transaction, so each setting is its own statement.
+          bun -e 'import postgres from "postgres"; const sql = postgres(process.env.DATABASE_URL); for (const setting of ["log_lock_waits = on", "deadlock_timeout = 1000", "log_min_duration_statement = 20000"]) await sql.unsafe(`ALTER SYSTEM SET ${setting}`); await sql`SELECT pg_reload_conf()`; await sql.end()'
           bun run db:${{ matrix.provision }}
 
</file context>
Fix with cubic

bun run db:${{ matrix.provision }}
- name: Verify migration replay is a no-op
Expand Down
15 changes: 9 additions & 6 deletions apps/sim/lib/knowledge/__integration__/search-source-setup.integration.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -492,14 +492,17 @@ describe('Search source identity and concurrent creation', () => {
.where(eq(resourcePolicy.resourceId, searchGroupId))
})
await locked
const append = ensureWorkspaceAccountsGroup(ids.workspaceId, ids.aliceId, {
provider: 'jira',
label: 'Jira',
required: false,
})
/** Handled from the start: the refusal can land before the grant's commit resolves here. */
const refused = expect(
ensureWorkspaceAccountsGroup(ids.workspaceId, ids.aliceId, {
provider: 'jira',
label: 'Jira',
required: false,
})
).rejects.toThrow('has workflow access')
releaseGrant()
await grant
await expect(append).rejects.toThrow('has workflow access')
await refused
const existing = await ensureWorkspaceAccountsGroup(ids.workspaceId, ids.aliceId, {
provider: 'google-drive',
label: 'Google Drive',
Expand Down
9 changes: 9 additions & 0 deletions apps/sim/lib/mothership/jobs/lane.integration.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,7 @@ import { jobOccurrenceId, MIN_JOB_CADENCE_MS } from '@/lib/mothership/jobs/occur
import { JOB_DISPATCH_GRACE_MS, reconcileJobRuns } from '@/lib/mothership/jobs/reconcile'
import { takeOverAbandonedRuns } from '@/lib/mothership/request/application/abandoned-runs'
import { startUnattendedTurn } from '@/lib/mothership/request/lifecycle/unattended'
import { readEvents } from '@/lib/mothership/request/session/buffer'
import {
chatStreamLockKey,
StreamControllerSupersededError,
Expand Down Expand Up @@ -506,6 +507,14 @@ describe.runIf(Boolean(redisUrl))('the scheduled job lane', () => {
])
await runJobLane(new Date())
await waitFor(async () => worker.open.has(occurrenceId), 'the first leg')
/** The worker has written the text, but the process must persist it before it dies. */
await waitFor(
async () =>
(await readEvents(occurrenceId, '0')).some(
(event) => event.type === 'text' && event.payload.text === 'Started triage. '
),
'the first leg text to persist before the process dies'
)

controllers.get(occurrenceId)?.abort(new StreamControllerSupersededError())
worker.open.get(occurrenceId)?.response.socket?.destroy()
Expand Down
5 changes: 3 additions & 2 deletions packages/db/script-migrations/0037_seal_account_tokens.integration.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -358,7 +358,8 @@ describe('sealing stored account tokens in PostgreSQL', () => {
await insertAccount('cancelled-access', 'cancelled-refresh')
await onAccountUpdate('PERFORM pg_sleep(20)')

const run = migrate()
/** Handled from the start: the cancel can fail the run before the polling loop sees it. */
const failed = expect(migrate()).rejects.toThrow('canceling statement due to user request')
for (let attempt = 0; attempt < 100; attempt++) {
const [cancelled] = await admin<{ cancelled: boolean }[]>`
SELECT pg_cancel_backend(pid) AS cancelled FROM pg_stat_activity
Expand All @@ -368,7 +369,7 @@ describe('sealing stored account tokens in PostgreSQL', () => {
await sleep(50)
}

await expect(run).rejects.toThrow('canceling statement due to user request')
await failed
expect(await isRecorded()).toBe(false)
expect((await storedRows())[0].access_token).toBe('cancelled-access')
}, 30_000)
Expand Down
Loading