Skip to content

fix(backend-cpu): CoroutineSchedule deadlock that hangs the jvm CI leg - #1266

Merged
michalharakal merged 1 commit into
developfrom
fix/coroutine-schedule-pool-deadlock
Sep 6, 2026
Merged

michalharakal merged 1 commit into
developfrom
fix/coroutine-schedule-pool-deadlock

Conversation

@michalharakal

Copy link
Copy Markdown
Contributor

Why develop is red

Three of the last four test (jvm) legs (develop after #1262, develop after #1264, PR #1265) hit the 40-minute timeout inside :skainet-backends:skainet-backend-cpu:jvmTest, each with ~11 GB of memory free. build-job then fails in 2 s because it is only the aggregate gate. It was not the OOM that #1264 assumed.

Root cause

CoroutineSchedule.forRange ran a region as runBlocking { coroutineScope { launch(Dispatchers.Default) … } }. A coroutineScope waits for every child, including the ones the pool never got a thread for. On the 4-vCPU ubuntu-latest runner, once all four Dispatchers.Default workers were inside a region (which CoroutineScheduleTest.aRegionStartedFromADefaultDispatcherWorkerCompletes does deliberately), nobody was left to run the children and the test JVM parked forever. It never reproduces on a many-core laptop, which is why the PR runs were sometimes green.

Reproduced locally by pinning the test JVM to four processors (-XX:ActiveProcessorCount=4): the task hangs at that test, and a thread dump shows all four dispatcher workers parked in BlockingCoroutine.joinBlocking called from CoroutineSchedule.forRange.

Fix

A region is now a shared chunk queue. tasks - 1 helpers are dispatched to the dispatcher's executor, the calling thread runs chunk 0 and then drains the queue itself, and finally waits only for chunks some thread has already claimed. A caller can therefore always finish a region on its own, whatever the pool is doing; a helper that never ran finds the queue empty and exits without touching the body.

The Schedule.forRange contract is unchanged: the first failure stops unstarted chunks and is rethrown once running ones finish (later failures attached as suppressed), nested regions run inline, every chunk's writes happen-before the return. Public API unchanged (apiCheck passes).

Tests

  • New aRegionEnteredFromEveryThreadOfItsOwnPoolStillCompletes: two callers saturate a two-thread pool behind a barrier. Deterministic. Fails with TimeoutException after 30 s on the old implementation, passes on this one.
  • callerThreadRunsTheFirstChunkAndWorkersHelpWithTheRest: the caller still runs chunk 0, but may now help with later chunks, so the "exactly one chunk on the caller" assertion is relaxed.
  • Existing Default-dispatcher test kept; passes with the JVM pinned to 4 processors, 5/5 repeated runs.

Local verification: full skainet-backend-cpu:jvmTest (469 tests, 0 failures), apiCheck, skainet-docs-samples:jvmTest.

Docs / CI

The schedules explanation page and the SKEEP-005 proposal describe the new mechanism and why the first version hung. The build.yml comment on the jvm leg now states the real cause; the serial workers.max=1 setting stays since it costs nothing in wall-clock.

🤖 Generated with Claude Code

…from its own pool

The jvm CI leg has timed out at 40 minutes on three of the last four runs
since SKEEP-005 (#1262), hanging inside skainet-backend-cpu:jvmTest with
11 GB of memory free. It was not the OOM #1264 assumed but a coroutine
deadlock: CoroutineSchedule.forRange ran a region as
runBlocking { coroutineScope { launch(Dispatchers.Default) ... } }, and a
coroutineScope waits for every child, including the ones the pool never
got a thread for. On the 4-vCPU runner, once every Dispatchers.Default
worker was inside a region (CoroutineScheduleTest does exactly that),
nobody was left to run the children. Never reproduced on a 14-core laptop.

A region is now a shared chunk queue: tasks - 1 helpers are dispatched to
the dispatcher's executor, the caller runs chunk 0 and then drains the
queue itself, and it waits only for chunks a thread has already claimed.
A caller can therefore always finish a region alone, whatever the pool is
doing; a helper that never ran finds the queue empty and exits. The
contract is unchanged: first failure stops unstarted chunks and is
rethrown once running ones finish (later failures suppressed), nested
regions run inline, writes happen-before the return. Public API unchanged.

Tests: a new deterministic pool-exhaustion test (two callers saturate a
two-thread pool) times out on the old implementation and passes now; the
caller-runs-first-chunk test allows the caller to help with later chunks.
Verified with the test JVM pinned to 4 processors (-XX:ActiveProcessorCount=4),
which reproduced the CI hang before the fix.

Docs: schedules explanation and the SKEEP-005 proposal describe the new
mechanism; the build.yml comment on the jvm leg states the real cause.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 6, 2026

Copy link
Copy Markdown

📖 Documentation Preview

The documentation has been built successfully for this PR.

Generated Files:

  • Operator documentation: docs/modules/operators/_generated_/
  • JSON schema output: operators.json

Artifacts:

  • Download the documentation-preview-1266 artifact to view the complete documentation locally.

This comment will be updated automatically when the PR is updated.

@michalharakal
michalharakal merged commit 3c7ef36 into develop Sep 6, 2026
20 checks passed
@michalharakal
michalharakal deleted the fix/coroutine-schedule-pool-deadlock branch September 6, 2026 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant