fix(backend-cpu): CoroutineSchedule deadlock that hangs the jvm CI leg - #1266
Merged
Merged
Conversation
…from its own pool The jvm CI leg has timed out at 40 minutes on three of the last four runs since SKEEP-005 (#1262), hanging inside skainet-backend-cpu:jvmTest with 11 GB of memory free. It was not the OOM #1264 assumed but a coroutine deadlock: CoroutineSchedule.forRange ran a region as runBlocking { coroutineScope { launch(Dispatchers.Default) ... } }, and a coroutineScope waits for every child, including the ones the pool never got a thread for. On the 4-vCPU runner, once every Dispatchers.Default worker was inside a region (CoroutineScheduleTest does exactly that), nobody was left to run the children. Never reproduced on a 14-core laptop. A region is now a shared chunk queue: tasks - 1 helpers are dispatched to the dispatcher's executor, the caller runs chunk 0 and then drains the queue itself, and it waits only for chunks a thread has already claimed. A caller can therefore always finish a region alone, whatever the pool is doing; a helper that never ran finds the queue empty and exits. The contract is unchanged: first failure stops unstarted chunks and is rethrown once running ones finish (later failures suppressed), nested regions run inline, writes happen-before the return. Public API unchanged. Tests: a new deterministic pool-exhaustion test (two callers saturate a two-thread pool) times out on the old implementation and passes now; the caller-runs-first-chunk test allows the caller to help with later chunks. Verified with the test JVM pinned to 4 processors (-XX:ActiveProcessorCount=4), which reproduced the CI hang before the fix. Docs: schedules explanation and the SKEEP-005 proposal describe the new mechanism; the build.yml comment on the jvm leg states the real cause. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
📖 Documentation Preview The documentation has been built successfully for this PR. Generated Files:
Artifacts:
This comment will be updated automatically when the PR is updated. |
This was referenced Sep 6, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why develop is red
Three of the last four
test (jvm)legs (develop after #1262, develop after #1264, PR #1265) hit the 40-minute timeout inside:skainet-backends:skainet-backend-cpu:jvmTest, each with ~11 GB of memory free.build-jobthen fails in 2 s because it is only the aggregate gate. It was not the OOM that #1264 assumed.Root cause
CoroutineSchedule.forRangeran a region asrunBlocking { coroutineScope { launch(Dispatchers.Default) … } }. AcoroutineScopewaits for every child, including the ones the pool never got a thread for. On the 4-vCPUubuntu-latestrunner, once all fourDispatchers.Defaultworkers were inside a region (whichCoroutineScheduleTest.aRegionStartedFromADefaultDispatcherWorkerCompletesdoes deliberately), nobody was left to run the children and the test JVM parked forever. It never reproduces on a many-core laptop, which is why the PR runs were sometimes green.Reproduced locally by pinning the test JVM to four processors (
-XX:ActiveProcessorCount=4): the task hangs at that test, and a thread dump shows all four dispatcher workers parked inBlockingCoroutine.joinBlockingcalled fromCoroutineSchedule.forRange.Fix
A region is now a shared chunk queue.
tasks - 1helpers are dispatched to the dispatcher's executor, the calling thread runs chunk 0 and then drains the queue itself, and finally waits only for chunks some thread has already claimed. A caller can therefore always finish a region on its own, whatever the pool is doing; a helper that never ran finds the queue empty and exits without touching the body.The
Schedule.forRangecontract is unchanged: the first failure stops unstarted chunks and is rethrown once running ones finish (later failures attached as suppressed), nested regions run inline, every chunk's writes happen-before the return. Public API unchanged (apiCheckpasses).Tests
aRegionEnteredFromEveryThreadOfItsOwnPoolStillCompletes: two callers saturate a two-thread pool behind a barrier. Deterministic. Fails withTimeoutExceptionafter 30 s on the old implementation, passes on this one.callerThreadRunsTheFirstChunkAndWorkersHelpWithTheRest: the caller still runs chunk 0, but may now help with later chunks, so the "exactly one chunk on the caller" assertion is relaxed.Local verification: full
skainet-backend-cpu:jvmTest(469 tests, 0 failures),apiCheck,skainet-docs-samples:jvmTest.Docs / CI
The schedules explanation page and the SKEEP-005 proposal describe the new mechanism and why the first version hung. The
build.ymlcomment on the jvm leg now states the real cause; the serialworkers.max=1setting stays since it costs nothing in wall-clock.🤖 Generated with Claude Code