fix(runtime): give the agent turn path a 16 MiB stack - #985
Open
raskevichai wants to merge 1 commit into
Open
Conversation
On aarch64 Linux every inbound Telegram message segfaulted. Under systemd with Restart=always the process crash-looped, one restart per message, and the message was dropped without a reply. SESSION_TURN_STACK_SIZE was aliased to HEAVY_RUNTIME_STACK_SIZE, so the threads that run SessionManager.processMessage*() / Agent.turn() got 2 MiB. The turn path is the deepest stack in the runtime -- channel decode, session, agent loop, provider call, tool dispatch -- and aarch64 frames are larger than x86_64 ones, so the stack pointer crossed into the PROT_NONE guard page below the thread stack. The reporter confirmed the mechanism from both sides: at the crash $sp sat inside the guard mapping, the return address appeared zero times on the stack (so not runaway recursion), and an LD_PRELOAD shim clamping every pthread stack to 16 MiB made the bot answer normally with zero restarts. `nullclaw agent` was never affected because it runs on the 8 MiB main-thread stack. Set the budget to 16 MiB and stop aliasing the heavy-runtime constant, since the two roles have genuinely different depth requirements. This covers every thread that executes a turn: the five channel polling loops, the parallel per-session message workers, the gateway turn thread, and the inbound dispatcher. Stack size is address space, not resident memory -- Linux commits stack pages on first touch, so threads that stay shallow still cost a few pages of RSS. Closes nullclaw#976.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #976.
Root cause
SESSION_TURN_STACK_SIZEwas aliased toHEAVY_RUNTIME_STACK_SIZE:That constant sizes every thread which runs
SessionManager.processMessage*()/Agent.turn(). The turn path is the deepest stack in the runtime (channel decode -> session -> agent loop -> provider -> tool dispatch), and aarch64 frames are larger than x86_64 ones, so 2 MiB was not enough there: the stack pointer crossed into thePROT_NONEguard page below the thread stack and the process died on every inbound message.nullclaw agent -m "hi"was never affected because it runs on the 8 MiB main-thread stack, which is why the config, provider and LLM paths all looked healthy.Why 16 MiB
The reporter established the mechanism from both directions:
$spwas inside the guard mapping immediately below the worker stack, and the return address appeared zero times on the stack - so a stack overflow, not runaway recursion.LD_PRELOADshim clamping everypthread_attr_setstacksizecall to 16 MiB made the bot reply normally withNRestartsstaying at 0.Their
LD_PRELOADtrace observed 512 KiB and 2 MiB stacks, which line up exactly withCOORDINATION_STACK_SIZE/AUXILIARY_LOOP_STACK_SIZE(512 KiB) andHEAVY_RUNTIME_STACK_SIZE(2 MiB) inthread_stacks.zig.16 MiB is the value verified on the affected hardware. 8 MiB is only known to be sufficient on x86_64 (that is the main-thread size), so it is not a safe target for the platform that actually fails.
This is address space, not resident memory. Linux commits stack pages on first touch, so a thread that stays shallow still costs a few pages of RSS. The
~1 MB peak RSSconstraint inCLAUDE.mdis unaffected; virtual size grows, RSS does not.Radius
One constant, but it covers every thread that executes a turn:
spawnTelegramPollingand siblings)messageTaskWorker)inboundDispatcherThreadindaemon.zig- which is literally the "inbound worker" named in the issueThe alias to
HEAVY_RUNTIME_STACK_SIZEis also dropped: websocket gateway loops and subagents have genuinely different depth requirements from the turn path, and tying them together is what let this regress silently.Tests
session turn stack size can spawn a thread- matches the existing pattern for the other budgets, and guards against a size the pthread backend would reject.session turn stack clears the main-thread budget- asserts the budget exceeds both the 8 MiB main-thread reference andHEAVY_RUNTIME_STACK_SIZE, so re-aliasing the constant fails the build.Sanity check: restoring
= HEAVY_RUNTIME_STACK_SIZEmakes the second test fail; restoring the fix makes the suite green again.Verification
I do not have aarch64 hardware, so I could not reproduce the segfault or measure the true peak stack usage of the turn path. This change follows the reporter's diagnosis and their verified 16 MiB workaround. Confirmation from anyone running a Raspberry Pi or other aarch64 host would be worth having before release.
Related
Two commenters on the issue attributed this to the default Zig stack size (~512 KiB) in
channel_loop.zig/telegram.zig. That is not what the code does - the size is set explicitly at every spawn site throughthread_stacks.zig, and the threads on the turn path were getting 2 MiB.