VM hang after 10 days uptime: RCU stall → NFS/health check/fsnotify cascade (v2.0.5) #2352

OrbStack Diagnostic Report

OrbStack info:
  Version: 2.0.5
  Commit: cfe47627f138ffd822c958553b0a93eaf2692c71 (v2.0.5)

System info:
  macOS: 26.3 (25D125)
  CPU: arm64, 10 cores
  CPU model: Apple M1 Max
  Model: Mac13,1
  Memory: 32 GiB

Full report: https://orbstack.dev/_admin/diag/orbstack-diagreport_2026-03-08T18-54-23.865599Z.zip

Summary

After ~10 days of continuous uptime (booted Feb 21), the OrbStack VM entered an unrecoverable hang on March 3. The vmgr process consumed ~400% CPU for the next 5 days without recovering. Only kill -9 + relaunch resolved it. All container data was preserved.

Data directory is on an external APFS volume (/Volumes/qwiizlab/OrbStack-data).

Precise Failure Timeline (March 3, 2026)

1. 18:39:56 — failed to check update: posix_spawn_file_actions_adddup2: bad file descriptor
   (suggests file descriptor table corruption or exhaustion)

2. kernel [856192] — sched: DL replenish lagged too much
   (scheduler deadline overrun — CPU contention in the VM)

3. kernel [856277] — RCU stall detected:

   rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
   rcu: 3-...0: (3 ticks this GP) idle=729c/1/0x4000000000000000 softirq=5974656/5974656 fqs=9927
   rcu: (detected by 1, t=90002 jiffies, g=27731665, q=28642 ncpus=10)
   

   A vCPU was stuck for too long without yielding to RCU.

4. 18:46:13 — First sampling stacks due to VM hang warning (vmgr detected the VM was unresponsive)

5. 20:21:48 — First health check failed (vcontrol endpoint timeout)

6. 20:30:02 — First host-nfs forward: dial failed (NFS mount broken)

7. 20:36:00 — First failed to send fsnotify events (file watch broken)

The VM never recovered after this point.

Preceding Context

Shortly before the hang, NFS dref errors appeared:

krpc: dref failed: /mnt/mac/Users/scott/gt/baseball/grafana-loki/config/loki-config.yaml = -20
krpc: dref failed: /mnt/mac/Users/scott/gt/baseball/grafana-loki/config/promtail-config.yaml = -20

The vCPU stack traces at hang time showed the kernel stuck in:

• TLB flush / page table teardown (free_pgd_range, free_pgtables)
• smp_call_function cross-CPU IPI
• RCU synchronization (sync_rcu_exp_select_node_cpus)

Error Totals (Mar 3 through Mar 8)

Possible Contributing Factor

The macOS disk cleanup tool mole (mo clean) was run around this timeframe. It's possible mole deleted cached files or temp files that the VM's NFS mount or host integration depended on, triggering the krpc dref errors and subsequent RCU stall. However, ~/Library/Group Containers/ was intact.

Resolution

kill -9 on the vmgr process, then relaunched OrbStack. Fresh VM booted against existing data.img.raw — all containers and images preserved.

Attached Diagnostics

The pre-restart vmgr.log (103MB, compressed to ~4MB) containing the full failure sequence is available in the attached zip. The orbctl report diagnostic zip is also included.