Skip to content

Pre-existing containers fail to start after OrbStack restart — "failed to mount rootfs" (open tree in mount ns: ESRCH); recreating fixes it #2605

Description

@hau823823

Describe the bug

After an OrbStack restart, all pre-existing containers failed to start. Each died 135–180 ms after start, with RestartCount=0 and an empty State.Error — the daemon believed it had started them successfully; the container's first process died on its own.

The failure is not storage-level data loss. docker export on a "broken" container still yields a complete rootfs (61 libc-related files present), and a freshly created container from the same image runs fine. Only the pre-existing container instances were affected.

Six containers failed at once, with three different error signatures — all of them variations of "cannot read a file that provably exists in the image layers":

Container image First unreadable file Exit code
postgres:16, redis:7 (×2), mongo:7 (arm64 native) libc.so.6 127
apache/kafka:3.7.2 (arm64 native) /__cacert_entrypoint.sh 2
bitgo/express:latest (amd64 via Rosetta) /lib64/ld-linux-x86-64.so.2 133

Container logs:

/usr/bin/env: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory
/bin/sh: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory
sh: can't open '/__cacert_entrypoint.sh': Permission denied
rosetta error: failed to open elf at /lib64/ld-linux-x86-64.so.2

Note that exit 127 here is not "command not found" — sh / env were found and started; the dynamic loader failed to open libc.so.6.

Recreating the containers (docker compose down && up -d, docker rm + docker run) fixes it immediately. Pressing ▶ in the OrbStack UI does not — it restarts the same broken instance, which dies again in ~200 ms.

Relevant log evidence

From ~/.orbstack/log/vmgr.1.log, at the exact second the containers died:

📦 scon | time="07-21 05:15:54" level=info  msg="add forward" container=docker spec="tcp:0.0.0.0:5433"
📦 scon | time="07-21 05:15:54" level=info  msg="add forward" container=docker spec="tcp:0.0.0.0:27017"
    (… 5 containers' port forwards added …)

📦 scon | time="07-21 05:15:55" level=error msg="failed to set host state while adding docker container to domainproxy" error="get netns cookie: no such process"
📦 scon | time="07-21 05:15:55" level=error msg="failed to mount rootfs" cid=1cfc57c8c38917047362a8d3db6d85932fcfc3c073a734b7c0d4cedab7af13b6 error="open tree in mount ns: no such process"

📦 scon | time="07-21 05:15:55" level=info  msg="remove forward" key="tcp:0.0.0.0:5433"
    (… all port forwards removed — containers are dead …)

Failed vs. successful boot comparison on the same machine (this is the part I think is most useful):

Failed boot (vmgr.1.log) Successful boot (vmgr.log)
failed to mount rootfs present absent
get netns cookie: no such process present absent
failed to add forward … can't assign requested address present present (harmless startup-timing noise, appears in both)
Container outcome all dead all healthy

Both errors are ESRCH ("no such process"), and their timestamp (05:15:55) is after the containers' FinishedAt (05:15:54.90). So I can't tell from these logs whether the failed rootfs mount is the cause or a downstream symptom of an already-dead container — but these two lines are reproducibly present on the failed boot and absent on the healthy one, so they appear related rather than being background noise.

My best guess (explicitly a guess): a startup-sequencing/handshake issue where the container's init execs the entrypoint before scon has attached the rootfs into the container's mount namespace. That would explain the intact on-disk data, the ESRCH, and the intermittency. I have no visibility into scon internals to confirm this.

One caveat on the evidence: failed to mount rootfs appears only once, for a single cid, while five containers failed. The other four died without any logged mount error, so this single line does not by itself account for all failures.

Things I ruled out

  • Image corruption / architecture mismatchdocker run --rm <same image> works fine (verified for both arm64 native and amd64-via-Rosetta).
  • Layer data lossdocker export <broken container> | tar -tf - | grep -c libc → 61 files present.
  • Rosetta brokendocker run --rm --platform linux/amd64 alpine:3 uname -mx86_64, works.
  • Host reboot — host uptime was 8 days; no reboot involved.
  • OrbStack upgrade — the app bundle had not been modified since 2026-06-05; this was not an upgrade-triggered regression.
  • Compose config — unchanged; the exact same config had been running fine for 4 days.

To Reproduce

The failure is intermittent and I do not have a deterministic reproducer.

  1. Start containers with docker compose up -d (my set: postgres:16, mongo:7, redis:7 ×2, apache/kafka:3.7.2) plus one amd64 container run via Rosetta (bitgo/express:latest).
  2. Leave them running for a few days.
  3. Restart OrbStack (VM restart — no host reboot).
  4. Containers auto-start and immediately die with the errors above.

Data point on the intermittency: on this same machine, an OrbStack restart on 2026-07-17 (14:55 → 14:56) was completely fine, and the containers I recreated after the failure survived another OrbStack restart at 13:31 the same day without issue. So roughly 1 in 3 restarts triggered it here.

Expected behavior

Pre-existing containers should start normally after an OrbStack restart, with their rootfs correctly attached — as they do on a clean boot.

Environment

OrbStack: 2.2.1 (2020100)
Commit:   0e182b501fcd9e05b99ffb363fce03610390c400 (v2.2.1)
macOS:    26.5.2 (25F84)
CPU:      arm64, 10 cores, Apple M5
Model:    Mac17,3
Memory:   24 GiB
Docker:   29.4.0
Storage:  overlayfs (io.containerd.snapshotter.v1)

Diagnostic report

Full report: https://orbstack.dev/_admin/diag/orbstack-diagreport_2026-07-21T06-04-16.808532Z.zip

Note: this report was generated after the containers were already recreated, so it captures the post-recovery state. The important part is vmgr_logs/vmgr.1.log inside it, which still contains the failed boot.

Additional context

I have both vmgr.1.log (failed boot) and vmgr.log (successful boot) saved from the same machine and can provide them — the side-by-side is likely more informative than either alone.

Possibly related but, as far as I can tell, distinct:

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions