fix(mongo): survive losing the first-boot bind race to the init server - #13591
fix(mongo): survive losing the first-boot bind race to the init server#13591abnegate wants to merge 2 commits into
Conversation
On a fresh volume the standard mongo entrypoint runs a temporary server on 127.0.0.1:27017 to create the users and run docker-entrypoint-initdb.d, then starts the real server on 0.0.0.0:27017 without waiting for the temporary one to let go of the port. The real server spends about a second opening WiredTiger before it binds, so it only wins by however long the temporary server's shutdown takes. On an idle machine that shutdown takes ~127ms and the real server binds with a second to spare. On a loaded CI runner it took 1126ms, the bind was attempted at 1150ms, and the real server exited with Error setting up transport layer ... 0.0.0.0:27017 :: caused by :: setup bind :: caused by :: Address already in use 24ms short. The container goes with it and `docker compose up --wait` fails every service that depends_on mongodb, so the lane reports "dependency failed to start: container appwrite-mongodb is unhealthy" having run no tests at all. It landed on one of the 191 lanes in four of the last five runs. The data directory is fully initialised by the time this fires, so the recovery is to start again: the second pass finds it populated, skips the temporary server entirely, and binds once the port is free. The steady state is untouched -- an initialised directory still hands straight over with exec, so mongod keeps PID 1 and its signal handling on every boot after the first. Verified by holding 27017 across the real server's bind on a first boot: before, the container exits 133 and never serves; after, it reports the conflict, retries and comes up. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
✨ Benchmark resultsComparing
Per-scenario breakdown & investigation detailsMetrics below reflect the current branch (after). Δ P95 compares against the base.
Top API waits (after)
|
…race The first-boot retry treated every failure, and every signal, as a reason to start MongoDB again. `docker stop` inside the first-boot window killed the child, `wait` returned, and the script started the server the caller had just asked to go away. The container then sat there until Docker force-killed it at the end of the grace period: measured on mongo:8.2.5, 30s and exit 137 both mid-initialisation and once the server was already serving, against 0s and exit 143 now. An initialisation script that failed once the storage files existed was worse. The second pass found /data/db populated, skipped /docker-entrypoint-initdb.d, and served a database with no application user while the original failure went unreported. Measured: container running, reachable, application user absent, against exit 1 and no server now. Retry only mongod's EXIT_NET_ERROR. 48 is what it exits with when it cannot bind, and the standard entrypoint ends in `exec "$@"`, so the status arrives unchanged; a temporary server that loses the bind fails inside `mongod --fork`, whose parent reports 1, so only the real server can reach 48. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Both P1s were valid and are fixed in Shutdown Restarts MongoDB. Correct, and worse than a lingering container: the script started the server the caller had just asked to go away. The trap now records which signal arrived, forwards it, and — because a trapped signal makes Measured on
All Failures Trigger Retry. Also correct, and the more dangerous of the two. An initialisation script failing after the storage files existed left the second pass looking at a populated The retry is now gated on On the one red check: |
On a fresh volume the standard mongo entrypoint runs a temporary server on
127.0.0.1:27017to create the users and rundocker-entrypoint-initdb.d, then starts the real server on0.0.0.0:27017without waiting for the temporary one to release the port. From a failing run:The real server spends about a second opening WiredTiger before it binds, so it only ever wins by however long the temporary server's shutdown takes.
When it loses, the container exits with it and
docker compose up --waitfails every service thatdepends_onmongodb. The lane then reportshaving run no tests at all. Measured on
feat-query-lib: it took out one of the 191 lanes in four of the last five runs, a different lane each time (Locale, Teams, Migrations, Proxy). It is not specific to that branch — nothing in it touches this file.The fix
Start again rather than retry blindly. The data directory is fully initialised by the time this fires — users created, init scripts run, temporary server cleanly shut down — so only the final server start failed. The second pass finds the directory populated, skips the temporary server entirely, and binds once the port is free.
The steady state is untouched: an initialised data directory still hands straight over with
exec, somongodkeeps PID 1 and its signal handling on every boot after the first. Atrapcovers the single first-boot window where it runs as a child, sodocker stopstill shuts the database down cleanly.Verification
The 24 ms window would not reproduce on an idle arm64 host — twelve cold starts all passed, and constraining CPU did not move it. So the failure condition it produces was forced instead, by holding
27017across the real server's bind on a first boot. Counting only the runs where the conflict actually fired:Note this reproduces the failure condition rather than the race itself; the real confirmation is CI staying clean across runs.
🤖 Generated with Claude Code