Skip to content

fix(mongo): survive losing the first-boot bind race to the init server - #13591

Open
abnegate wants to merge 2 commits into
mainfrom
fix/mongo-first-boot-bind-race-main
Open

fix(mongo): survive losing the first-boot bind race to the init server#13591
abnegate wants to merge 2 commits into
mainfrom
fix/mongo-first-boot-bind-race-main

Conversation

@abnegate

Copy link
Copy Markdown
Member

On a fresh volume the standard mongo entrypoint runs a temporary server on 127.0.0.1:27017 to create the users and run docker-entrypoint-initdb.d, then starts the real server on 0.0.0.0:27017 without waiting for the temporary one to release the port. From a failing run:

57.883  real server starts          <- before the temporary one is even asked to stop
57.891  temporary server begins shutdown
59.009  temporary server "Now exiting"
59.033  real server: Error setting up transport layer ...
        0.0.0.0:27017 :: caused by :: setup bind :: caused by :: Address already in use

The real server spends about a second opening WiredTiger before it binds, so it only ever wins by however long the temporary server's shutdown takes.

temporary server shutdown outcome
idle machine 127 ms binds with ~1 s to spare
loaded CI runner 1126 ms bind attempted at 1150 ms, lost by 24 ms

When it loses, the container exits with it and docker compose up --wait fails every service that depends_on mongodb. The lane then reports

dependency failed to start: container appwrite-mongodb is unhealthy

having run no tests at all. Measured on feat-query-lib: it took out one of the 191 lanes in four of the last five runs, a different lane each time (Locale, Teams, Migrations, Proxy). It is not specific to that branch — nothing in it touches this file.

The fix

Start again rather than retry blindly. The data directory is fully initialised by the time this fires — users created, init scripts run, temporary server cleanly shut down — so only the final server start failed. The second pass finds the directory populated, skips the temporary server entirely, and binds once the port is free.

The steady state is untouched: an initialised data directory still hands straight over with exec, so mongod keeps PID 1 and its signal handling on every boot after the first. A trap covers the single first-boot window where it runs as a child, so docker stop still shuts the database down cleanly.

Verification

The 24 ms window would not reproduce on an idle arm64 host — twelve cold starts all passed, and constraining CPU did not move it. So the failure condition it produces was forced instead, by holding 27017 across the real server's bind on a first boot. Counting only the runs where the conflict actually fired:

entrypoint conflict hit retry taken container server reachable
before 1 0 exited 133 no
after 1 1 running yes

Note this reproduces the failure condition rather than the race itself; the real confirmation is CI staying clean across runs.

🤖 Generated with Claude Code

On a fresh volume the standard mongo entrypoint runs a temporary server
on 127.0.0.1:27017 to create the users and run docker-entrypoint-initdb.d,
then starts the real server on 0.0.0.0:27017 without waiting for the
temporary one to let go of the port. The real server spends about a
second opening WiredTiger before it binds, so it only wins by however
long the temporary server's shutdown takes.

On an idle machine that shutdown takes ~127ms and the real server binds
with a second to spare. On a loaded CI runner it took 1126ms, the bind
was attempted at 1150ms, and the real server exited with

  Error setting up transport layer ... 0.0.0.0:27017 :: caused by ::
  setup bind :: caused by :: Address already in use

24ms short. The container goes with it and `docker compose up --wait`
fails every service that depends_on mongodb, so the lane reports
"dependency failed to start: container appwrite-mongodb is unhealthy"
having run no tests at all. It landed on one of the 191 lanes in four of
the last five runs.

The data directory is fully initialised by the time this fires, so the
recovery is to start again: the second pass finds it populated, skips the
temporary server entirely, and binds once the port is free. The steady
state is untouched -- an initialised directory still hands straight over
with exec, so mongod keeps PID 1 and its signal handling on every boot
after the first.

Verified by holding 27017 across the real server's bind on a first boot:
before, the container exits 133 and never serves; after, it reports the
conflict, retries and comes up.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

The PR appears safe to merge, with the previously identified shutdown and initialization-failure handling defects resolved.

Summary

  • Retries only after MongoDB reports its network initialization exit status.
  • Propagates other initialization failures without retrying.
  • Forwards termination signals, waits for child shutdown, and avoids restarting after an explicit stop.
  • Retains direct exec behavior for initialized data directories.

Reviews (2) · Last reviewed commit: "fix(mongo): stop on a signal and raise a..."

Comment thread mongo-entrypoint.sh Outdated
Comment thread mongo-entrypoint.sh Outdated
@github-actions

github-actions Bot commented Sep 10, 2026

Copy link
Copy Markdown

✨ Benchmark results

Comparing main (before) → fix/mongo-first-boot-bind-race-main (after).

⚠️ Before benchmark did not complete — change column unavailable.

Metric Before After Change
🚀 Requests/sec n/a 266.32 n/a
⏱️ Latency P50 n/a 65.78 ms n/a
⏱️ Latency P95 n/a 151.11 ms n/a
Per-scenario breakdown & investigation details

Metrics below reflect the current branch (after). Δ P95 compares against the base.

Scenario P50 (ms) P95 (ms) Requests RPS Δ P95 (ms)
API total 65.78 151.11 16,758 266.32 n/a
Account 129.41 239.53 882 14.64 n/a
TablesDB 63.7 115.44 9,114 148.14 n/a
Storage 58.63 129.07 4,410 73.23 n/a
Functions 95.43 181.15 2,352 39.54 n/a

Top API waits (after)

API request Max wait (ms)
functions.variables.update 368.49
account.name.update 360.02
storage.buckets.create 355.53
account.prefs.update 347.28
functions.create 303.66

…race

The first-boot retry treated every failure, and every signal, as a reason
to start MongoDB again.

`docker stop` inside the first-boot window killed the child, `wait`
returned, and the script started the server the caller had just asked to
go away. The container then sat there until Docker force-killed it at the
end of the grace period: measured on mongo:8.2.5, 30s and exit 137 both
mid-initialisation and once the server was already serving, against 0s
and exit 143 now.

An initialisation script that failed once the storage files existed was
worse. The second pass found /data/db populated, skipped
/docker-entrypoint-initdb.d, and served a database with no application
user while the original failure went unreported. Measured: container
running, reachable, application user absent, against exit 1 and no server
now.

Retry only mongod's EXIT_NET_ERROR. 48 is what it exits with when it
cannot bind, and the standard entrypoint ends in `exec "$@"`, so the
status arrives unchanged; a temporary server that loses the bind fails
inside `mongod --fork`, whose parent reports 1, so only the real server
can reach 48.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@abnegate

Copy link
Copy Markdown
Member Author

Both P1s were valid and are fixed in 87449f3d06. Recording it here because the push moved the anchored lines, so both threads auto-resolved as outdated rather than being answered.

Shutdown Restarts MongoDB. Correct, and worse than a lingering container: the script started the server the caller had just asked to go away. The trap now records which signal arrived, forwards it, and — because a trapped signal makes wait return 128+signal without reaping — waits again until the child is actually gone, so mongod finishes its own shutdown before PID 1 leaves and takes the container with it. It then exits 128+signal instead of starting anything.

Measured on mongo:8.2.5, docker stop both mid-initialisation and once the server was already serving:

before after
stop duration 30s (Docker's grace) 0s
exit code 137 (SIGKILL) 143 (SIGTERM)

All Failures Trigger Retry. Also correct, and the more dangerous of the two. An initialisation script failing after the storage files existed left the second pass looking at a populated /data/db, so it skipped /docker-entrypoint-initdb.d entirely and served a database with no application user, with the original failure unreported. Measured: container running and reachable with the user absent, versus exit 1 and no server now.

The retry is now gated on 48 only — mongod's EXIT_NET_ERROR, what it exits with when it cannot bind. The status arrives unchanged because the standard entrypoint ends in exec "$@", and a temporary server that loses the bind fails inside mongod --fork, whose parent reports 1 — so only the real server can reach 48. Every other status is raised.

On the one red check: Tests / E2E / Antivirus (dedicated) is not this change. main pins appwrite/php-clamav at 2.0.0, which sizes every INSTREAM frame at the chunk length rather than the bytes read and loops on !feof(), so it sends roughly twice the file and desyncs the framing — the log here shows INSTREAM: Size limit reached, (requested: 247036), a length read out of file bytes. ClamAV answers with an error, the bool API reports that as a detection, and Storage deletes the upload and returns 403. Fixed in php-clamav 2.0.2; that lane is 2 failures in 19 runs on main itself. Bumping it is a separate change, not one to bury in a MongoDB entrypoint PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant