fix(mongo): initialise the data directory off the port the server binds - #13593
Open
abnegate wants to merge 2 commits into
Open
fix(mongo): initialise the data directory off the port the server binds#13593abnegate wants to merge 2 commits into
abnegate wants to merge 2 commits into
Conversation
On a fresh volume the standard mongo entrypoint creates the users and runs /docker-entrypoint-initdb.d against a temporary mongod on 127.0.0.1:27017, stops it with `mongod --shutdown`, and execs the real server on 0.0.0.0:27017. That handover is unsynchronised. `mongod --shutdown` returns once the server it signalled clears <dbpath>/mongod.lock, but a mongod releases its listening socket only when the process exits, several steps after the lock goes. The real server binds about a millisecond after it starts, so whenever the tail of the temporary server's shutdown outlasts the handover it gets Error setting up transport layer ... 0.0.0.0:27017 :: caused by :: setup bind :: caused by :: Address already in use and exits 48. The container goes with it, so `docker compose up -d --wait` fails every service that depends_on mongodb and the lane reports "dependency failed to start: container appwrite-mongodb is unhealthy" having run no tests at all. Run 34374778610 lost Tests / E2E / PostgreSQL (shared) / Locale that way: the temporary server reached "Dropping the scope cache for shutdown" and had still not logged "Now exiting" when the real one tried to bind 1.08s later. Initialise here instead, on a port the real server never binds, and wait on the process rather than on its lock file. The standard entrypoint then finds an initialised directory, starts no second server and execs straight through, so mongod is still PID 1 on every boot including the first. The users are unchanged: root@admin with the root role, and _APP_DB_USER@admin with readWrite on _APP_DB_SCHEMA. mongo-init.js moves out of /docker-entrypoint-initdb.d, because anything left in there sends the standard entrypoint back to the temporary server regardless of the data directory. mongo-init-replicaset.sh goes too: unreferenced since the healthcheck took over replica set initiation, and written to be mounted into that same directory. Verified by freezing the temporary server the instant it clears mongod.lock, which is what a loaded runner does to its shutdown tail: the current entrypoint exits 48 on every attempt, this one serves. Also five `docker compose down -v` / `up -d --wait` cycles on a fresh volume, and a restart on an initialised one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
|
✨ Benchmark resultsComparing
Per-scenario breakdown & investigation detailsMetrics below reflect the current branch (after). Δ P95 compares against the base.
Top API waits (after)
|
mongod creates /data/db/WiredTiger before either user is created, so gating on that file meant a first boot interrupted between the two -- a killed container, a failed mongosh -- came back with authentication on and nobody to authenticate as, with no way out but deleting the volume. Gate on a marker written only once every user exists, and make each step skippable so a resumed pass finishes whatever did not happen. Checked against a data directory seeded with no users and one seeded with root but not the application user: both now reach a serving container holding both users, where the previous commit ran no initialisation at all and never accepted a login. A directory the released entrypoint initialised is picked up unchanged, replica set and all, by the one resumable pass it takes to write the marker. The compose generator test drops the mount literals it was mirroring and asserts the property that makes the allowlist load-bearing: a generated deployment carries no bind mount relative to the repository, because it is not run from there. It still fails on the stale allowlist, and now also catches a relative mount nobody added to the allowlist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
docker compose up -d --waitintermittently fails before any test runs, on a freshvolume only:
Seen on run 34374778610,
job
Tests / E2E / PostgreSQL (shared) / Locale. The lane reports a health failureand skips "Run tests" entirely.
Why
On a fresh
/data/dbthe standard mongo entrypoint creates the users and runs/docker-entrypoint-initdb.dagainst a temporary mongod on127.0.0.1:27017, stopsit with
mongod --shutdown, then execs the real server on0.0.0.0:27017.That handover is unsynchronised:
mongod --shutdownreturns once the server it signalled clears<dbpath>/mongod.lock. Verified directly: point--shutdownat a process thatignores
SIGTERMand it blocks; deletemongod.lockunderneath it and it returnsimmediately, with the process still running. Deleting the
--pidfilepathfileinstead changes nothing.
steps after the lock goes.
"Shutdown: Closing listener sockets"is logged while127.0.0.1:27017is stillLISTENin/proc/net/tcp.setUpTransportLayerruns beforeWiredTiger), so there is no slack to absorb a slow shutdown tail.
When the tail outlasts the handover:
mongod exits 48, the container goes with it, and every service that
depends_onmongodb fails under
--wait. In the CI dump the temporary server had reached"Dropping the scope cache for shutdown"and had still not logged"Now exiting"when the real one tried to bind 1.08s later.
Two things it is not:
--bind_ip_allis not carried into the init phase (thestandard entrypoint strips it and pins
--bind_ip 127.0.0.1 --port 27017, and the CIlog confirms
"bindIp":"127.0.0.1"); and lingeringTIME_WAIT/FIN_WAIT2sockets on127.0.0.1:27017do not block the wildcard bind, which I checked by holdingconnections open across a shutdown and binding over them successfully.
How
Initialise the data directory in
mongo-entrypoint.sh, on27018, and wait on theprocess (
wait "$INIT_SERVER") rather than on its lock file. The standardentrypoint then finds an initialised directory, starts no second server, and execs
straight through, so
mongodis PID 1 on every boot including the first.mongo-init.jsmoves out of/docker-entrypoint-initdb.d, because anything left inthere sends the standard entrypoint back to the temporary server regardless of the
data directory.
mongo-init-replicaset.shgoes too: unreferenced since thehealthcheck took over replica set initiation, and written to be mounted into that
same directory.
Users are unchanged, byte for byte against the current entrypoint:
root@adminwith therootrole, and_APP_DB_USER@adminwithreadWriteon_APP_DB_SCHEMA.Verification
Forced the race by freezing the temporary server the instant it clears
mongod.lock, which is what a loaded runner does to its shutdown tail:Address already in usemongodPID 1, both usersdocker compose down -vthenup -d --wait mongodb appwrite-mongo-express, fresh volumeisWritablePrimary=true setName=rs0tests/unit/Docker/Compose/GeneratorTest.php11/11. The newtestRewritesMongoBindMountsToHostPathOnPublishedVersionscovers a real gap: theexisting rewrite test passes
version => 'local', which short-circuits theHOST_PATH_REWRITABLE_BINDScheck, so nothing exercised the constant this PR edits.Seen red against the pre-change
Generator.phpand green after.composer lint(Pint, PSR-12) clean on both PHP files,prettier --checkclean onthe compose files,
shellcheckclean onmongo-entrypoint.sh.Not changed
The healthcheck still doubles as the replica set initializer. Reviewed while here,
left alone as out of scope for this failure, worth a follow-up:
rs.initiateruns from the health probe, so the set is not initiated until thefirst probe one full
intervalafter start, and mongodb needs two probes (~20s) toreport healthy on every fresh stack.
2>/dev/nullplus two swallowedcatch (e) {}make a wrong password or a rejectedinitiate indistinguishable from "not ready yet".
.State.Health.Logis alwaysempty, which is the opposite of what you want when diagnosing this class of CI
failure.
Supersedes #13591, which takes the failure as given and restarts mongod after it, and
whose commit message attributes the collision to the real server spending "about a
second opening WiredTiger before it binds" (it binds first, ~1ms in).
🤖 Generated with Claude Code