Skip to content

fix(mongo): initialise the data directory off the port the server binds - #13593

Open
abnegate wants to merge 2 commits into
mainfrom
fix/mongo-first-boot-port-handover
Open

fix(mongo): initialise the data directory off the port the server binds#13593
abnegate wants to merge 2 commits into
mainfrom
fix/mongo-first-boot-port-handover

Conversation

@abnegate

Copy link
Copy Markdown
Member

What

docker compose up -d --wait intermittently fails before any test runs, on a fresh
volume only:

 Container appwrite-mongodb  Error
dependency failed to start: container appwrite-mongodb is unhealthy

Seen on run 34374778610,
job Tests / E2E / PostgreSQL (shared) / Locale. The lane reports a health failure
and skips "Run tests" entirely.

Why

On a fresh /data/db the standard mongo entrypoint creates the users and runs
/docker-entrypoint-initdb.d against a temporary mongod on 127.0.0.1:27017, stops
it with mongod --shutdown, then execs the real server on 0.0.0.0:27017.

That handover is unsynchronised:

  • mongod --shutdown returns once the server it signalled clears
    <dbpath>/mongod.lock. Verified directly: point --shutdown at a process that
    ignores SIGTERM and it blocks; delete mongod.lock underneath it and it returns
    immediately, with the process still running. Deleting the --pidfilepath file
    instead changes nothing.
  • A mongod releases its listening socket only when the process exits, several
    steps after the lock goes. "Shutdown: Closing listener sockets" is logged while
    127.0.0.1:27017 is still LISTEN in /proc/net/tcp.
  • The real server binds ~1ms after it starts (setUpTransportLayer runs before
    WiredTiger), so there is no slack to absorb a slow shutdown tail.

When the tail outlasts the handover:

Error setting up transport layer ... 0.0.0.0:27017 :: caused by ::
setup bind :: caused by :: Address already in use

mongod exits 48, the container goes with it, and every service that depends_on
mongodb fails under --wait. In the CI dump the temporary server had reached
"Dropping the scope cache for shutdown" and had still not logged "Now exiting"
when the real one tried to bind 1.08s later.

Two things it is not: --bind_ip_all is not carried into the init phase (the
standard entrypoint strips it and pins --bind_ip 127.0.0.1 --port 27017, and the CI
log confirms "bindIp":"127.0.0.1"); and lingering TIME_WAIT/FIN_WAIT2 sockets on
127.0.0.1:27017 do not block the wildcard bind, which I checked by holding
connections open across a shutdown and binding over them successfully.

How

Initialise the data directory in mongo-entrypoint.sh, on 27018, and wait on the
process (wait "$INIT_SERVER") rather than on its lock file. The standard
entrypoint then finds an initialised directory, starts no second server, and execs
straight through, so mongod is PID 1 on every boot including the first.

mongo-init.js moves out of /docker-entrypoint-initdb.d, because anything left in
there sends the standard entrypoint back to the temporary server regardless of the
data directory. mongo-init-replicaset.sh goes too: unreferenced since the
healthcheck took over replica set initiation, and written to be mounted into that
same directory.

Users are unchanged, byte for byte against the current entrypoint:
root@admin with the root role, and _APP_DB_USER@admin with readWrite on
_APP_DB_SCHEMA.

Verification

Forced the race by freezing the temporary server the instant it clears
mongod.lock, which is what a loaded runner does to its shutdown tail:

result
current entrypoint, stalled tail 3/3 exit 48, Address already in use
this entrypoint, same stall 3/3 served, mongod PID 1, both users
this entrypoint, no stall 2/2 served
docker compose down -v then up -d --wait mongodb appwrite-mongo-express, fresh volume 5/5 pass (12-16s)
restart on an initialised volume init skipped, isWritablePrimary=true setName=rs0
healthcheck run verbatim probe 1 initiates and returns 1, probe 2 returns 0

tests/unit/Docker/Compose/GeneratorTest.php 11/11. The new
testRewritesMongoBindMountsToHostPathOnPublishedVersions covers a real gap: the
existing rewrite test passes version => 'local', which short-circuits the
HOST_PATH_REWRITABLE_BINDS check, so nothing exercised the constant this PR edits.
Seen red against the pre-change Generator.php and green after.

composer lint (Pint, PSR-12) clean on both PHP files, prettier --check clean on
the compose files, shellcheck clean on mongo-entrypoint.sh.

Not changed

The healthcheck still doubles as the replica set initializer. Reviewed while here,
left alone as out of scope for this failure, worth a follow-up:

  • rs.initiate runs from the health probe, so the set is not initiated until the
    first probe one full interval after start, and mongodb needs two probes (~20s) to
    report healthy on every fresh stack.
  • 2>/dev/null plus two swallowed catch (e) {} make a wrong password or a rejected
    initiate indistinguishable from "not ready yet". .State.Health.Log is always
    empty, which is the opposite of what you want when diagnosing this class of CI
    failure.

Supersedes #13591, which takes the failure as given and restarts mongod after it, and
whose commit message attributes the collision to the real server spending "about a
second opening WiredTiger before it binds" (it binds first, ~1ms in).

🤖 Generated with Claude Code

On a fresh volume the standard mongo entrypoint creates the users and runs
/docker-entrypoint-initdb.d against a temporary mongod on 127.0.0.1:27017, stops
it with `mongod --shutdown`, and execs the real server on 0.0.0.0:27017.

That handover is unsynchronised. `mongod --shutdown` returns once the server it
signalled clears <dbpath>/mongod.lock, but a mongod releases its listening socket
only when the process exits, several steps after the lock goes. The real server
binds about a millisecond after it starts, so whenever the tail of the temporary
server's shutdown outlasts the handover it gets

  Error setting up transport layer ... 0.0.0.0:27017 :: caused by ::
  setup bind :: caused by :: Address already in use

and exits 48. The container goes with it, so `docker compose up -d --wait` fails
every service that depends_on mongodb and the lane reports "dependency failed to
start: container appwrite-mongodb is unhealthy" having run no tests at all. Run
34374778610 lost Tests / E2E / PostgreSQL (shared) / Locale that way: the
temporary server reached "Dropping the scope cache for shutdown" and had still
not logged "Now exiting" when the real one tried to bind 1.08s later.

Initialise here instead, on a port the real server never binds, and wait on the
process rather than on its lock file. The standard entrypoint then finds an
initialised directory, starts no second server and execs straight through, so
mongod is still PID 1 on every boot including the first. The users are unchanged:
root@admin with the root role, and _APP_DB_USER@admin with readWrite on
_APP_DB_SCHEMA.

mongo-init.js moves out of /docker-entrypoint-initdb.d, because anything left in
there sends the standard entrypoint back to the temporary server regardless of
the data directory. mongo-init-replicaset.sh goes too: unreferenced since the
healthcheck took over replica set initiation, and written to be mounted into that
same directory.

Verified by freezing the temporary server the instant it clears mongod.lock,
which is what a loaded runner does to its shutdown tail: the current entrypoint
exits 48 on every attempt, this one serves. Also five `docker compose down -v` /
`up -d --wait` cycles on a fresh volume, and a restart on an initialised one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

The code changes appear safe to merge, with only the existing non-blocking test-design concern remaining.

Fix All in Claude CodeFindings

  1. P2 Test mirrors mount configuration
Fix with agent prompt
### Issue 1
tests/unit/Docker/Compose/GeneratorTest.php:139-140
These assertions hard-code the exact source paths, container destinations, and mount modes already defined by the compose template and rewrite allowlist. This couples the test to implementation details, so harmless mount-layout refactors fail without proving that generated deployments can initialize MongoDB. The repository requires tests to validate observable behavior rather than mirror source code or configuration, and that requirement must be satisfied before merging. Replace these assertions with a behavioral check at the deployment or initialization boundary. The same pattern also appears in the updated assertions at lines 104–105 and 126–129.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

  • Adds resumable, marker-based creation of the root and application users.
  • Removes the obsolete replica-set initialization script.
  • Updates Compose generation and its bind-mount tests.
  • The prior implementation-coupled-test finding remains outstanding: although the newly added test now checks the generated deployment invariant, exact mount strings are still mirrored at lines 104–105 and 126–129.

Reviews (2) · Last reviewed commit: "fix(mongo): make first-boot initialisati..."

Comment thread mongo-entrypoint.sh Outdated
Comment thread tests/unit/Docker/Compose/GeneratorTest.php Outdated
@github-actions

github-actions Bot commented Sep 10, 2026

Copy link
Copy Markdown

✨ Benchmark results

Comparing main (before) → fix/mongo-first-boot-port-handover (after).

Metric Before After Change
🚀 Requests/sec 267.7 269.96 +0.8%
⏱️ Latency P50 64.62 ms 65.52 ms +1.4%
⏱️ Latency P95 153.39 ms 150.59 ms -1.8%
Per-scenario breakdown & investigation details

Metrics below reflect the current branch (after). Δ P95 compares against the base.

Scenario P50 (ms) P95 (ms) Requests RPS Δ P95 (ms)
API total 65.52 150.59 17,100 269.96 -2.8
Account 127.53 230.64 900 14.97 -7.35
TablesDB 63.45 112.67 9,300 150.06 -4.53
Storage 60.13 129.96 4,500 74.2 +3.63
Functions 96.66 179.09 2,400 40.26 -3.84

Top API waits (after)

API request Max wait (ms)
account.name.update 340.03
functions.create 277.17
tablesdb.rows.list 274.04
storage.files.create 273.95
storage.buckets.create 271.55

mongod creates /data/db/WiredTiger before either user is created, so gating on
that file meant a first boot interrupted between the two -- a killed container,
a failed mongosh -- came back with authentication on and nobody to authenticate
as, with no way out but deleting the volume.

Gate on a marker written only once every user exists, and make each step
skippable so a resumed pass finishes whatever did not happen. Checked against a
data directory seeded with no users and one seeded with root but not the
application user: both now reach a serving container holding both users, where
the previous commit ran no initialisation at all and never accepted a login. A
directory the released entrypoint initialised is picked up unchanged, replica
set and all, by the one resumable pass it takes to write the marker.

The compose generator test drops the mount literals it was mirroring and asserts
the property that makes the allowlist load-bearing: a generated deployment
carries no bind mount relative to the repository, because it is not run from
there. It still fails on the stale allowlist, and now also catches a relative
mount nobody added to the allowlist.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant