π Description
After upgrading Appwrite from 1.9.0 to 1.9.6, openruntimes-executor went into a permanent crash loop. Took a while to figure out why because the error message is useless:
Fatal error: Uncaught Utopia\DI\Exceptions\NotFoundException: Dependency response not found in .../di/src/DI/Container.php:74
...
Next Utopia\Http\Exception: Error handler had an error: Dependency response not found in .../Http/Http.php:534
This loops forever and never says what actually failed.
Turns out the real cause is in app/http.php's onStart hook:
$hostname = gethostname();
$selfContainer = $orchestration->list(['name' => $hostname])[0] ?? throw new \RuntimeException('Own container not found');
It looks itself up in Docker by matching hostname against the container name. Our compose had container_name: openruntimes-executor and hostname: exc1, different values, so the lookup fails and it throws.
Checked appwrite/appwrite's own compose history: up to 1.9.0 they also had container_name different from hostname, and only made them match starting in 1.9.5. So this became a real requirement at some point but was never called out anywhere, not in the changelog, not in the upgrade notes for 1.9.5/1.9.6. If you're upgrading from an older install you just hit this with zero warning.
On top of that, the reason we never saw the real error message: when the onStart hook throws, the framework tries to run the global error hooks, which need a response dependency that doesn't exist yet at startup time. That lookup fails too, and the second failure is all that gets logged. The original Own container not found never makes it to the logs.
Two asks:
- Document that
container_name must match hostname for the executor, this is a breaking requirement for anyone upgrading from pre-1.9.5.
- Fix the error handling so a failure during
onStart actually gets logged before the error-hook machinery kicks in and (possibly) fails on its own.
Reproduced on executor 0.25.4 and 0.27.0, self-hosted via Docker Compose.
This should fix #12796 , same symptom, but that report doesn't identify the actual trigger. Here's the root cause for Docker Compose setups.
π Have you spent some time to check if this issue has been raised before?
π’ Have you read the Code of Conduct?
π Description
After upgrading Appwrite from 1.9.0 to 1.9.6, openruntimes-executor went into a permanent crash loop. Took a while to figure out why because the error message is useless:
This loops forever and never says what actually failed.
Turns out the real cause is in
app/http.php'sonStarthook:It looks itself up in Docker by matching
hostnameagainst the containername. Our compose hadcontainer_name: openruntimes-executorandhostname: exc1, different values, so the lookup fails and it throws.Checked appwrite/appwrite's own compose history: up to 1.9.0 they also had
container_namedifferent fromhostname, and only made them match starting in 1.9.5. So this became a real requirement at some point but was never called out anywhere, not in the changelog, not in the upgrade notes for 1.9.5/1.9.6. If you're upgrading from an older install you just hit this with zero warning.On top of that, the reason we never saw the real error message: when the
onStarthook throws, the framework tries to run the global error hooks, which need aresponsedependency that doesn't exist yet at startup time. That lookup fails too, and the second failure is all that gets logged. The originalOwn container not foundnever makes it to the logs.Two asks:
container_namemust matchhostnamefor the executor, this is a breaking requirement for anyone upgrading from pre-1.9.5.onStartactually gets logged before the error-hook machinery kicks in and (possibly) fails on its own.Reproduced on executor 0.25.4 and 0.27.0, self-hosted via Docker Compose.
This should fix #12796 , same symptom, but that report doesn't identify the actual trigger. Here's the root cause for Docker Compose setups.
π Have you spent some time to check if this issue has been raised before?
π’ Have you read the Code of Conduct?