Skip to content

openruntimes-executor crash loop after upgrading Appwrite 1.9.0 β†’ 1.9.6 (container_name/hostname mismatch, undocumented)Β #13016

Description

@svprdga

πŸ’­ Description

After upgrading Appwrite from 1.9.0 to 1.9.6, openruntimes-executor went into a permanent crash loop. Took a while to figure out why because the error message is useless:

Fatal error: Uncaught Utopia\DI\Exceptions\NotFoundException: Dependency response not found in .../di/src/DI/Container.php:74
...
Next Utopia\Http\Exception: Error handler had an error: Dependency response not found in .../Http/Http.php:534

This loops forever and never says what actually failed.

Turns out the real cause is in app/http.php's onStart hook:

$hostname = gethostname();
$selfContainer = $orchestration->list(['name' => $hostname])[0] ?? throw new \RuntimeException('Own container not found');

It looks itself up in Docker by matching hostname against the container name. Our compose had container_name: openruntimes-executor and hostname: exc1, different values, so the lookup fails and it throws.

Checked appwrite/appwrite's own compose history: up to 1.9.0 they also had container_name different from hostname, and only made them match starting in 1.9.5. So this became a real requirement at some point but was never called out anywhere, not in the changelog, not in the upgrade notes for 1.9.5/1.9.6. If you're upgrading from an older install you just hit this with zero warning.

On top of that, the reason we never saw the real error message: when the onStart hook throws, the framework tries to run the global error hooks, which need a response dependency that doesn't exist yet at startup time. That lookup fails too, and the second failure is all that gets logged. The original Own container not found never makes it to the logs.

Two asks:

  1. Document that container_name must match hostname for the executor, this is a breaking requirement for anyone upgrading from pre-1.9.5.
  2. Fix the error handling so a failure during onStart actually gets logged before the error-hook machinery kicks in and (possibly) fails on its own.

Reproduced on executor 0.25.4 and 0.27.0, self-hosted via Docker Compose.

This should fix #12796 , same symptom, but that report doesn't identify the actual trigger. Here's the root cause for Docker Compose setups.

πŸ‘€ Have you spent some time to check if this issue has been raised before?

  • I checked and didn't find similar issue

🏒 Have you read the Code of Conduct?

Metadata

Metadata

Assignees

No one assigned

    Labels

    product / databasesFixes and upgrades for the Appwrite Database.product / functionsFixes and upgrades for the Appwrite Functions.product / self-hostedIssues only found when self-hosting Appwriteproduct / vcsFixes and upgrades for the Appwrite VCS.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions