Skip to content

Improve sites test for stability#10331

Merged
Meldiron merged 6 commits into1.7.xfrom
fix-cicd-tests-sites
Aug 18, 2025
Merged

Improve sites test for stability#10331
Meldiron merged 6 commits into1.7.xfrom
fix-cicd-tests-sites

Conversation

@Meldiron
Copy link
Copy Markdown
Contributor

What does this PR do?

Improves of stability of Sites tests

Test Plan

(Write your test plan here. If you changed any code, please provide us with clear instructions on how you verified your changes work. Screenshots may also be helpful.)

Related PRs and Issues

  • (Related PR or issue)

Checklist

  • Have you read the Contributing Guidelines on issues?
  • If the PR includes a change to an API's metadata (desc, label, params, etc.), does it also include updated API specs and example docs?

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Aug 16, 2025

📝 Walkthrough

Walkthrough

Added three test-only volume mounts to the appwrite service in docker-compose.yml. Enhanced E2E site deployment test flow: fail early on deployment status "failed", inspect openruntimes-executor ExitCode and collect executor logs on non-zero, and increased assertEventually timeout. Made several E2E test stability changes (assertEventually for adapter detection, added #[Retry(count: 3)] to three tests). Added a post-failure "Failure Logs" step to multiple GitHub Actions E2E jobs to print docker compose logs. Added extensive logging across the Builds worker. Introduced a Critical exception type and made eventual retry logic rethrow Critical immediately.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

  • Fix: logs order #10052 — Overlaps changes in the Builds worker (logging and post-build/screenshot sequencing) that this PR also modifies.
  • Design nice looking error pages #9606 — Modifies Sites E2E tests and readiness/assert logic; closely related to the test stability and assertEventually changes here.

Suggested reviewers

  • eldadfux
  • loks0n

Tip

🔌 Remote MCP (Model Context Protocol) integration is now available!

Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats.


📜 Recent review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

💡 Knowledge Base configuration:

  • MCP integration is disabled by default for public repositories
  • Jira integration is disabled by default for public repositories
  • Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between aed97c5 and a754763.

📒 Files selected for processing (1)
  • tests/e2e/Services/Sites/SitesCustomServerTest.php (3 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • tests/e2e/Services/Sites/SitesCustomServerTest.php
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
  • GitHub Check: Benchmark
  • GitHub Check: scan
✨ Finishing Touches
  • 📝 Generate Docstrings
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch fix-cicd-tests-sites

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

Status, Documentation and Community

  • Visit our Status Page to check the current availability of CodeRabbit.
  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Aug 16, 2025

Security Scan Results for PR

Docker Image Scan Results

Package Version Vulnerability Severity
binutils 2.42-r0 CVE-2025-0840 HIGH
git 2.45.3-r0 CVE-2025-48384 HIGH
git 2.45.3-r0 CVE-2025-48385 HIGH
git-init-template 2.45.3-r0 CVE-2025-48384 HIGH
git-init-template 2.45.3-r0 CVE-2025-48385 HIGH
icu 74.2-r0 CVE-2025-5222 HIGH
icu-data-en 74.2-r0 CVE-2025-5222 HIGH
icu-dev 74.2-r0 CVE-2025-5222 HIGH
icu-libs 74.2-r0 CVE-2025-5222 HIGH
libexpat 2.6.4-r0 CVE-2024-8176 HIGH
libxml2 2.12.7-r0 CVE-2024-56171 HIGH
libxml2 2.12.7-r0 CVE-2025-24928 HIGH
libxml2 2.12.7-r0 CVE-2025-27113 HIGH
libxml2 2.12.7-r0 CVE-2025-32414 HIGH
libxml2 2.12.7-r0 CVE-2025-32415 HIGH
pyc 3.12.9-r0 CVE-2024-12718 HIGH
pyc 3.12.9-r0 CVE-2025-4138 HIGH
pyc 3.12.9-r0 CVE-2025-4330 HIGH
pyc 3.12.9-r0 CVE-2025-4517 HIGH
python3 3.12.9-r0 CVE-2024-12718 HIGH
python3 3.12.9-r0 CVE-2025-4138 HIGH
python3 3.12.9-r0 CVE-2025-4330 HIGH
python3 3.12.9-r0 CVE-2025-4517 HIGH
python3-pyc 3.12.9-r0 CVE-2024-12718 HIGH
python3-pyc 3.12.9-r0 CVE-2025-4138 HIGH
python3-pyc 3.12.9-r0 CVE-2025-4330 HIGH
python3-pyc 3.12.9-r0 CVE-2025-4517 HIGH
python3-pycache-pyc0 3.12.9-r0 CVE-2024-12718 HIGH
python3-pycache-pyc0 3.12.9-r0 CVE-2025-4138 HIGH
python3-pycache-pyc0 3.12.9-r0 CVE-2025-4330 HIGH
python3-pycache-pyc0 3.12.9-r0 CVE-2025-4517 HIGH
sqlite-libs 3.45.3-r1 CVE-2025-29087 HIGH
xz 5.6.2-r0 CVE-2025-31115 HIGH
xz-libs 5.6.2-r0 CVE-2025-31115 HIGH
golang.org/x/crypto v0.31.0 CVE-2025-22869 HIGH
golang.org/x/oauth2 v0.24.0 CVE-2025-22868 HIGH
stdlib 1.22.10 CVE-2025-47907 HIGH

Source Code Scan Results

🎉 No vulnerabilities found!

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
docker-compose.yml (1)

74-74: Guard docker.sock mount behind a tests-only override or profile

Mounting the host Docker socket into the appwrite service is powerful and risky. Even though this is a dev compose and the comment says “Only needed for tests,” it grants root-equivalent control of the host Docker daemon to the container.

  • Prefer moving this mount into a dedicated tests-only override (e.g., docker-compose.tests.yml) and using it only in CI: docker compose -f docker-compose.yml -f docker-compose.tests.yml up.
  • Alternatively, gate under a compose profile used only in tests.

Also, note that tests call docker compose logs ... from inside the appwrite container. That requires the Docker CLI and compose plugin to be present in the image. If that’s not guaranteed, consider switching the test code to docker logs (see inline suggestion in SitesBase.php).

Would you like me to draft a docker-compose.tests.yml override that adds only this mount for CI runs?

tests/e2e/Services/Sites/SitesCustomServerTest.php (1)

408-411: Avoid passing $site by reference and set explicit polling budget

The reference capture isn’t used later and can confuse readers. Also, consider setting an explicit timeout/interval for consistency with other tests.

Apply this diff:

-        $this->assertEventually(function () use ($siteId, &$site) {
-            $site = $this->getSite($siteId);
-            $this->assertEquals('ssr', $site['body']['adapter']);
-        });
+        $this->assertEventually(function () use ($siteId) {
+            $site = $this->getSite($siteId);
+            $this->assertEquals('ssr', $site['body']['adapter']);
+        }, 100000, 500);
📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

💡 Knowledge Base configuration:

  • MCP integration is disabled by default for public repositories
  • Jira integration is disabled by default for public repositories
  • Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between 78c5d5a and 95e54d0.

📒 Files selected for processing (3)
  • docker-compose.yml (1 hunks)
  • tests/e2e/Services/Sites/SitesBase.php (1 hunks)
  • tests/e2e/Services/Sites/SitesCustomServerTest.php (1 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (2)
tests/e2e/Services/Sites/SitesCustomServerTest.php (1)
tests/e2e/Services/Sites/SitesBase.php (1)
  • getSite (176-184)
tests/e2e/Services/Sites/SitesBase.php (1)
src/Appwrite/Platform/Workers/Functions.php (1)
  • fail (256-305)
🔇 Additional comments (2)
tests/e2e/Services/Sites/SitesBase.php (2)

52-54: Early fail on deployment failures is a solid stability improvement

Failing fast when deployment status is "failed" will shorten feedback loops and produce clearer diagnostics. LGTM.


67-67: Extended eventual timeout to 300s is reasonable for flakier builds

Doubling the wait budget helps stabilize E2E in CI without over-polling.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Aug 16, 2025

✨ Benchmark results

  • Requests per second: 1,035
  • Requests with 200 status code: 186,409
  • P99 latency: 0.177873327

⚡ Benchmark Comparison

Metric This PR Latest version
RPS 1,035 1,004
200 186,409 180,733
P99 0.177873327 0.193746007

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

♻️ Duplicate comments (1)
tests/e2e/Services/Sites/SitesBase.php (1)

57-66: Harden exit code check and avoid docker compose plugin dependency for logs

  • Compare numeric exit code, not string, to avoid surprises.
  • Prefer docker logs with the explicit container name to avoid requiring the Compose plugin inside the test container, and to limit output size for readability.

Apply this diff:

-            Console::execute("docker inspect openruntimes-executor --format='{{.State.ExitCode}}'", '', $this->stdout, $this->stderr);
-            if (\trim($this->stdout) !== '0') {
-                $msg = 'Executor has a problem: ' . $this->stderr . ' (' . $this->stdout . '), current status: ';
-
-                Console::execute("docker compose logs openruntimes-executor", '', $this->stdout, $this->stderr);
-                $msg .= $this->stdout . ' (' . $this->stderr . ')';
-
-                throw new Critical($msg . json_encode($deployment['body'], JSON_PRETTY_PRINT));
-            }
+            $inspectOut = '';
+            $inspectErr = '';
+            Console::execute("docker inspect openruntimes-executor --format='{{.State.ExitCode}}'", '', $inspectOut, $inspectErr);
+            $exitCode = (int) \trim($inspectOut);
+            if ($exitCode !== 0) {
+                $msg = 'Executor has a problem: ' . \trim($inspectErr) . ' (exitCode=' . $exitCode . '), current status: ';
+
+                $logsOut = '';
+                $logsErr = '';
+                // Avoid compose plugin dependency; limit log size for readability
+                Console::execute("docker logs --tail=500 openruntimes-executor", '', $logsOut, $logsErr);
+                $msg .= $logsOut . ' (' . $logsErr . ')';
+
+                throw new Critical($msg . json_encode($deployment['body'], JSON_PRETTY_PRINT));
+            }
🧹 Nitpick comments (3)
.github/workflows/tests.yml (1)

137-144: Bound and harden failure log collection to keep CI output readable and resilient

Large unbounded logs can overwhelm CI output, and docker compose may exit non‑zero if a service is absent. Tail and no-color reduce noise; || true ensures logs collection never masks the original failure.

Apply this diff to all Failure Logs steps:

-          echo "=== Appwrite Worker Builds Logs ==="
-          docker compose logs appwrite-worker-builds
-          echo "=== OpenRuntimes Executor Logs ==="
-          docker compose logs openruntimes-executor
+          echo "=== Appwrite Worker Builds Logs ==="
+          docker compose logs --tail=200 --no-color appwrite-worker-builds || true
+          echo "=== OpenRuntimes Executor Logs ==="
+          docker compose logs --tail=200 --no-color openruntimes-executor || true

Also applies to: 212-219, 299-306, 340-347, 395-402, 437-444, 493-499

tests/extensions/Async/Eventually.php (1)

36-41: Guard against null throw on timeout with no captured exception

If the probe never throws an Exception (e.g., throws an Error/Throwable or returns falsey without asserting), $lastException could be null leading to a fatal when thrown. Prefer a safe fallback.

Example adjustment (outside the selected lines):

// before
throw $lastException;

// after
throw $lastException ?? new \RuntimeException('Eventually timed out without a captured exception.');
src/Appwrite/Platform/Modules/Functions/Workers/Builds.php (1)

822-825: Use info level for normal “finished” messages

listLogs finished reads as a normal completion, not a warning. Consider lowering the severity.

-                        Console::warning('listLogs finished');
+                        Console::info('listLogs finished');
📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

💡 Knowledge Base configuration:

  • MCP integration is disabled by default for public repositories
  • Jira integration is disabled by default for public repositories
  • Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between 95e54d0 and 746a76d.

📒 Files selected for processing (6)
  • .github/workflows/tests.yml (7 hunks)
  • docker-compose.yml (1 hunks)
  • src/Appwrite/Platform/Modules/Functions/Workers/Builds.php (18 hunks)
  • tests/e2e/Services/Sites/SitesBase.php (2 hunks)
  • tests/extensions/Async/Eventually.php (2 hunks)
  • tests/extensions/Async/Exceptions/Critical.php (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • docker-compose.yml
🧰 Additional context used
🧬 Code Graph Analysis (4)
tests/extensions/Async/Exceptions/Critical.php (1)
src/Appwrite/Extend/Exception.php (1)
  • Exception (7-408)
tests/extensions/Async/Eventually.php (1)
tests/extensions/Async/Exceptions/Critical.php (1)
  • Critical (5-7)
src/Appwrite/Platform/Modules/Functions/Workers/Builds.php (1)
src/Appwrite/Event/Database.php (1)
  • getCollection (85-88)
tests/e2e/Services/Sites/SitesBase.php (1)
tests/extensions/Async/Exceptions/Critical.php (1)
  • Critical (5-7)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (20)
  • GitHub Check: E2E Service Test (Webhooks)
  • GitHub Check: E2E Service Test (Users)
  • GitHub Check: E2E Service Test (Messaging)
  • GitHub Check: E2E Service Test (VCS)
  • GitHub Check: E2E Service Test (Migrations)
  • GitHub Check: E2E Service Test (GraphQL)
  • GitHub Check: E2E Service Test (Sites)
  • GitHub Check: E2E Service Test (Databases)
  • GitHub Check: E2E Service Test (Teams)
  • GitHub Check: E2E Service Test (Locale)
  • GitHub Check: E2E Service Test (Tokens)
  • GitHub Check: E2E Service Test (Functions)
  • GitHub Check: E2E Service Test (FunctionsSchedule)
  • GitHub Check: E2E Service Test (Projects)
  • GitHub Check: E2E Service Test (Account)
  • GitHub Check: E2E Service Test (Dev Keys)
  • GitHub Check: Unit Test
  • GitHub Check: E2E Service Test (Site Screenshots)
  • GitHub Check: E2E General Test
  • GitHub Check: scan
🔇 Additional comments (7)
tests/extensions/Async/Exceptions/Critical.php (1)

5-7: Minimal purpose-built exception looks good

Simple, scoped exception type for fast-fail behavior in async tests. No issues.

tests/extensions/Async/Eventually.php (2)

5-5: Correctly imports Critical to enable early termination path

Import placement and namespace usage are correct.


27-29: Good: Critical exceptions now abort retries immediately

Catching Critical before the generic Exception ensures the intended fast-fail behavior.

tests/e2e/Services/Sites/SitesBase.php (3)

6-6: New Critical import is correct and scoped for tests

Matches usage in the eventual assertion loop and aligns with the new async helper.


53-55: Early surfacing of deployment failures improves signal

Failing fast with detailed body when status == "failed" will reduce flakiness and timeouts. Good addition.


68-68: Extended eventual timeout (300s) is appropriate for flaky site builds

Doubling the window should reduce false negatives under load.

src/Appwrite/Platform/Modules/Functions/Workers/Builds.php (1)

120-121: Instrumentation additions improve debuggability without altering behavior

The added logs are well-placed across lifecycle phases (processing/building/runtime/logs/adapter/screenshot/activation/finalization) and will greatly aid test diagnostics.

Also applies to: 214-215, 285-286, 369-370, 412-413, 483-484, 531-532, 536-537, 681-682, 731-734, 822-825, 832-833, 889-890, 900-901, 905-909, 1082-1083, 1094-1095, 1188-1189, 1248-1249, 1266-1267, 1280-1281

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
tests/e2e/Services/Sites/SitesCustomServerTest.php (2)

408-411: Good switch to eventual consistency; remove by-ref capture and bound the wait

Polling for adapter detection is the right call. Capturing $site by reference isn’t needed (not used after the block) and can be dropped. Also consider an explicit timeout to cap worst-case retries.

Apply this diff:

-        $this->assertEventually(function () use ($siteId, &$site) {
-            $site = $this->getSite($siteId);
-            $this->assertEquals('ssr', $site['body']['adapter']);
-        });
+        $this->assertEventually(function () use ($siteId) {
+            $site = $this->getSite($siteId);
+            $this->assertEquals('ssr', $site['body']['adapter']);
+        }, 120000, 500);

421-421: Retries added—confirm idempotent cleanup across attempts

Adding #[Retry(count: 3)] is reasonable. Please verify this test leaves no residual sites/domains when a prior attempt fails mid-run, so that retries won’t encounter conflicts or leak resources.

Optional: You may further reduce flakiness in this test by polling for adapter detection (like the SSR test) rather than relying on a single read after deployment. If you want, I can draft that change.

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

💡 Knowledge Base configuration:

  • MCP integration is disabled by default for public repositories
  • Jira integration is disabled by default for public repositories
  • Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between 746a76d and aed97c5.

📒 Files selected for processing (1)
  • tests/e2e/Services/Sites/SitesCustomServerTest.php (3 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (1)
tests/e2e/Services/Sites/SitesCustomServerTest.php (2)
tests/e2e/Services/Sites/SitesBase.php (1)
  • getSite (177-185)
tests/e2e/Services/Functions/FunctionsCustomServerTest.php (2)
  • Retry (569-608)
  • Retry (2111-2179)
🪛 GitHub Actions: Linter
tests/e2e/Services/Sites/SitesCustomServerTest.php

[error] 1-1: Pint lint failed with 1 PSR-12 issue: no_whitespace_in_blank_lines in tests/e2e/Services/Sites/SitesCustomServerTest.php. Run 'vendor/bin/pint' to fix.

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (20)
  • GitHub Check: E2E Service Test (Tokens)
  • GitHub Check: E2E Service Test (Webhooks)
  • GitHub Check: E2E Service Test (Messaging)
  • GitHub Check: E2E Service Test (Teams)
  • GitHub Check: E2E Service Test (Realtime)
  • GitHub Check: E2E Service Test (Health)
  • GitHub Check: E2E Service Test (GraphQL)
  • GitHub Check: E2E Service Test (Projects)
  • GitHub Check: E2E Service Test (FunctionsSchedule)
  • GitHub Check: E2E Service Test (Locale)
  • GitHub Check: E2E Service Test (Databases)
  • GitHub Check: E2E Service Test (Sites)
  • GitHub Check: E2E Service Test (Account)
  • GitHub Check: E2E Service Test (Console)
  • GitHub Check: E2E Service Test (Avatars)
  • GitHub Check: E2E Service Test (Dev Keys)
  • GitHub Check: E2E Service Test (Site Screenshots)
  • GitHub Check: E2E General Test
  • GitHub Check: Unit Test
  • GitHub Check: scan

@Meldiron Meldiron merged commit 6c46039 into 1.7.x Aug 18, 2025
40 of 41 checks passed
@stnguyen90 stnguyen90 deleted the fix-cicd-tests-sites branch October 1, 2025 23:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants