Skip to content

Feat: Lazy-load relationships#9669

Merged
abnegate merged 42 commits into1.8.xfrom
lazy-load-relationships
Jun 16, 2025
Merged

Feat: Lazy-load relationships#9669
abnegate merged 42 commits into1.8.xfrom
lazy-load-relationships

Conversation

@ItzNotABug
Copy link
Copy Markdown
Contributor

@ItzNotABug ItzNotABug commented Apr 19, 2025

What does this PR do?

Skips fetching related documents unless explicitly requested via select queries.

  1. Default behavior

    GET

    /v1/databases/theatre/collections/movies/documents/avengers
    

    Response

    {
        "title": "Avengers",
        "$id": "avengers",
        "$permissions": [],
        "$databaseId": "theatre",
        "$collectionId": "movies",
        "$createdAt": "2025-04-19T11:59:18.339+00:00",
        "$updatedAt": "2025-04-19T11:59:18.339+00:00"
    }
  2. With nested relationship selects

    GET

    /v1/databases/theatre/collections/movies/documents/avengers
    

    PARAMS

    queries[0]: {"method":"select","values":["ratings.score"]}
    queries[1]: {"method":"select","values":["actors.name","actors.role"]}
    

    Response

    {
        "title": "Avengers",
        "actors": [
            {
                "name": "Chris Evans",
                "role": "Captain America",
                "$databaseId": "theatre",
                "$collectionId": "actors"
            },
            ...
        ],
        "ratings": [
            {
                "score": 9,
                "$databaseId": "theatre",
                "$collectionId": "ratings"
            }
        ],
        "$databaseId": "theatre",
        "$collectionId": "movies"
    }

Test Plan

Manual.

Related PRs and Issues

Checklist

  • Have you read the Contributing Guidelines on issues?
  • If the PR includes a change to an API's metadata (desc, label, params, etc.), does it also include updated API specs and example docs?

Summary by CodeRabbit

  • New Features

    • Improved handling of related documents in database queries, allowing more precise control over which related fields are included in results.
    • Added project-specific database and route context to request filters for enhanced query processing.
    • Introduced version 1.8.0 request and response filters to maintain compatibility with updated API formats.
    • Added support to preserve legacy behavior of including related documents by default in database queries for version 1.8 clients.
  • Bug Fixes

    • Enhanced backward compatibility for legacy clients to ensure related documents are returned by default when expected.
  • Tests

    • Updated and refined end-to-end tests to use explicit field selections and ensure correct behavior for relationship and attribute rename scenarios.
    • Added wait periods in bulk update tests to improve reliability on slower systems.
  • Chores

    • Updated GitHub Actions workflow to support testing against the 1.6.x database response format.

@ItzNotABug ItzNotABug self-assigned this Apr 19, 2025
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Apr 19, 2025

Walkthrough

The updates introduce enhanced handling of selection queries in database document endpoints, ensuring explicit control over relationship traversal and attribute inclusion. A new request filter for version 1.8.0 is added to maintain legacy behavior with related documents. The request filter base class is extended to support project database and route context. End-to-end tests are refined to use explicit selection queries, and test stability is improved with added waits. Minor CI workflow cleanup is applied.

Changes

File(s) Change Summary
app/controllers/api/databases.php Added conditional logic for relationship traversal and refined attribute selection in document endpoints.
app/controllers/general.php Added support for version 1.8.0 filters by injecting project database and route into request and response filters.
src/Appwrite/Utopia/Request/Filter.php Added constructor and methods to manage project database and route; introduced parameter retrieval helper.
src/Appwrite/Utopia/Request/Filters/V19.php Added return type declaration for convertQueryAttribute method.
src/Appwrite/Utopia/Request/Filters/V20.php Added new filter class to handle version 1.8.0 request format, preserving legacy related document selection behavior.
src/Appwrite/Utopia/Response/Filters/V20.php Added new filter class stub for version 1.8.0 response format conversion (no logic implemented yet).
tests/e2e/Services/Databases/DatabasesBase.php Updated tests to use explicit field selection in queries and added queries parameters to relevant requests.
tests/e2e/Services/Databases/DatabasesCustomServerTest.php Modified tests to include queries for renamed relationship attributes and added sleeps for cache purging.
.github/workflows/tests.yml Removed trailing whitespace after environment variable block in E2E Shared Mode Service Test job.

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant API_Controller
    participant RequestFilterV20
    participant ProjectDB
    participant Database

    Client->>API_Controller: GET /databases/{dbId}/collections/{colId}/documents
    API_Controller->>RequestFilterV20: Apply filter (with ProjectDB, Route)
    RequestFilterV20->>ProjectDB: Fetch related collection keys (if needed)
    RequestFilterV20->>API_Controller: Return modified queries with legacy related keys
    API_Controller->>Database: find/getDocument (with/without skipRelationships)
    Database-->>API_Controller: Documents
    API_Controller-->>Client: Response (with selected fields and relationships)
Loading

Poem

🐇 Hopping through queries, so clever and bright,
Selecting just fields that are needed just right.
Relationships fetched only when we say,
Filters keep order and tests lead the way.
Old versions remembered, new ones take flight—
In code’s cozy warren, all bunnies delight!
✨🌿


📜 Recent review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between f09042d and 44a5a3e.

📒 Files selected for processing (2)
  • app/controllers/api/databases.php (3 hunks)
  • tests/e2e/Services/Databases/DatabasesCustomServerTest.php (8 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
  • tests/e2e/Services/Databases/DatabasesCustomServerTest.php
  • app/controllers/api/databases.php
⏰ Context from checks skipped due to timeout of 90000ms (19)
  • GitHub Check: E2E Service Test (Storage)
  • GitHub Check: E2E Service Test (Tokens)
  • GitHub Check: E2E Service Test (Webhooks)
  • GitHub Check: E2E Service Test (Databases)
  • GitHub Check: E2E Service Test (Teams)
  • GitHub Check: E2E Service Test (Sites)
  • GitHub Check: E2E Service Test (Proxy)
  • GitHub Check: E2E Service Test (Health)
  • GitHub Check: E2E Service Test (Realtime)
  • GitHub Check: E2E Service Test (Console)
  • GitHub Check: E2E Service Test (Account)
  • GitHub Check: E2E Service Test (Functions)
  • GitHub Check: E2E Service Test (Locale)
  • GitHub Check: E2E Service Test (FunctionsSchedule)
  • GitHub Check: E2E Service Test (Avatars)
  • GitHub Check: Unit Test
  • GitHub Check: E2E General Test
  • GitHub Check: Setup & Build Appwrite Image
  • GitHub Check: scan
✨ Finishing Touches
  • 📝 Generate Docstrings

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Explain this complex logic.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai explain this code block.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and explain its main purpose.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 19, 2025

Security Scan Results for PR

Docker Image Scan Results

Package Version Vulnerability Severity
binutils 2.42-r0 CVE-2025-0840 HIGH
libexpat 2.6.4-r0 CVE-2024-8176 HIGH
libxml2 2.12.7-r0 CVE-2024-56171 HIGH
libxml2 2.12.7-r0 CVE-2025-24928 HIGH
libxml2 2.12.7-r0 CVE-2025-27113 HIGH
libxml2 2.12.7-r0 CVE-2025-32414 HIGH
libxml2 2.12.7-r0 CVE-2025-32415 HIGH
pyc 3.12.9-r0 CVE-2024-12718 HIGH
pyc 3.12.9-r0 CVE-2025-4138 HIGH
pyc 3.12.9-r0 CVE-2025-4517 HIGH
python3 3.12.9-r0 CVE-2024-12718 HIGH
python3 3.12.9-r0 CVE-2025-4138 HIGH
python3 3.12.9-r0 CVE-2025-4517 HIGH
python3-pyc 3.12.9-r0 CVE-2024-12718 HIGH
python3-pyc 3.12.9-r0 CVE-2025-4138 HIGH
python3-pyc 3.12.9-r0 CVE-2025-4517 HIGH
python3-pycache-pyc0 3.12.9-r0 CVE-2024-12718 HIGH
python3-pycache-pyc0 3.12.9-r0 CVE-2025-4138 HIGH
python3-pycache-pyc0 3.12.9-r0 CVE-2025-4517 HIGH
sqlite-libs 3.45.3-r1 CVE-2025-29087 HIGH
xz 5.6.2-r0 CVE-2025-31115 HIGH
xz-libs 5.6.2-r0 CVE-2025-31115 HIGH
golang.org/x/crypto v0.31.0 CVE-2025-22869 HIGH

Source Code Scan Results

🎉 No vulnerabilities found!

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 19, 2025

✨ Benchmark results

  • Requests per second: 903
  • Requests with 200 status code: 162,623
  • P99 latency: 0.208522357

⚡ Benchmark Comparison

Metric This PR Latest version
RPS 903 1,132
200 162,623 203,761
P99 0.208522357 0.166849626

@ItzNotABug ItzNotABug marked this pull request as ready for review May 3, 2025 05:04
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (3)
src/Appwrite/Utopia/Request/Filter.php (2)

10-12: Consider giving typed properties an explicit default value

Although the constructor normally initialises $route and $dbForProject, it is still possible for an extending class to forget calling parent::__construct(), leaving the properties un-initialised and triggering a Typed property must not be accessed before initialization error. Adding a = null default makes the class more defensive at virtually zero cost.

-    private ?Route $route;
-    private ?Database $dbForProject;
+    private ?Route $route = null;
+    private ?Database $dbForProject = null;

47-56: Broaden the caught throwable & remove redundant null-coalescing

Route::getParamValue() may throw any Throwable, not only Exception (e.g. Error). Catching \Throwable is safer.
Also, the null-coalescing operator already covers the route is null case, so the surrounding try/​catch only needs to protect against the method throwing. This tiny refactor keeps the intent but avoids double work:

-        try {
-            $value = $this->route?->getParamValue($key) ?? $default;
-        } catch (\Exception $e) {
-            $value = $default;
-        }
+        try {
+            return $this->route?->getParamValue($key) ?? $default;
+        } catch (\Throwable) {
+            return $default;
+        }
-
-        return $value;
src/Appwrite/Utopia/Request/Filters/V19.php (1)

30-36: Redundant prefix check

$isDatabaseModel is always true when $model equals either databases.listDocuments or databases.getDocument. Removing the extra str_starts_with simplifies the guard clause:

-        $isDatabaseModel = \str_starts_with($model, 'databases.');
-        $isTargetOperation = \in_array($model, ['databases.listDocuments', 'databases.getDocument']);
-
-        if (! $isDatabaseModel || ! $isTargetOperation) {
+        if (!\in_array($model, ['databases.listDocuments', 'databases.getDocument'], true)) {
             return $content;
         }
📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between f76aff1 and 04539bf.

⛔ Files ignored due to path filters (1)
  • composer.lock is excluded by !**/*.lock
📒 Files selected for processing (5)
  • app/controllers/api/databases.php (2 hunks)
  • app/controllers/general.php (1 hunks)
  • composer.json (3 hunks)
  • src/Appwrite/Utopia/Request/Filter.php (2 hunks)
  • src/Appwrite/Utopia/Request/Filters/V19.php (1 hunks)
🔇 Additional comments (7)
app/controllers/general.php (1)

833-834:

Details

✅ Verification successful

Dependency injection improves request filter capabilities for handling relationship queries.

The change injects the project database instance and route object into the RequestV19 filter, enabling it to access collection metadata and modify query selections to control relationship loading behavior. This aligns with the PR objective of implementing lazy loading for relationships, ensuring backward compatibility for versions before 1.7.0.


🏁 Script executed:

#!/bin/bash
# Verify how the V19 filter class uses these dependencies

# Look for the V19 filter class implementation
echo "Searching for V19 filter implementation..."
rg --type php -A 15 "class V19" src/

Length of output: 3774


🏁 Script executed:

#!/bin/bash
# Display the full V19 request filter implementation to verify constructor and manageSelectQueries usage
rg --type php -A200 "class V19 extends Filter" src/Appwrite/Utopia/Request/Filters/V19.php

Length of output: 3500


🏁 Script executed:

#!/bin/bash
# Verify Filter base class constructor signature and dependency handling
rg --type php -A200 "class Filter" src/Appwrite/Utopia/Request/Filters/Filter.php

Length of output: 170


🏁 Script executed:

#!/bin/bash
# Search for base Filter class definition across the src directory
rg --type php -n "class Filter" src/

Length of output: 165


🏁 Script executed:

#!/bin/bash
# Show the Request Filter base class implementation
sed -n '1,200p' src/Appwrite/Utopia/Request/Filter.php

# Check for constructor definition in Filter
rg --type php -n "__construct" src/Appwrite/Utopia/Request/Filter.php

Length of output: 1457


Dependency injection correctly enhances RequestV19 filter

Injecting the project’s Database instance and the Route into the RequestV19 filter aligns perfectly with how the base Filter constructor is defined (__construct(Database $dbForProject = null, Route $route = null)). This grants the filter access to:

  • getDbForProject() for resolving collection metadata
  • getParamValue() for extracting databaseId and collectionId from the route

which it uses in manageSelectQueries() to preserve 1.6.x behavior (including related documents by default) while maintaining lazy-loading support for 1.7.x+. This change meets the PR’s goal of backward-compatible relationship loading.

composer.json (3)

52-52: Cache dependency updated.

The utopia-php/cache package has been updated to version 0.13.*, which likely includes improvements or features needed to support the lazy-loading relationship functionality.


97-102: Added custom repository source for the database package.

This configuration directs Composer to fetch the utopia-php/database package directly from the GitHub repository instead of Packagist, which is necessary to use the development branch specified above.


56-56:

Details

❓ Verification inconclusive

Using a development branch for database package.

The dependency on utopia-php/database has been changed from a stable version to a development branch specifically created for handling wildcards in queries. This branch is required for implementing the relationship lazy-loading functionality described in the PR.


🏁 Script executed:

#!/bin/bash
# Check if there's a reference to a related PR for the database package

# Look for references to the related database PR in the current repo
echo "Searching for references to the utopia-php/database PR..."
rg -i "utopia.*database.*PR" --type md

Length of output: 165


Please verify the “dev-manage-wildcards” alias for utopia-php/database

It looks like you’ve switched utopia-php/database to a development branch alias:

• composer.json: line 56

"utopia-php/database": "dev-manage-wildcards as 0.66.0",

Before we merge, confirm that:

  • The dev-manage-wildcards branch exists and is up-to-date in the utopia-php/database repo.
  • That branch has been published (or is accessible) via Packagist under the correct version alias.
  • Any consumers of this package in the repo will still resolve correctly to the expected API.
app/controllers/api/databases.php (2)

3513-3522: Well-implemented optimization of database queries!

This change implements the lazy-loading of relationships by checking for selection queries before deciding how to fetch documents. When no selections are present, the code skips loading relationships, which should significantly improve performance for documents with many relationships.


3673-3684: Consistent implementation of lazy-loading in the getDocument endpoint.

Similar to the change in listDocuments, this implements lazy-loading for single document retrieval. The conditional relationship loading is well-structured and mirrors the implementation in the list endpoint, ensuring consistent behavior throughout the API.

src/Appwrite/Utopia/Request/Filters/V19.php (1)

43-45: Verify key used in Query::groupByType() result

You access the grouped array via ['selections'].
In Utopia Database, the key is usually derived from the constant (Query::TYPE_SELECT), so it may be 'selects'. A mismatch yields an empty $selections, making the wildcard-detection loop ineffective when explicit selects are supplied.

Please double-check the constant name or update the index:

$groups = Query::groupByType($parsed);
$selections = $groups[Query::TYPE_SELECT.'s'] ?? [];

(Adjust if the library already uses the plural form you expect.)

@ItzNotABug ItzNotABug changed the base branch from 1.7.x to 1.8.x June 9, 2025 08:12
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (4)
.github/workflows/tests.yml (1)

362-362: Add newline at end of file
YAML files should end with a newline to satisfy POSIX standards and avoid the new-line-at-end-of-file lint error.

Apply this diff to add the missing newline:

--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -362 +363
+
🧰 Tools
🪛 YAMLlint (1.37.1)

[error] 362-362: no new line character at the end of file

(new-line-at-end-of-file)

src/Appwrite/Utopia/Request/Filters/V20.php (2)

35-41: Harden handling of the queries payload

manageSelectQueries() assumes that queries is either absent or already an array of query-strings.
A single scalar (e.g. "limit(50)") will make Query::parseQueries() throw, falling into the generic catch and silently disabling the filter.

+        // Normalise – always work with an array
+        if (isset($content['queries']) && !\is_array($content['queries'])) {
+            $content['queries'] = [$content['queries']];
+        }

61-76: Consider caching relationship keys for listDocuments

getRelatedCollectionKeys() is invoked on every wildcard listDocuments request and performs two DB reads.
If the same collection is queried frequently, memoising the keys (e.g. static cache keyed by $collectionId) would cut DB round-trips and latency without affecting correctness.

src/Appwrite/Utopia/Response/Filters/V20.php (1)

7-13: Implement or remove placeholder filter

V20 returns the payload untouched.
If no transformation is planned, drop the class to avoid maintenance overhead; otherwise add a TODO explaining the forthcoming mapping logic.

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 93472ac and f09042d.

📒 Files selected for processing (7)
  • .github/workflows/tests.yml (3 hunks)
  • app/controllers/general.php (3 hunks)
  • src/Appwrite/Utopia/Request/Filters/V19.php (1 hunks)
  • src/Appwrite/Utopia/Request/Filters/V20.php (1 hunks)
  • src/Appwrite/Utopia/Response/Filters/V20.php (1 hunks)
  • tests/e2e/Services/Databases/DatabasesBase.php (7 hunks)
  • tests/e2e/Services/Databases/DatabasesCustomServerTest.php (8 hunks)
✅ Files skipped from review due to trivial changes (1)
  • src/Appwrite/Utopia/Request/Filters/V19.php
🚧 Files skipped from review as they are similar to previous changes (3)
  • app/controllers/general.php
  • tests/e2e/Services/Databases/DatabasesBase.php
  • tests/e2e/Services/Databases/DatabasesCustomServerTest.php
🧰 Additional context used
🪛 YAMLlint (1.37.1)
.github/workflows/tests.yml

[error] 276-276: trailing spaces

(trailing-spaces)


[error] 362-362: no new line character at the end of file

(new-line-at-end-of-file)

⏰ Context from checks skipped due to timeout of 90000ms (3)
  • GitHub Check: Setup & Build Appwrite Image
  • GitHub Check: Setup & Build Appwrite Image
  • GitHub Check: scan
🔇 Additional comments (2)
.github/workflows/tests.yml (2)

121-125: Clean up trailing whitespace
Removed unnecessary spaces after the sleep 10 command in the E2E General Test job to eliminate diff noise and satisfy YAML lint rules.


273-277: Clean up trailing whitespace
Removed extraneous spaces at the end of the conditional block in the Shared Mode Service Test job to improve readability and avoid lint warnings.

🧰 Tools
🪛 YAMLlint (1.37.1)

[error] 276-276: trailing spaces

(trailing-spaces)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants