Skip to content

fix: fix set payload in batch update points, fix operation conversion…#977

Merged
joein merged 5 commits into
devfrom
fix-update-operations
May 15, 2025
Merged

fix: fix set payload in batch update points, fix operation conversion…#977
joein merged 5 commits into
devfrom
fix-update-operations

Conversation

@joein
Copy link
Copy Markdown
Member

@joein joein commented Apr 28, 2025

#974

  • fix key conversion for SetPayload and OverwritePayload
  • fix vector normalization during cosine vector updates

@joein joein requested a review from coszio April 28, 2025 13:24
@netlify
Copy link
Copy Markdown

netlify Bot commented Apr 28, 2025

Deploy Preview for poetic-froyo-8baba7 ready!

Name Link
🔨 Latest commit 21d31f2
🔍 Latest deploy log https://app.netlify.com/projects/poetic-froyo-8baba7/deploys/6825ff180657d100084afac9
😎 Deploy Preview https://deploy-preview-977--poetic-froyo-8baba7.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Apr 28, 2025

📝 Walkthrough

Walkthrough

The changes introduce support for an optional key attribute in payload update operations across both gRPC and REST representations within the conversion logic. This is achieved by updating the conversion methods in qdrant_client/conversions/conversion.py to extract and assign the key field for set_payload and overwrite_payload operations during conversions in both directions. In qdrant_client/local/local_collection.py, the _add_point and _update_named_vectors methods now enforce vector data types as float32, check for NaNs, and normalize vectors when the COSINE distance metric is used. The batch_update_points method was updated to pass the key parameter to the set_payload function, enabling partial payload updates. The test suite in tests/congruence_tests/test_payload.py has been expanded and refactored to parameterize tests over the communication protocol, unify collection initialization, broaden exception handling, and add comprehensive tests for batch update operations covering various update operation models, verifying congruence between local and remote clients.

Note

⚡️ AI Code Reviews for VS Code, Cursor, Windsurf

CodeRabbit now has a plugin for VS Code, Cursor and Windsurf. This brings AI code reviews directly in the code editor. Each commit is reviewed immediately, finding bugs before the PR is raised. Seamless context handoff to your AI code agent ensures that you can easily incorporate review feedback.
Learn more here.


Note

⚡️ Faster reviews with caching

CodeRabbit now supports caching for code and dependencies, helping speed up reviews. This means quicker feedback, reduced wait times, and a smoother review experience overall. Cached data is encrypted and stored securely. This feature will be automatically enabled for all accounts on May 16th. To opt out, configure Review - Disable Cache at either the organization or repository level. If you prefer to disable all data retention across your organization, simply turn off the Data Retention setting under your Organization Settings.
Enjoy the performance boost—your workflow just got faster.


📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 1c003fa and 21d31f2.

📒 Files selected for processing (1)
  • tests/congruence_tests/test_payload.py (9 hunks)
🧰 Additional context used
🧠 Learnings (1)
tests/congruence_tests/test_payload.py (1)
Learnt from: joein
PR: qdrant/qdrant-client#977
File: tests/congruence_tests/test_payload.py:645-651
Timestamp: 2025-05-14T20:18:46.248Z
Learning: In the Qdrant client, there is no separate `models.OverwritePayload` class. The `models.OverwritePayloadOperation` class is designed to accept a `models.SetPayload` object as its `overwrite_payload` parameter.
⏰ Context from checks skipped due to timeout of 90000ms (8)
  • GitHub Check: Redirect rules - poetic-froyo-8baba7
  • GitHub Check: Header rules - poetic-froyo-8baba7
  • GitHub Check: Pages changed - poetic-froyo-8baba7
  • GitHub Check: Python 3.9.x on ubuntu-latest test
  • GitHub Check: Python 3.13.x on ubuntu-latest test
  • GitHub Check: Python 3.11.x on ubuntu-latest test
  • GitHub Check: Python 3.12.x on ubuntu-latest test
  • GitHub Check: Python 3.10.x on ubuntu-latest test
🔇 Additional comments (8)
tests/congruence_tests/test_payload.py (8)

34-39: Improved test coverage across communication protocols

Good refactoring to parameterize tests over prefer_grpc. This ensures both gRPC and HTTP clients are tested with the same logic, providing better verification that the client behavior is consistent regardless of the communication protocol.


183-184: Enhanced maintainability with standardized collection initialization

Good refactoring to use the initialize_fixture_collection helper across all test functions. This reduces code duplication and ensures consistent collection setup across tests.

Also applies to: 267-268


310-380: Comprehensive test cases for payload updates with nested paths

Excellent addition of test cases that cover a wide range of nested path scenarios:

  • Updating elements in nested arrays
  • Handling non-existent array elements
  • Adding fields to deeply nested dictionaries
  • Replacing arrays with dictionaries
  • Testing edge cases with complex nested structures

This significantly improves test coverage for the key functionality being fixed in this PR.


415-416: More robust exception handling for cross-protocol tests

Good improvement to catch both UnexpectedResponse and grpc.RpcError exceptions. This ensures that tests handle errors properly regardless of whether HTTP or gRPC protocol is used.

Also applies to: 433-434


450-485: New test for batch upsert operations

Great addition of tests for the UpsertOperation with both PointsBatch and PointsList. This helps verify that the batch update API works correctly for upsert operations.


507-567: Comprehensive tests for payload deletion operations

Good test coverage for both DeletePayloadOperation and ClearPayloadOperation, testing with both direct point IDs and filters. This verifies that payload fields can be deleted or cleared correctly through batch operations.


569-618: Thorough testing of vector update and deletion operations

Good test coverage for both UpdateVectorsOperation and DeleteVectorsOperation, verifying that vectors can be updated or deleted through batch operations with both direct point IDs and filters.


620-670: Verification of payload operations with and without key parameter

Excellent tests for SetPayloadOperation and OverwritePayloadOperation, including the important case with a key parameter (line 647) for nested payload updates. This directly validates the fix mentioned in the PR objectives for key conversion in payload operations.

✨ Finishing Touches
  • 📝 Generate Docstrings

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Explain this complex logic.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai explain this code block.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and explain its main purpose.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
qdrant_client/local/local_collection.py (1)

2355-2361: Cast multivector updates to float32 to avoid implicit up-casting

Same dtype concern as above – after the first float64 insert the entire list for that vector name will be promoted.

-                vector_np = np.array(vector)
+                vector_np = np.asarray(vector, dtype=np.float32)

Optionally extract the (now duplicated) “normalise-if-cosine” logic into a small helper to DRY the three call-sites in this class.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between acb3312 and cf9eae7.

📒 Files selected for processing (3)
  • qdrant_client/conversions/conversion.py (4 hunks)
  • qdrant_client/local/local_collection.py (2 hunks)
  • tests/congruence_tests/test_payload.py (11 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (1)
qdrant_client/conversions/conversion.py (5)
qdrant_client/local/qdrant_local.py (1)
  • set_payload (826-836)
qdrant_client/local/local_collection.py (1)
  • set_payload (2451-2475)
qdrant_client/qdrant_client.py (1)
  • set_payload (1855-1949)
qdrant_client/qdrant_remote.py (1)
  • set_payload (2245-2295)
qdrant_client/client_base.py (1)
  • set_payload (303-311)
⏰ Context from checks skipped due to timeout of 90000ms (5)
  • GitHub Check: Python 3.13.x on ubuntu-latest test
  • GitHub Check: Python 3.12.x on ubuntu-latest test
  • GitHub Check: Python 3.11.x on ubuntu-latest test
  • GitHub Check: Python 3.10.x on ubuntu-latest test
  • GitHub Check: Python 3.9.x on ubuntu-latest test
🔇 Additional comments (5)
qdrant_client/conversions/conversion.py (4)

1912-1912: Good addition of key parameter support in GrpcToRest conversion

The change correctly adds support for converting the optional key field from gRPC to REST for the set_payload operation, ensuring nested field path specifications are preserved through the conversion layer.

The implementation properly checks if the field exists using HasField("key") before attempting to access it, which is a good practice for dealing with optional protobuf fields.


1938-1938: Good addition of key parameter support in GrpcToRest conversion for overwrite_payload

The change correctly adds support for converting the optional key field from gRPC to REST for the overwrite_payload operation, maintaining consistency with the set_payload operation implementation.

This ensures that nested field path specifications are properly preserved when converting between representations.


4216-4216: Good addition of key parameter support in RestToGrpc conversion

The change properly adds support for converting the optional key field from REST to gRPC for the set_payload operation, ensuring bidirectional compatibility with the GrpcToRest conversion added earlier in the file.


4240-4240: Good addition of key parameter support in RestToGrpc conversion for overwrite_payload

The change properly adds support for converting the optional key field from REST to gRPC for the overwrite_payload operation, maintaining consistency with the set_payload operation implementation.

These changes complete the bidirectional conversion support for the key parameter, ensuring that nested field path specifications work correctly across REST and gRPC interfaces.

qdrant_client/local/local_collection.py (1)

2541-2544: Handle None selector & keep behaviour symmetric with REST/gRPC

points_selector = update_op.set_payload.points or update_op.set_payload.filter
will be None if both fields are absent (allowed by the schema).
self.set_payload() then receives selector=None, but its signature expects one of the supported selector types and will raise an opaque AttributeError.

Consider an explicit check:

                 points_selector = update_op.set_payload.points or update_op.set_payload.filter
-                self.set_payload(
-                    update_op.set_payload.payload, points_selector, update_op.set_payload.key
-                )
+                if points_selector is None:
+                    raise ValueError("SetPayloadOperation must provide either `points` or `filter`")
+                self.set_payload(
+                    update_op.set_payload.payload, points_selector, update_op.set_payload.key
+                )

This yields an early, user-friendly error message and keeps behaviour aligned between local & remote clients.

Comment thread qdrant_client/local/local_collection.py Outdated
Comment on lines +645 to +651
new_payload = {"text_data": {"some_key": "overwritten_value"}}
overwrite_payload_op_points_no_key = models.OverwritePayloadOperation(
overwrite_payload=models.SetPayload(points=[1], payload=new_payload)
)

local_client.batch_update_points(
collection_name=COLLECTION_NAME, update_operations=[overwrite_payload_op_points_no_key]
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Apr 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

OverwritePayloadOperation wraps OverwritePayload, not SetPayload

Using models.SetPayload here still works only because both dataclasses share field names, but it breaks type-level guarantees and may stop compiling once the models diverge.

-    overwrite_payload_op_points_no_key = models.OverwritePayloadOperation(
-        overwrite_payload=models.SetPayload(points=[1], payload=new_payload)
+    overwrite_payload_op_points_no_key = models.OverwritePayloadOperation(
+        overwrite_payload=models.OverwritePayload(points=[1], payload=new_payload)
     )
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
new_payload = {"text_data": {"some_key": "overwritten_value"}}
overwrite_payload_op_points_no_key = models.OverwritePayloadOperation(
overwrite_payload=models.SetPayload(points=[1], payload=new_payload)
)
local_client.batch_update_points(
collection_name=COLLECTION_NAME, update_operations=[overwrite_payload_op_points_no_key]
new_payload = {"text_data": {"some_key": "overwritten_value"}}
overwrite_payload_op_points_no_key = models.OverwritePayloadOperation(
overwrite_payload=models.OverwritePayload(points=[1], payload=new_payload)
)
local_client.batch_update_points(
collection_name=COLLECTION_NAME, update_operations=[overwrite_payload_op_points_no_key]
)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is no models.OverwritePayload
models.OverwritePayloadOperation accepts models.SetPayload

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash

# 1. Look for OverwritePayload class definition
echo "Searching for OverwritePayload class definition..."
rg "class OverwritePayload" --type py

# 2. Look for OverwritePayloadOperation class definition
echo -e "\nSearching for OverwritePayloadOperation class definition..."
rg "class OverwritePayloadOperation" --type py -A 15

# 3. Check imports and usages to see the structure relationship
echo -e "\nChecking imports and usages in http/models..."
fd "models\.py" --type f | xargs cat | grep -A 50 "class OverwritePayloadOperation"

Length of output: 3147


@joein You're absolutely right. After checking the codebase, I can confirm:

  1. There is no models.OverwritePayload class defined anywhere in the code
  2. The OverwritePayloadOperation class is defined to take a SetPayload object:
class OverwritePayloadOperation(BaseModel, extra="forbid"):
    overwrite_payload: "SetPayload" = Field(..., description="")

Your original code using models.SetPayload inside OverwritePayloadOperation is correct. Sorry for the confusion in my earlier review comment.


🧬 Code Graph Analysis Results

Relevant snippets for understanding tests/congruence_tests/test_payload.py and the user comment about models.OverwritePayload and models.OverwritePayloadOperation:


From tests/congruence_tests/test_payload.py (lines 350-400 approx)

  • The test test_set_or_overwrite_payload_operation uses models.SetPayloadOperation and models.OverwritePayloadOperation.
  • It shows usage of models.SetPayload inside SetPayloadOperation and OverwritePayloadOperation.
  • Specifically, OverwritePayloadOperation is constructed with overwrite_payload=models.SetPayload(...).
  • This indicates that OverwritePayloadOperation expects a SetPayload instance, not a separate OverwritePayload model.

From qdrant_client/http/models/models.py

  • At lines 1628-1629:
class OverwritePayloadOperation(BaseModel, extra="forbid"):
    overwrite_payload: "SetPayload" = Field(..., description="")
  • This shows that OverwritePayloadOperation contains a field overwrite_payload of type SetPayload.

  • There is no separate OverwritePayload model; the overwrite operation uses SetPayload as its payload container.

  • At lines 2462-2481:

class SetPayload(BaseModel, extra="forbid"):
    """
    This data structure is used in API interface and applied across multiple shards
    """

    payload: "Payload" = Field(..., description="This data structure is used in API interface and applied across multiple shards")
    points: Optional[List["ExtendedPointId"]] = Field(default=None, description="Assigns payload to each point in this list")
    filter: Optional["Filter"] = Field(default=None, description="Assigns payload to each point that satisfy this filter condition")
    shard_key: Optional["ShardKeySelector"] = Field(default=None, description="This data structure is used in API interface and applied across multiple shards")
    key: Optional[str] = Field(default=None, description="Assigns payload to each point that satisfy this path of property")
  • SetPayload is a model representing the payload to set, with optional points or filter selectors and an optional key.

From qdrant_client/qdrant_client.py (lines 2111-2152):

  • Method batch_update_points accepts a list of update_operations which can include SetPayloadOperation and OverwritePayloadOperation.
  • The client calls _client.batch_update_points with these operations.

From qdrant_client/local/local_collection.py (lines 2526-2565):

  • Method batch_update_points processes update operations:
    • For SetPayloadOperation, it calls set_payload with the contained set_payload field.
    • For OverwritePayloadOperation, it calls overwrite_payload with the contained overwrite_payload field.
  • Both set_payload and overwrite_payload expect a SetPayload instance.
  • This confirms that OverwritePayloadOperation uses SetPayload as its payload container.

From qdrant_client/qdrant_remote.py (lines 2297-2346):

  • Method overwrite_payload in remote client accepts payload: types.Payload and points: types.PointsSelector.
  • It constructs a models.SetPayload instance internally for the REST API call.
  • This matches the usage in the test where OverwritePayloadOperation contains a SetPayload.

Summary of relevance to user comment:

  • The user states: "there is no models.OverwritePayload; models.OverwritePayloadOperation accepts models.SetPayload".
  • The code confirms this:
    • There is no separate OverwritePayload model.
    • OverwritePayloadOperation has a field overwrite_payload of type SetPayload.
    • The test code uses models.SetPayload inside OverwritePayloadOperation.
  • This is consistent with the design in the models and client code.

Included relevant snippets:

1. tests/congruence_tests/test_payload.py (lines ~350-400)

@pytest.mark.parametrize("prefer_grpc", [True, False])
def test_set_or_overwrite_payload_operation(prefer_grpc):
    local_client: QdrantClient = init_local()
    initialize_fixture_collection(local_client)
    remote_client: QdrantClient = init_remote(prefer_grpc=prefer_grpc)
    initialize_fixture_collection(remote_client)
    num_vectors = 10
    upload(local_client, remote_client, num_vectors)

    new_payload = {"text_data": "new_value"}
    set_payload_op_points_no_key = models.SetPayloadOperation(
        set_payload=models.SetPayload(points=[1], payload=new_payload)
    )

    local_client.batch_update_points(
        collection_name=COLLECTION_NAME, update_operations=[set_payload_op_points_no_key]
    )
    remote_client.batch_update_points(
        collection_name=COLLECTION_NAME, update_operations=[set_payload_op_points_no_key]
    )
    compare_collections(local_client, remote_client, num_vectors)

    new_nested_payload = {"nested_data": "new_nested_value"}
    set_payload_op_filter_key = models.SetPayloadOperation(
        set_payload=models.SetPayload(
            filter=models.Filter(must=[models.HasIdCondition(has_id=[1])]),
            payload=new_nested_payload,
            key="text_data",
        )
    )

    local_client.batch_update_points(
        collection_name=COLLECTION_NAME, update_operations=[set_payload_op_filter_key]
    )
    remote_client.batch_update_points(
        collection_name=COLLECTION_NAME, update_operations=[set_payload_op_filter_key]
    )
    compare_collections(local_client, remote_client, num_vectors)

    new_payload = {"text_data": {"some_key": "overwritten_value"}}
    overwrite_payload_op_points_no_key = models.OverwritePayloadOperation(
        overwrite_payload=models.SetPayload(points=[1], payload=new_payload)
    )

    local_client.batch_update_points(
        collection_name=COLLECTION_NAME, update_operations=[overwrite_payload_op_points_no_key]
    )
    remote_client.batch_update_points(
        collection_name=COLLECTION_NAME, update_operations=[overwrite_payload_op_points_no_key]
    )
    compare_collections(local_client, remote_client, num_vectors)

2. qdrant_client/http/models/models.py (lines 1628-1630 and 2462-2481)

class OverwritePayloadOperation(BaseModel, extra="forbid"):
    overwrite_payload: "SetPayload" = Field(..., description="")

class SetPayload(BaseModel, extra="forbid"):
    """
    This data structure is used in API interface and applied across multiple shards
    """

    payload: "Payload" = Field(
        ..., description="This data structure is used in API interface and applied across multiple shards"
    )
    points: Optional[List["ExtendedPointId"]] = Field(
        default=None, description="Assigns payload to each point in this list"
    )
    filter: Optional["Filter"] = Field(
        default=None, description="Assigns payload to each point that satisfy this filter condition"
    )
    shard_key: Optional["ShardKeySelector"] = Field(
        default=None, description="This data structure is used in API interface and applied across multiple shards"
    )
    key: Optional[str] = Field(
        default=None, description="Assigns payload to each point that satisfy this path of property"
    )

3. qdrant_client/local/local_collection.py (lines 2526-2565)

def batch_update_points(
        self,
        update_operations: Sequence[types.UpdateOperation],
    ) -> None:
        for update_op in update_operations:
            if isinstance(update_op, models.UpsertOperation):
                if isinstance(update_op.upsert, models.PointsBatch):
                    self.upsert(update_op.upsert.batch)
                elif isinstance(update_op.upsert, models.PointsList):
                    self.upsert(update_op.upsert.points)
                else:
                    raise ValueError(f"Unsupported upsert type: {type(update_op.upsert)}")
            elif isinstance(update_op, models.DeleteOperation):
                self.delete(update_op.delete)
            elif isinstance(update_op, models.SetPayloadOperation):
                points_selector = update_op.set_payload.points or update_op.set_payload.filter
                self.set_payload(
                    update_op.set_payload.payload, points_selector, update_op.set_payload.key
                )
            elif isinstance(update_op, models.OverwritePayloadOperation):
                points_selector = (
                    update_op.overwrite_payload.points or update_op.overwrite_payload.filter
                )
                self.overwrite_payload(update_op.overwrite_payload.payload, points_selector)
            elif isinstance(update_op, models.DeletePayloadOperation):
                points_selector = (
                    update_op.delete_payload.points or update_op.delete_payload.filter
                )
                self.delete_payload(update_op.delete_payload.keys, points_selector)
            elif isinstance(update_op, models.ClearPayloadOperation):
                self.clear_payload(update_op.clear_payload)
            elif isinstance(update_op, models.UpdateVectorsOperation):
                self.update_vectors(update_op.update_vectors.points)
            elif isinstance(update_op, models.DeleteVectorsOperation):
                points_selector = (
                    update_op.delete_vectors.points or update_op.delete_vectors.filter
                )
                self.delete_vectors(update_op.delete_vectors.vector, points_selector)
            else:
                raise ValueError(f"Unsupported update operation: {type(update_op)}")

4. qdrant_client/qdrant_remote.py (lines 2297-2346)

def overwrite_payload(
        self,
        collection_name: str,
        payload: types.Payload,
        points: types.PointsSelector,
        wait: bool = True,
        ordering: Optional[types.WriteOrdering] = None,
        shard_key_selector: Optional[types.ShardKeySelector] = None,
        **kwargs: Any,
    ) -> types.UpdateResult:
        if self._prefer_grpc:
            points_selector, opt_shard_key_selector = self._try_argument_to_grpc_selector(points)
            shard_key_selector = shard_key_selector or opt_shard_key_selector

            if isinstance(ordering, models.WriteOrdering):
                ordering = RestToGrpc.convert_write_ordering(ordering)

            if isinstance(shard_key_selector, get_args_subscribed(models.ShardKeySelector)):
                shard_key_selector = RestToGrpc.convert_shard_key_selector(shard_key_selector)

            return GrpcToRest.convert_update_result(
                self.grpc_points.OverwritePayload(
                    grpc.SetPayloadPoints(
                        collection_name=collection_name,
                        wait=wait,
                        payload=RestToGrpc.convert_payload(payload),
                        points_selector=points_selector,
                        ordering=ordering,
                        shard_key_selector=shard_key_selector,
                    ),
                    timeout=self._timeout,
                ).result
            )
        else:
            _points, _filter = self._try_argument_to_rest_points_and_filter(points)
            result: Optional[types.UpdateResult] = (
                self.openapi_client.points_api.overwrite_payload(
                    collection_name=collection_name,
                    wait=wait,
                    ordering=ordering,
                    set_payload=models.SetPayload(
                        payload=payload,
                        points=_points,
                        filter=_filter,
                        shard_key=shard_key_selector,
                    ),
                ).result
            )
            assert result is not None, "Overwrite payload returned None"
            return result

These snippets collectively confirm that:

  • models.OverwritePayloadOperation contains a SetPayload instance named overwrite_payload.
  • There is no separate OverwritePayload model.
  • The test code uses models.SetPayload inside OverwritePayloadOperation.
  • The client code and local collection code expect and handle SetPayload for overwrite operations.

This directly addresses the user comment about the absence of models.OverwritePayload and the acceptance of models.SetPayload by models.OverwritePayloadOperation.


✏️ Learnings added
Learnt from: joein
PR: qdrant/qdrant-client#977
File: tests/congruence_tests/test_payload.py:645-651
Timestamp: 2025-05-14T20:18:46.248Z
Learning: In the Qdrant client, there is no separate `models.OverwritePayload` class. The `models.OverwritePayloadOperation` class is designed to accept a `models.SetPayload` object as its `overwrite_payload` parameter.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

Copy link
Copy Markdown
Contributor

@coszio coszio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to ask to add a few comments in the tests to identify sections of the test easier, they seem to have multiple steps of setup->assertion. Having a description of each should make it easier to maintain

Comment thread qdrant_client/local/local_collection.py Outdated
Comment thread tests/congruence_tests/test_payload.py Outdated
payload = {}
set_payload(payload, new_payload, key)
# payload = {}
# set_payload(payload, new_payload, key) # todo: uncomnment when https://github.com/qdrant/qdrant/issues/6449 is resolved
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

interesting, nice find

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@joein joein requested a review from coszio May 14, 2025 21:02
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between b0a50b6 and 1c003fa.

📒 Files selected for processing (1)
  • tests/congruence_tests/test_payload.py (9 hunks)
🧰 Additional context used
🧠 Learnings (1)
tests/congruence_tests/test_payload.py (1)
Learnt from: joein
PR: qdrant/qdrant-client#977
File: tests/congruence_tests/test_payload.py:645-651
Timestamp: 2025-05-14T20:18:46.248Z
Learning: In the Qdrant client, there is no separate `models.OverwritePayload` class. The `models.OverwritePayloadOperation` class is designed to accept a `models.SetPayload` object as its `overwrite_payload` parameter.
🧬 Code Graph Analysis (1)
tests/congruence_tests/test_payload.py (7)
tests/congruence_tests/test_common.py (4)
  • initialize_fixture_collection (93-109)
  • init_local (385-390)
  • init_remote (393-395)
  • compare_collections (175-197)
tests/congruence_tests/test_discovery.py (1)
  • local_client (38-42)
tests/congruence_tests/conftest.py (2)
  • local_client (13-17)
  • remote_client (21-25)
qdrant_client/qdrant_client.py (11)
  • QdrantClient (28-3167)
  • set_payload (1855-1949)
  • batch_update_points (2111-2152)
  • upsert (1565-1642)
  • delete (1808-1853)
  • count (1473-1512)
  • delete_payload (2018-2065)
  • clear_payload (2067-2109)
  • update_vectors (1644-1698)
  • delete_vectors (1700-1748)
  • overwrite_payload (1951-2016)
qdrant_client/local/local_collection.py (10)
  • set_payload (2453-2477)
  • batch_update_points (2528-2567)
  • upsert (2298-2334)
  • delete (2431-2441)
  • count (1903-1906)
  • delete_payload (2495-2511)
  • clear_payload (2513-2526)
  • update_vectors (2366-2376)
  • delete_vectors (2378-2393)
  • overwrite_payload (2479-2493)
qdrant_client/qdrant_remote.py (10)
  • set_payload (2245-2295)
  • batch_update_points (2442-2480)
  • upsert (1840-1918)
  • delete (2199-2243)
  • count (1738-1779)
  • delete_payload (2348-2394)
  • clear_payload (2396-2440)
  • update_vectors (1920-1959)
  • delete_vectors (1961-2011)
  • overwrite_payload (2297-2346)
qdrant_client/http/exceptions.py (1)
  • UnexpectedResponse (13-41)
⏰ Context from checks skipped due to timeout of 90000ms (5)
  • GitHub Check: Python 3.12.x on ubuntu-latest test
  • GitHub Check: Python 3.11.x on ubuntu-latest test
  • GitHub Check: Python 3.13.x on ubuntu-latest test
  • GitHub Check: Python 3.9.x on ubuntu-latest test
  • GitHub Check: Python 3.10.x on ubuntu-latest test
🔇 Additional comments (9)
tests/congruence_tests/test_payload.py (9)

5-5: Import for improved exception handling.

Adding the grpc import enables catching both REST-based UnexpectedResponse and gRPC-specific RpcError exceptions, enhancing error handling across both client communication methods.


34-40: Good parameterization of test to support both protocols.

Parameterizing the test with prefer_grpc and using the initialize_fixture_collection helper improves test structure and ensures coverage across both gRPC and REST communication protocols.


417-417: Improved error handling for both protocols.

Using a tuple to catch both UnexpectedResponse and grpc.RpcError handles exceptions from both REST and gRPC clients, ensuring consistent test behavior regardless of the protocol being used.


310-382: Comprehensive tests for nested payload updates.

The added test cases thoroughly validate key payload update scenarios:

  • Updating values in nested arrays
  • Modifying non-existing array elements
  • Adding fields to dictionaries in arrays
  • Adding fields to empty payloads
  • Replacing arrays with dictionaries and vice versa
  • Deep nesting operations

These tests ensure that the key parameter functionality works correctly across various complex payload structures.


452-487: Verified batch update with upsert operations.

This new test validates batch_update_points functionality with both PointsBatch and PointsList upsert operations, ensuring consistent behavior between local and remote clients across both protocols.


509-569: Complete validation of payload modification operations.

This test comprehensively verifies deletion and clearing of payloads via the batch API:

  • Delete payload keys for specific points
  • Delete payload keys using a filter
  • Clear entire payload for points

The test ensures consistency between local and remote clients with each operation.


571-620: Thorough validation of vector operations.

This test covers updating and deleting vectors through the batch API:

  • Updating vector values for specific points
  • Deleting vectors specified by point IDs
  • Deleting vectors using a filter

These tests are essential for verifying the vector normalization process fixed in this PR.


622-672: Well-designed test for payload operations with key support.

This test comprehensively validates the core fix in this PR - enabling key-based partial updates in batch operations:

  1. Setting payload for specific points without a key
  2. Setting nested payload with a key parameter
  3. Overwriting payload completely

The test confirms consistent behavior between local and remote clients across both protocols.


661-664: Correct usage of models.SetPayload with OverwritePayloadOperation.

The implementation correctly uses models.SetPayload as the parameter for overwrite_payload in the OverwritePayloadOperation. This aligns with the codebase structure where there is no separate models.OverwritePayload class.

Comment thread tests/congruence_tests/test_payload.py Outdated
Comment on lines +331 to +333
set_payload(
payload, new_payload, key
) # todo: uncomment when https://github.com/qdrant/qdrant/issues/6449 is resolved
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Verification agent

🧩 Analysis chain

Test marked with TODO pending issue resolution.

This test case for setting a key on an empty payload references a known issue (GitHub #6449). Consider adding a comment explaining the expected behavior once the issue is resolved.

Can you verify the current status of this issue by checking:


🏁 Script executed:

#!/bin/bash
# Check the status of the referenced GitHub issue
gh issue view https://github.com/qdrant/qdrant/issues/6449 --json title,state,body,comments | jq '{title: .title, state: .state, comments: .comments | length}'

Length of output: 304


Add expected behavior note for TODO on issue #6449

The GitHub issue (qdrant/qdrant#6449) is still open, so the # todo marker remains valid. To aid future maintainers, please update the inline comment to spell out what the test should do once that issue is resolved.

• File: tests/congruence_tests/test_payload.py
• Lines: 331–333

Suggested update:

-    set_payload(
-        payload, new_payload, key
-    )  # todo: uncomment when https://github.com/qdrant/qdrant/issues/6449 is resolved
+    set_payload(
+        payload, new_payload, key
+    )  # TODO: uncomment when https://github.com/qdrant/qdrant/issues/6449 is resolved 
+    # Expected behavior: setting a key on an empty payload should add the key with the given value
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
set_payload(
payload, new_payload, key
) # todo: uncomment when https://github.com/qdrant/qdrant/issues/6449 is resolved
set_payload(
payload, new_payload, key
) # TODO: uncomment when https://github.com/qdrant/qdrant/issues/6449 is resolved
# Expected behavior: setting a key on an empty payload should add the key with the given value

@joein joein merged commit e32daed into dev May 15, 2025
14 of 34 checks passed
joein added a commit that referenced this pull request Jun 16, 2025
#977)

* fix: fix set payload in batch update points, fix operation conversions, add tests

* new: add some checks

* add comments to test cases

* fix: return comment

* tests: uncomment test due to a fix in core
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants