Skip to content

Add local pHash deduplication utility #462

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vinay553 merged 6 commits into from
Jun 5, 2026
Merged

Add local pHash deduplication utility #462

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
e196bda
Add local pHash deduplication utility
vinay553 Jun 4, 2026
88e711a
Address local dedup review feedback
vinay553 Jun 4, 2026
c3d0726
Fix local dedup pylint issues
vinay553 Jun 4, 2026
a7042b0
Address dedup review nits and type annotations
vinay553 Jun 5, 2026
4654cf3
Clarify rotated partition purpose
vinay553 Jun 5, 2026
a0409c0
Run pre-commit formatting
vinay553 Jun 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions nucleus/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
"DeduplicationJob",
"DeduplicationResult",
"DeduplicationStats",
"LocalDeduplicationResult",
"BoxPrediction",
"CameraParams",
"CategoryAnnotation",
Expand Down Expand Up @@ -43,6 +44,7 @@
"SegmentationPrediction",
"Slice",
"VideoScene",
"deduplicate_by_phash",
]

import datetime
Expand Down Expand Up @@ -147,6 +149,10 @@
NucleusAPIError,
)
from .job import CustomerJobTypes
from .local_deduplication import (
LocalDeduplicationResult,
deduplicate_by_phash,
)
from .model import Model
from .model_run import ModelRun
from .payload_constructor import (
Expand Down
4 changes: 3 additions & 1 deletion nucleus/data_transfer_object/dataset_info.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ class DatasetInfo(DictCompatibleModel):
item_metadata_schema: Optional[Dict[str, Any]] = None
tags: List[str] = []

@validator("tags", pre=True, always=True) # pylint: disable=used-before-assignment
@validator(
"tags", pre=True, always=True
) # pylint: disable=used-before-assignment
def coerce_null_tags(cls, v): # pylint: disable=no-self-argument
return v if v is not None else []
8 changes: 6 additions & 2 deletions nucleus/dataset.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -454,7 +454,9 @@ def add_tags(self, tags: List[str]) -> List[str]:
Updated list of all tags on the dataset.
"""
if isinstance(tags, str):
raise TypeError("tags must be a list of strings, not a single string")
raise TypeError(
"tags must be a list of strings, not a single string"
)
response = self._client.make_request(
{"tags": tags}, f"dataset/{self.id}/tags", requests.post
)
Expand All @@ -470,7 +472,9 @@ def remove_tags(self, tags: List[str]) -> List[str]:
Updated list of remaining tags on the dataset.
"""
if isinstance(tags, str):
raise TypeError("tags must be a list of strings, not a single string")
raise TypeError(
"tags must be a list of strings, not a single string"
)
response = self._client.make_request(
{"tags": tags}, f"dataset/{self.id}/tags", requests.delete
)
Expand Down
Loading