Skip to content

refactor: count items from hnsw index #1765

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
shobhit9957 wants to merge 16 commits into from
Closed

refactor: count items from hnsw index #1765

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
bf6382e
solved the good first issue in the hnswlib.py file
shobhit9957 Aug 24, 2023
9466389
Edited the code.
shobhit9957 Aug 24, 2023
83fa2b7
fix: #1764, assigned the first indices
shobhit9957 Aug 25, 2023
45f4ae4
fix : #1764 Fix accessing HNSW index document count
shobhit9957 Aug 25, 2023
042d5c3
fix: #1764 removed the comment
shobhit9957 Aug 25, 2023
8ecffda
fixes: #1765, implementing element_count()
shobhit9957 Aug 28, 2023
13a9ebf
fixes: #1765 added the attribute(in the hnswlib.py)
shobhit9957 Aug 28, 2023
0b428c6
fixes #1765
shobhit9957 Sep 3, 2023
803fc5d
fixes: 1765 got the first index in the dictionary
shobhit9957 Sep 4, 2023
1057805
fixes: #1765, step 1
shobhit9957 Sep 5, 2023
f12bf20
fixes: #1765 step 1 again
shobhit9957 Sep 6, 2023
81988c1
fixes: #1765
shobhit9957 Sep 19, 2023
d576e60
Revert "fixes: #1765 step 1 again"
shobhit9957 Sep 19, 2023
229d73f
fixes: #1765
shobhit9957 Sep 19, 2023
678f1dd
fixes: #1765
shobhit9957 Sep 20, 2023
436937b
fixes: #1765
shobhit9957 Sep 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docarray/index/abstract.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,14 @@ class _NewSchema(self._schema): # type: ignore
self._subindices: Dict[str, BaseDocIndex] = {}
self._init_subindex()

def is_index_empty(self) -> bool:
"""
Check if index is empty by comparing the number of documents to zero.

:return: True if the index is empty, False otherwise.
"""
return self.num_docs() == 0

###############################################
# Inner classes for query builder and configs #
# Subclasses must subclass & implement these #
Expand Down
12 changes: 5 additions & 7 deletions docarray/index/backends/hnswlib.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,7 @@
_raise_not_composable,
_raise_not_supported,
)
from docarray.index.backends.helper import (
_collect_query_args,
)
from docarray.index.backends.helper import _collect_query_args
from docarray.proto import DocProto
from docarray.typing.tensor.abstract_tensor import AbstractTensor
from docarray.typing.tensor.ndarray import NdArray
Expand Down Expand Up @@ -425,11 +423,11 @@ def _doc_exists(self, doc_id: str) -> bool:

def num_docs(self) -> int:
"""
Get the number of documents.
Get number of documents.
"""
if self._num_docs == 0:
self._num_docs = self._get_num_docs_sqlite()
return self._num_docs
if self.is_index_empty == 0:

@JoanFM JoanFM Sep 20, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this does not make sense, is_index_empty should not be compared to 0, it already gives a boolean

self.is_index_empty = self._get_num_docs_hnsw()
return self.is_index_empty

###############################################
# Helpers #
Expand Down