feat(nodejs): add IVF_HNSW_FLAT vector index support#3367
Draft
shenganzhang wants to merge 2 commits into
Draft
Conversation
Expose the `IVF_HNSW_FLAT` index (IVF partitions with an HNSW graph per partition, no quantization) through the Rust core and Python SDK. Until now this index type was documented but not actually wired up in the lancedb Python wrapper, even though `pylance` accepts it. Rust core changes: * Add `Index::IvfHnswFlat` / `IndexType::IvfHnswFlat` variants and the corresponding `IvfHnswFlatIndexBuilder` (modelled on the existing `IvfHnswSqIndexBuilder`), with display/parse round-trips. * Build the Lance index params via the existing `VectorIndexParams::ivf_hnsw(...)` helper for consistency with the other IVF_HNSW_* variants. * Forward the new variant in `RemoteTable::create_index` and cover both the default-config and customised-config serialisation in the parametrised remote test. Python binding changes: * New `HnswFlat` dataclass + `IvfHnswFlat` backwards-compat alias. * PyO3 `extract_index_params` recognises the `HnswFlat` config class. * `LanceTable.create_index(index_type="IVF_HNSW_FLAT", ...)` and the sync `RemoteTable.create_index` both dispatch to the new config. * `IndexStatistics.index_type` Literal and `_lancedb.pyi` stubs cover the new type so pyright stays happy. * Async + sync integration tests and a dispatcher unit test mirror the pattern used for `IVF_HNSW_SQ`. A matching Node.js / TypeScript binding will land in a follow-up PR. Closes lancedb#3331
Mirror the Python `IVF_HNSW_FLAT` support added in the previous commit
on the Node.js / TypeScript side, so users of all bindings can build
IVF + per-partition HNSW indexes without quantization.
* napi: new `Index.hnsw_flat` factory wrapping
`IvfHnswFlatIndexBuilder`, parallel to the existing `hnsw_sq` and
`hnsw_pq` factories.
* ts: new `HnswFlatOptions` interface and `Index.hnswFlat(...)` static
method, re-exported from `lancedb/index.ts`.
* test: cover index creation through `Index.hnswFlat({ numPartitions })`
in `__test__/table.test.ts`.
Refs lancedb#3331
Co-authored-by: Cursor <cursoragent@cursor.com>
github-actions
Bot
added
enhancement
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Mirror the Python
IVF_HNSW_FLATsupport added in #3366 on the Node.js / TypeScript side, so users of all bindings can build IVF + per-partition HNSW indexes without quantisation.nodejs/src/index.rs): newIndex.hnsw_flatfactory wrappingIvfHnswFlatIndexBuilder, parallel to the existinghnsw_sq/hnsw_pqfactories.nodejs/lancedb/indices.ts): newHnswFlatOptionsinterface andIndex.hnswFlat(...)static method, re-exported fromnodejs/lancedb/index.ts.nodejs/__test__/table.test.ts): cover index creation throughIndex.hnswFlat({ numPartitions }).Refs #3331
Test plan