Conversation
Bypass fmgr dispatch in GetScanItems by resolving the opclass distance
proc (FUNCTION 1: vector_l2_squared_distance / vector_negative_inner_product)
to exported C kernels, and replace the per-tuple slot + tuplesort
machinery with a work_mem-bounded {distance, tid} candidate array that is
sorted once per batch. Ties are broken by TID; capacity overflow or any
unrecognized kernel falls back to the original path.
Index tuples for dim <= 30 use a 1-byte short varlena header; kernels take
(dim, ax, bx) and IvfflatVectorParts resolves both header layouts inline.
A/B on the frozen benchmark (3 operators x 14 lists/probes cells, R=5,
N=100k, dim=128): recall unchanged (worst delta 0.0011 on acceptance
cells), wall time mean 1.43x / median 1.44x faster.
Adds test/sql/ivfflat_fastscan.sql covering short-header (dim 8) and
4-byte-header (dim 64) scans for L2/IP/cosine, the work_mem overflow
fallback, and lateral rescans. Full suite: 15/15.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Profiling an IVFFlat index scan at moderate
probesshows that the distancecomputation itself is only a small fraction of scan CPU. Most time goes to
per-tuple machinery in
GetScanItems:called through
FunctionCallInvokefor every candidate, paying fullargument marshalling (each
Vector *re-validated,dimre-checked).heap tuple and run through
tuplesort, even though only the topk(bounded bywork_mem) can survive.Both costs scale with the number of scanned candidates (probes x cluster
size), not with result size, so they dominate exactly when recall settings
are aggressive.
What this PR changes
ivfflatbeginscanresolves the opclass distance proc (FUNCTION 1) toone of three exported C kernels —
vector_l2_squared_distance,vector_negative_inner_product(shared byip_opsandcosine_ops) —and stores the function pointer on the scan. Kernels take
(int dim, const float *ax, const float *bx).IvfflatVectorParts()resolves both varlena header layouts inline:index tuples for
dim <= 30use a 1-byte short header, and the previousdirect
(Vector *)cast was only valid for 4-byte headers. (This was thesource of out-of-bounds reads on small-
dimdata — see Testing.)GetScanItemsfast path accumulates{distance, tid}pairs into awork_mem-bounded candidate array, sorted once per batch. Ties arebroken by TID.
unrecognized opclass proc all revert to the original slot + tuplesort
path. NULL order-by values keep using the original path.
Benchmark
Frozen configuration: deterministic clustered-Gaussian synthetic data, N=100k, dim=128, 1,000 queries,
P2 matrix (3 operators x 14 lists/probes cells), R=5 repetitions per arm,
same-machine A/B (OpenTenBase / PostgreSQL 19beta3, gcc 13.3, x86_64).
Recall = mean Recall@10 per cell; QPS = 1,000 queries / freeze-robust mean wall seconds (concurrency=1); wall time = median of repetitions with
freeze outliers (>5x median) excluded.
against a 0.02 tolerance.
largest single-cell gain 2.93x (trend cells at low lists).
Testing
test/sql/ivfflat_fastscan.sqlcovers both varlena header forms(dim 8 short-header, dim 64 4-byte-header) for L2/IP/cosine, the
work_memoverflow fallback, and lateral rescans. The dim<=30 shortheader case is a regression guard: the naive
(Vector *)cast reads agarbage
dimthere and crashes.full-probes equality with exact scans (diff = 0) on both header forms.
TID, whereas the tuplesort path's tie order is unspecified. The new test
pins the fallback query with a secondary sort key where ties occur.
Notes
build already auto-vectorizes the inner loops with AVX2).