Why does queryNode.externalCollection.useTakeForOutput default to true? take() re-reads TOS every query and hurts under concurrency #53140

wu-xingying · 2026-09-03T03:33:19Z

wu-xingying
Sep 3, 2026

Summary

For external collections, queryNode.externalCollection.useTakeForOutput defaults to
true, which routes output-field fetching through the take() path instead of
bulk_subscript. In our testing, take() appears to re-read the source files from
object storage (TOS/S3) on every query, with no reuse across queries, whereas the
bulk_subscript path (i.e. useTakeForOutput=false) populates a cache and reuses it.

This gap only becomes significant when the output fields are large or
variable-length — e.g. vector (list<float>), TEXT/JSON, or other var-len columns —
because that is when per-query download + decode dominates. For small fixed-width
scalar outputs the cost is negligible and the two paths look the same. In the
measurements below we use a vector field as the output to make the effect
visible. The gap is largest with large source files and, especially, under
concurrent queries against the same segments.

I'd like to understand the design rationale behind defaulting external collections
to take(), and whether the downside below is expected or something to be improved.

Environment

Milvus version: 3.0.x
Param introduced in 3.0.0, refreshable: true
- queryNode.externalCollection.useTakeForOutput — default true
- queryNode.internalCollection.useTakeForOutput — default false

What we observe

	`useTakeForOutput=true` (external default)	`useTakeForOutput=false`
Output fetch path	`take()`	`bulk_subscript`
Cache reuse across queries	No — re-reads source files from object storage	Yes — data lands in cache and is reused
Repeated / concurrent queries on same segment	Each query re-downloads & re-decodes the relevant row group(s)	First query pays cost, subsequent queries hit cache
Peak memory	Spread across queries, but re-incurred each time	Paid once at/after load, then amortized

The two behaviors diverge sharply when:

source files are large (row groups are big → each take() pulls a lot), and
queries are concurrent on the same segments (N segments → N independent
downloads/decodes of overlapping column chunks; no sharing).

Questions

What is the design rationale for defaulting external collections to take()
(useTakeForOutput=true) while internal collections default to bulk_subscript
(false)?
Is the per-query re-read from object storage (no cross-query reuse on the take
path) expected by design, or considered a known limitation?
For large-file + high-concurrency external workloads, is flipping this to false
the recommended mitigation, and what are the trade-offs (e.g. moving the memory/decoding
cost to load time, local materialization, potential OOM risk)?
Are there plans (or existing knobs) to reduce the redundancy on the take path — e.g.
sharing a reader / column-chunk bytes across segments that point to the same physical
file and projection, so PreBuffer happens once and is reused across queries?

Some Experiments

Test setup: cohere 1M dataset (~3 GB on disk), single QueryNode, HNSW index,
one external collection over the source Parquet. targetRowsPerSegment maps the
1M rows to 4 segments by default. Same query, only the output path differs.

1. Sequential queries (single client, 3 identical searches)

Latency per phase (seconds):

output path	load	search #1	search #2	search #3
`bulk_subscript` (`useTakeForOutput=false`)	1.84	50.66	0.011	0.010
`take` (`useTakeForOutput=true`, default)	2.26	45.56	43.51	43.52

The first search is comparable (both pay the cold cost). From the second search
on, bulk_subscript serves from cache in ~10 ms, while take re-reads and
re-decodes from object storage every single time (~43 s, unchanged across
repeats). That is a ~4000x per-query gap on a warm collection.

2. Concurrent queries (5 concurrent searches × 3 rounds), take path

take()

bulk_subscript()

amasen02 · 2026-09-05T08:47:56Z

amasen02
Sep 5, 2026

Your benchmarks clearly isolate the behavior, and your read of the execution paths is spot on. Here is the background behind the design choice and the trade-offs:

1. Design Rationale for Defaulting `take()` on External Collections

The difference in defaults comes down to memory containment on unmanaged data:

Internal collections (useTakeForOutput=false): Segcore manages the full segment lifecycle (compaction, binlogs, indexed segments). Columnar data is already pinned in Segcore's memory-mapped buffers or local chunk manager cache. bulk_subscript reads directly from these local buffers with virtually zero I/O overhead.
External collections (useTakeForOutput=true): External collections query Parquet/Iceberg tables residing directly in object storage (S3/TOS/GCS) without full ingestion. Because external tables are often multi-terabyte data lakes, defaulting to bulk_subscript would force QueryNode to materialize and cache entire column chunks/row groups into local RAM/disk cache. On high-dimensional vectors or wide JSON/text columns, this quickly exhausts QueryNode memory buffers and triggers OOMs.

take() was chosen as the safe, conservative default: it acts as a streaming, point-lookup path using Arrow/Parquet reader projection to pull only the row IDs returned by the top-k candidate set without polluting QueryNode's memory pool with full columns.

2. Is Per-Query Object Storage Re-read Expected?

Yes, in the current 3.0.x implementation this is a known trade-off of the take() path. The reader constructs transient buffers for the query projection, streams the requested row offsets from object storage, and discards them once the gRPC result payload is serialized. There is no intermediate chunk-level LRU cache layer hooked into the take() execution path yet.

3. Recommended Mitigation for Bounded / High-Concurrency Workloads

If your working set fits in QueryNode memory (such as your 1M Cohere dataset at ~3 GB), flipping queryNode.externalCollection.useTakeForOutput: false is the recommended mitigation.

Since the parameter is refreshable: true, you can toggle it in milvus.yaml or via Helm values without restarting the cluster:

queryNode:
  externalCollection:
    useTakeForOutput: false

Trade-offs to monitor:

Memory Ceiling: First query pays the cold download and populates Segcore's chunk cache. Subsequent queries hit local memory (~10 ms), but you must ensure QueryNode's container memory limits (resources.limits.memory) and queryNode.memory.pool.size accommodate the uncompressed size of the projected output columns across all active segments.
Initial Query Latency: The cold query absorbs the chunk download and decode cost before serving results.

4. Reducing Redundancy on the `take` Path

Currently, there is no shared cross-segment reader cache for identical object URIs on the take() path. If you must keep useTakeForOutput: true (e.g. memory constraints prevent full column caching):

Parquet Row-Group Sizing: When generating the source Parquet files, tune the row-group size to 64 MB – 128 MB rather than larger 512 MB+ blocks. Smaller row groups reduce the byte-range amplification that take() pulls per point-lookup.
Column Projection: Only request the exact scalar fields needed in output_fields rather than returning full payload objects or raw high-dimensional vector embeddings in the search response.

1 reply

wu-xingying Sep 7, 2026
Author

Thanks, this is really helpful — a few clarifications / follow-ups on the take() characterization.

I don't think take() is a lightweight point-lookup that avoids the memory pool. In our
testing its effective granularity is a whole row group: the first take() pre-buffers
the entire compressed column chunk and decodes the full row group in one pass (for
list<float>, values + def-levels + rep-levels each rounded up, ~2.9x the source), then
selects the few target rows. So to return a handful of rows it materializes the whole row
group.

Crucially, this happens outside the caching-layer memory accounting — the take() path
does not go through the CacheSlot / water-level system, so its transient buffers are not
tracked and cannot be throttled to prevent OOM. External source files can have very large
row groups, so this is a real read-amplification / peak-memory risk, and it recurs on
every query and per segment under concurrency.

Given that, I'd like to challenge the framing a bit: if the recommended mitigation is to
keep row groups small (64–128 MB), then a single bulk_subscript chunk is also small — and
bulk_subscript additionally (a) avoids the repeated per-query re-download, (b) is
chunk-granular and LRU-evictable (so it's bounded and participates in the memory
water-level), which the take() path is not. In other words, with bounded row groups,
bulk_subscript + eviction seems to give better memory protection, not worse.

So my main concern remains the repeated download + concurrent-read redundancy on the
take() path, which is clearly the bigger practical penalty. Questions:

Is there any roadmap item to add a chunk-level cache (or shared reader) to the take()
path, so PreBuffer happens once and is reused across queries?
In particular, when multiple segments point at the same physical file + projection,
is there any plan to share the reader / dedupe the column-chunk bytes across segments,
rather than each segment downloading and decoding independently?
Would streaming the row-group decode in bounded batches (instead of one full-row-group
NextBatch) be considered, to cap the peak on the take() path?

Navigation Menu

Uh oh!

Why does queryNode.externalCollection.useTakeForOutput default to true? take() re-reads TOS every query and hurts under concurrency #53140

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 1 reply

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

Why does queryNode.externalCollection.useTakeForOutput default to true? take() re-reads TOS every query and hurts under concurrency #53140

Uh oh!

wu-xingying Sep 3, 2026

Summary

Environment

What we observe

Questions

Some Experiments

1. Sequential queries (single client, 3 identical searches)

2. Concurrent queries (5 concurrent searches × 3 rounds), take path

Replies: 1 comment · 1 reply

Uh oh!

amasen02 Sep 5, 2026

1. Design Rationale for Defaulting take() on External Collections

2. Is Per-Query Object Storage Re-read Expected?

3. Recommended Mitigation for Bounded / High-Concurrency Workloads

Trade-offs to monitor:

4. Reducing Redundancy on the take Path

Uh oh!

wu-xingying Sep 7, 2026 Author

wu-xingying
Sep 3, 2026

Replies: 1 comment 1 reply

amasen02
Sep 5, 2026

1. Design Rationale for Defaulting `take()` on External Collections

4. Reducing Redundancy on the `take` Path

wu-xingying Sep 7, 2026
Author