You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Keep generic structural encodings above other encodings #10748
Code that looks for a generic structural encoding (Constant, Dict, RunEnd, Sparse, Chunked, Masked or Interleave) usually checks only the root of the array tree. The Arrow dictionary executor checks array.is::<Dict>() || array.is::<Constant>() (vortex-arrow/src/executor/dictionary.rs), the Arrow run-end executor and the DuckDB exporter (vortex-duckdb/src/exporter/mod.rs) match on the root encoding, and the dict layout writer checks compressed.is::<Dict>() on the compressor output (vortex-layout/src/layouts/dict/writer.rs). When another encoding wraps one of these, the fast paths stop applying without an error.
Nothing keeps these encodings at the top today. The Narrow work (#10733) adds reduce rules that move them above a Narrow, for example Narrow(Dict(codes, values)) becomes Dict(codes, Narrow(values)), but those rules cover only Narrow. Extension wrappers have the same need (#7683).
Proposed solution
Make the rule general: after optimization, no encoding sits directly above a generic structural encoding. Each wrapping encoding can get its own pull-up reduce rules, or one shared mechanism can do it for every encoding that can sit below them. The rules only ever move structural encodings up, so no pair of rules can loop.
Open questions:
Is the list above the complete set of generic structural encodings?
Per-encoding rules, or one shared mechanism?
Additional context
Found while planning #10733. Research notes: #10633.
Activity