Skip to content

Keep generic structural encodings above other encodings #10748

Open

Description

@connortsui20

Problem or motivation

Code that looks for a generic structural encoding (Constant, Dict, RunEnd, Sparse, Chunked, Masked or Interleave) usually checks only the root of the array tree. The Arrow dictionary executor checks array.is::<Dict>() || array.is::<Constant>() (vortex-arrow/src/executor/dictionary.rs), the Arrow run-end executor and the DuckDB exporter (vortex-duckdb/src/exporter/mod.rs) match on the root encoding, and the dict layout writer checks compressed.is::<Dict>() on the compressor output (vortex-layout/src/layouts/dict/writer.rs). When another encoding wraps one of these, the fast paths stop applying without an error.

Nothing keeps these encodings at the top today. The Narrow work (#10733) adds reduce rules that move them above a Narrow, for example Narrow(Dict(codes, values)) becomes Dict(codes, Narrow(values)), but those rules cover only Narrow. Extension wrappers have the same need (#7683).

Proposed solution

Make the rule general: after optimization, no encoding sits directly above a generic structural encoding. Each wrapping encoding can get its own pull-up reduce rules, or one shared mechanism can do it for every encoding that can sit below them. The rules only ever move structural encodings up, so no pair of rules can loop.

Open questions:

  • Is the list above the complete set of generic structural encodings?
  • Per-encoding rules, or one shared mechanism?

Additional context

Found while planning #10733. Research notes: #10633.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureA feature request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions