[SPARK-58269][SQL] Infer generated column partition filters#57439
Open
szehon-ho wants to merge 3 commits into
Open
[SPARK-58269][SQL] Infer generated column partition filters#57439szehon-ho wants to merge 3 commits into
szehon-ho wants to merge 3 commits into
Conversation
- Use failOnError=false for DATE_FORMAT unix_timestamp comparisons so ANSI mode cannot throw on unparseable partition strings. - Skip range derivation for trunc(string) partitions where lexical and chronological order diverge.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR allows DataSource V2 connectors that consume Catalyst filters to opt in to partition-filter inference for identity-partitioned generated columns. When a query filters a generated column's base column, Spark derives a conservative filter on the generated partition column and includes it in filter pushdown.
The supported generation expressions include casts to date, date/time extraction and truncation,
date_format,substring, identity expressions, and composite year/month/day/hour partitions. Derived filters preserve null and truncation boundaries so they cannot remove matching rows.The PR also adds an in-memory test connector that evaluates Catalyst partition filters, unit coverage for expression recognition and derivation, and end-to-end coverage for pushdown and partition pruning.
Why are the changes needed?
Filters on a base column do not directly reference its generated partition column. Without inference, V2 data sources cannot use those filters for partition pruning and may scan every partition even when the generation expression makes a safe, selective partition predicate available.
Does this PR introduce any user-facing change?
Yes. DataSource V2 implementations that explicitly opt in can receive additional generated-column partition filters and prune partitions more effectively. Query results are unchanged because every derived filter is implied by the original data filter and is used only for pushdown.
How was this patch tested?
Added
GeneratedColumnPartitionFilterSuitefor expression-level derivation and negative cases, andDataSourceV2GeneratedColumnPartitionFilterSuitefor end-to-end pushdown, pruning, opt-out, null, parser-policy, case-sensitivity, nested-column, and multi-column behavior.Targeted test command (running at PR creation time):
JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 \ MAVEN_MIRROR_URL=https://maven-proxy.cloud.databricks.com \ build/sbt 'sql/testOnly org.apache.spark.sql.execution.datasources.v2.GeneratedColumnPartitionFilterSuite org.apache.spark.sql.execution.datasources.v2.DataSourceV2GeneratedColumnPartitionFilterSuite'Also verified
git diff --check, non-ASCII content, changed-file line lengths, and IDE diagnostics.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (GPT-5.6 Sol)