Repository navigation
REST: Apply the scan filter when the server omits the residual - #4379
Merged
Merged
Conversation
Use the original request filter for file scan tasks without a server residual so count() applies the same filter as Arrow scans.
Fokko
self-requested a review
November 17, 2026 18:52
Fokko
reviewed
Nov 17, 2026
Fokko
left a comment
Contributor
There was a problem hiding this comment.
Thanks @linzch3! Nice catch, and great find on the mismatch between count() and to_arrow(). The fallback to the original scan filter matches the REST spec, and explicit server residuals still win.
I pulled the branch and verified: the new regression test fails on main and passes with the change, and the full test_scan_planning_models.py suite is green (61 passed).
A couple of small notes:
pyiceberg/catalog/rest/__init__.py:907: the inlinerequest.filter if request.filter is not None else AlwaysTrue()is fine, but it could read slightly cleaner as a default on the parameter, e.g.
tasks = self._expand_plan_tasks(identifier, response, request.filter or AlwaysTrue())
pyiceberg/table/__init__.py:2377: good switch from truthiness tois not None, since an explicitfalseresidual would previously have been swallowed intoALWAYS_TRUE.
Non-blocking, CI is green. LGTM!
Fokko
enabled auto-merge
November 17, 2026 20:27
Fokko
approved these changes
Nov 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
When a REST server omits
residual-filterfrom a file scan task, PyIceberg treats the task as unfiltered. For a Parquet file withx = [1, 2, 1],scan(row_filter="x != 1").count()returns 3 whileto_arrow()returns 1.Use the original scan filter as the fallback residual for both the initial plan and any fetched task batches, as required by the Iceberg REST specification. Explicit server residuals still take precedence.
This addresses the REST case reported in #4278.
Are these changes tested?
The regression uses a local Parquet file and exercises both the initial response and pagination. It fails on
mainand the current head of #4278, and passes with this change. The planning tests also verify that an explicit true residual is preserved and that an unfiltered request still gets an unfiltered task.make test: 4,685 passed, 5 skipped.make lintpassed.Are there any user-facing changes?
Filtered counts now agree with Arrow scans when a server omits the residual. Affected files may be read to apply the filter.
AI assistance
An AI assistant helped trace the planning code and draft the tests; I reviewed and ran everything and take responsibility for the change.