Skip to content

REST: Apply the scan filter when the server omits the residual - #4379

Merged
Fokko merged 2 commits into
apache:mainfrom
linzch3:codex/rest-scan-residual
Nov 17, 2026
Merged

Fokko merged 2 commits into
apache:mainfrom
linzch3:codex/rest-scan-residual

Conversation

@linzch3

@linzch3 linzch3 commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

When a REST server omits residual-filter from a file scan task, PyIceberg treats the task as unfiltered. For a Parquet file with x = [1, 2, 1], scan(row_filter="x != 1").count() returns 3 while to_arrow() returns 1.

Use the original scan filter as the fallback residual for both the initial plan and any fetched task batches, as required by the Iceberg REST specification. Explicit server residuals still take precedence.

This addresses the REST case reported in #4278.

Are these changes tested?

The regression uses a local Parquet file and exercises both the initial response and pagination. It fails on main and the current head of #4278, and passes with this change. The planning tests also verify that an explicit true residual is preserved and that an unfiltered request still gets an unfiltered task.

make test: 4,685 passed, 5 skipped. make lint passed.

Are there any user-facing changes?

Filtered counts now agree with Arrow scans when a server omits the residual. Affected files may be read to apply the filter.

AI assistance

An AI assistant helped trace the planning code and draft the tests; I reviewed and ran everything and take responsibility for the change.

Use the original request filter for file scan tasks without a server residual so count() applies the same filter as Arrow scans.
@Fokko
Fokko self-requested a review November 17, 2026 18:52

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @linzch3! Nice catch, and great find on the mismatch between count() and to_arrow(). The fallback to the original scan filter matches the REST spec, and explicit server residuals still win.

I pulled the branch and verified: the new regression test fails on main and passes with the change, and the full test_scan_planning_models.py suite is green (61 passed).

A couple of small notes:

  • pyiceberg/catalog/rest/__init__.py:907: the inline request.filter if request.filter is not None else AlwaysTrue() is fine, but it could read slightly cleaner as a default on the parameter, e.g.
        tasks = self._expand_plan_tasks(identifier, response, request.filter or AlwaysTrue())
  • pyiceberg/table/__init__.py:2377: good switch from truthiness to is not None, since an explicit false residual would previously have been swallowed into ALWAYS_TRUE.

Non-blocking, CI is green. LGTM!

@Fokko
Fokko enabled auto-merge November 17, 2026 20:27

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's go @linzch3 Thanks!

@Fokko
Fokko added this pull request to the merge queue Nov 17, 2026
Merged via the queue into apache:main with commit 0ebc58b Nov 17, 2026
20 of 21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants