Skip to content

fix: Correct Flink PyArrow dependency constraints - #6604

Merged
ntkathole merged 2 commits into
feast-dev:masterfrom
zelinewang:fix/flink-pyarrow-extra
Jul 19, 2026
Merged

fix: Correct Flink PyArrow dependency constraints#6604
ntkathole merged 2 commits into
feast-dev:masterfrom
zelinewang:fix/flink-pyarrow-extra

Conversation

@zelinewang

Copy link
Copy Markdown
Contributor

The Flink extra currently combines a base pyarrow>=21.0.0 requirement with a
pyarrow<21.0.0 extra requirement. The attempted extra != 'flink' exclusion
does not prevent pip or uv from evaluating both requirements, so
feast[flink] is unsatisfiable.

This change puts the shared pyarrow>=16.1.0 lower bound in the base
dependencies and adds only pyarrow<21.0.0 to the Flink extra. A normal Feast
install can still select the latest PyArrow release, while a Flink install gets
the compatible >=16.1.0,<21.0.0 intersection.

The dependency regression check now lives in a focused test module that does
not import the Flink runtime.

Which issue(s) this PR fixes

Fixes #6583.

Checks

  • I've made sure the tests are passing.
  • My commits are signed off (git commit -s).
  • My PR title follows conventional commits format.

Testing Strategy

  • Unit tests
  • Integration tests
  • Manual tests
  • Testing is not required for this change

Validated with:

  • the focused dependency test on Python 3.10;
  • dry-run uv resolution for both .[flink] and the default project;
  • a wheel build and inspection of its Requires-Dist metadata;
  • Ruff check and format verification.

Release note

fix: Make the Flink extra installable by applying its PyArrow upper bound only when the extra is selected.

@zelinewang
zelinewang requested a review from a team as a code owner July 14, 2026 20:22

@shuchu shuchu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@shuchu

shuchu commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

the CI pipeline fails, @zelinewang can you try to update the "pixi.lock" file by running the following command?

pixi lock

@ntkathole

Copy link
Copy Markdown
Member

Also run make lock-python-dependencies-all to update requirements

@zelinewang

Copy link
Copy Markdown
Contributor Author

done, ran pixi lock plus the requirements recompile for all three python versions. heads up the pybuild-deps step fails because faiss-cpu==1.14.3 has no sdist on pypi and that pin is already in master's minimal-sdist files, so I left the three -build.txt files untouched. also the fresh resolve moves pyarrow to 21.0.0 on py3.10 (24 on 3.11/3.12), both within the corrected floor

@shuchu

shuchu commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

here is the error info:

pixi install -e duckdb-tests --locked
  /home/runner/.pixi/bin/pixi install -e duckdb-tests --locked --manifest-path pyproject.toml --color always
  Error:   × Lock-file version 7 is newer than supported
    help: Maximum supported version: 6 (pixi v0.63.1)
          Cannot continue with --locked or --frozen mode as the lock-file cannot be read.
          Try running `pixi self-update` to update to the latest version.

Comment thread pixi.lock Outdated
@zelinewang

Copy link
Copy Markdown
Contributor Author

oops, my local pixi was 0.73 which writes the v7 format. redid it with an older one, back to v6 now and the diff is way smaller. also fixed the missing DCO sign-off

@zelinewang
zelinewang force-pushed the fix/flink-pyarrow-extra branch from d3fc474 to 3309da1 Compare July 17, 2026 23:53
@ntkathole

Copy link
Copy Markdown
Member

@zelinewang needs rebase on master, there are conflicts

Signed-off-by: Zane Wang <zelinwang10@gmail.com>
Signed-off-by: Zane Wang <zelinwang10@gmail.com>
@zelinewang

Copy link
Copy Markdown
Contributor Author

rebased on master and redid the lockfiles there, conflicts are gone. the diff also picks up pyiceberg since the iceberg change hadn't updated requirements yet

@zelinewang
zelinewang force-pushed the fix/flink-pyarrow-extra branch from 3309da1 to 81cc411 Compare July 19, 2026 06:45
@codecov-commenter

codecov-commenter commented Jul 19, 2026

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 45.86%. Comparing base (10341e4) to head (81cc411).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #6604      +/-   ##
==========================================
- Coverage   45.86%   45.86%   -0.01%     
==========================================
  Files         412      412              
  Lines       48763    48763              
  Branches     6913     6913              
==========================================
- Hits        22367    22366       -1     
  Misses      24849    24849              
- Partials     1547     1548       +1     
Flag Coverage Δ
go-feature-server 30.58% <ø> (ø)
python-unit 47.16% <ø> (-0.01%) ⬇️
see 1 file with indirect coverage changes

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 10341e4...81cc411. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@ntkathole
ntkathole merged commit 70a9751 into feast-dev:master Jul 19, 2026
23 checks passed
franciscojavierarceo pushed a commit that referenced this pull request Jul 20, 2026
# [0.65.0](v0.64.0...v0.65.0) (2026-07-20)

### Bug Fixes

* add debug logging for FIPS mode detection fallback ([6c1b24e](6c1b24e))
* Build embedded UI from local source ([#6525](#6525)) ([3500349](3500349))
* Bump decommissioned Snowflake Python UDF runtime from 3.9 to 3.10 ([#6606](#6606)) ([#6608](#6608)) ([10341e4](10341e4))
* configure FIPS-compliant gRPC cipher suites for offline server ([6bc80a2](6bc80a2))
* Correct Flink PyArrow dependency constraints ([#6604](#6604)) ([70a9751](70a9751))
* Fix ValueError in signal handling for Trino worker threads ([#6428](#6428)) ([506d919](506d919))
* Fixed monitoring page issues ([7946018](7946018))
* Make pytest config compatible with newer pytest ([#5779](#5779)) ([a57ea33](a57ea33))
* Replace comma with space in DynamoDB-incompatible label tag value ([51e3a16](51e3a16))
* Resolve UI build warnings ([#6529](#6529)) ([abe92af](abe92af))
* Unblock nightly UI build ([#6570](#6570)) ([f296d4b](f296d4b))
* Use LONGBLOB for SQL registry proto columns on MySQL ([#6566](#6566)) ([7e4beb2](7e4beb2))

### Features

* Add click-to-zoom lightbox for blog post images ([#6575](#6575)) ([1cb23fd](1cb23fd))
* Add dark mode support to website and blog ([#6589](#6589)) ([7358fb8](7358fb8))
* Add OnlineStore for Aerospike ([#6532](#6532)) ([9cd35e1](9cd35e1))
* Add OpenLineage Consumer to Feast - receive, store, and visualize cross-producer lineage ([#6549](#6549)) ([a834126](a834126))
* Add registry list feature views by updated since ([#6092](#6092)) ([#6093](#6093)) ([006c606](006c606))
* Add ScyllaDB online store with vector search ([#6508](#6508)) ([1669661](1669661))
* Added compute and jobs UI ([ba2c05c](ba2c05c))
* Added Iceberg REST Catalog data source support ([e0a8573](e0a8573))
* Bring Your Own Spark - SparkApplication ([#6550](#6550)) ([dcd496f](dcd496f))
* **cassandra:** Add multi-DC support via per-datacenter execution profiles ([#6434](#6434)) ([0de9196](0de9196))
* Enhanced data source creation as a visual catalog with type-specific forms ([#6557](#6557)) ([d6acbba](d6acbba))
* Enhanced datasets UI functionality ([de11152](de11152))
* Implement RegistryServer.Proto RPC with RBAC-filtered response ([#6558](#6558)) ([#6552](#6552)) ([0d02614](0d02614))
* New zoned timestamp feature type ([#6536](#6536)) ([#6537](#6537)) ([eb042f0](eb042f0))
* **operator:** Auto-create RBAC for spark_application batch engine ([#6597](#6597)) ([f487b37](f487b37))
* **operator:** integrate cluster TLS profile for OCP 5.0 compliance ([43263a6](43263a6))
* Permissions CRUD UI and OIDC auth integration in UI ([6511da1](6511da1))
* Retrieve historical features from BigQuery without entity_df ([#6569](#6569)) ([cd5f6bb](cd5f6bb)), closes [#6558](#6558) [#6552](#6552)
* **spark:** SparkSource query+path and pre-computed offline read for BatchFeatureView ([#6440](#6440)) ([4dc8757](4dc8757))

### BREAKING CHANGES

* total_timeout_ms is renamed to batch_total_timeout_ms. Config files using the old name must be updated. No default value change.

Docs updated (reference + perf-tuning guide) with a short explainer on the per-attempt vs total deadline distinction. Two new unit tests pin the policy wiring: socket_timeout_ms propagates to all three scopes, and is omitted (not injected as None) when unset.

Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com>

* refactor(aerospike): use MAP_KEY_ORDERED, KEY_DIGEST, and instance-scoped client

Cheap-win cleanups flagged in review, all touching the same small patch of write-path and lifecycle code.

* Map CDTs are now created with MAP_KEY_ORDERED. map_get_by_key / map_remove_by_key on an ordered map are O(log N) in the map size instead of O(N); matters on reads of wide feature views and on the update() background scan (which walks every record in the project's set).

* Writes drop POLICY_KEY_SEND and rely on the client default (POLICY_KEY_DIGEST). The serialized entity key is no longer stored alongside each record, saving per-record storage the read path never consumes (batch_operate preserves request order; results are paired back by zip in online_read).

* _client moves from a class attribute to an instance attribute (set in __init__). Previously two AerospikeOnlineStore instances could share the cached client through class state until one wrote self._client. With the instance attribute the state is always per-instance from construction.

* Drop MongoDB references from class docstrings and comments (they referred to how the storage layout was derived rather than documenting current behavior). Also rewrite the _build_batch_writes docstring to describe the policies applied on the write path.

Unit test assertions for the write-path record are updated: bw.policy is now None (client default applies) and map ops carry map_policy={'map_order': MAP_KEY_ORDERED}. All three docker-backed integration tests still pass end-to-end (cross-FV upsert, update() background scan, full feature-store round-trip), so the read/write shape survives the ordering and policy changes against a real server.

Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com>

* feat(aerospike): add per-FV namespace/set overrides and prewriting hook

Adds three configuration knobs to AerospikeOnlineStoreConfig:

- namespace_overrides: pin individual feature views to a different
  Aerospike namespace (e.g. RAM-only vs. SSD-backed) without splitting
  the project across stores.
- set_overrides: place a feature view in its own set so admin ops on
  it (truncate, scan-based deletes during `feast apply`) do not touch
  records of other views.
- prewriting_hook: import-string-resolved callable invoked once per
  online_write_batch with the rows about to be written, returning the
  rows that actually go on the wire. Resolved and cached on first use;
  returning [] short-circuits the wire call.

Read, write, update and teardown paths all honour the per-FV ns/set
resolution. update() groups dropped feature views by their resolved
(ns, set) pair and issues one background scan per group. teardown()
truncates every unique (ns, set) pair the project may have written to,
including the store-level default.

Adds 22 unit tests for the new behaviour and updates 3 existing call
sites of _build_batch_writes for the new namespace= parameter. Adds a
sample hook module under examples/online_store/aerospike_overrides_and_hooks/
and corresponding sections in docs/reference/online-stores/aerospike.md.

Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com>

* test: update aerospike image tag

Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com>

* chore: sync README template and secrets baseline after master merge

Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com>

* chore: fix secrets baseline line number for v1 operator types

Adding aerospike to the feast-operator enum shifted the allowlisted
SecretRef entry in api/v1/featurestore_types.go by one line.

Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com>

* docs: update aerospike docs

Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com>

* fix(aerospike): wire batch max_retries and fix empty projection handling

Copilot review feedback on PR #6532:

- Add max_retries to the batch client policy (batch_operate/batch_write path)
- Treat empty projected feature maps as present FV slots (is not None)
- Return {} from _normalize_projected_features([]) instead of None
- Fix projection unit test mock/assertions
- Correct prewriting_hook config docstring

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com>

* style(aerospike): format online_read docs assignment for ruff

Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com>

* chore: update pixi.lock for aerospike optional extra

Regenerate the v6 lockfile with Pixi v0.63.1 after adding the aerospike extra to pyproject.toml.

Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com>

* fix(aerospike): add client init lock and batch chunking

Guard lazy client creation with a lock to avoid connection leaks under concurrent first use, and chunk batch reads/writes by batch_max_records so large materializations stay under Aerospike server batch limits.

Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com>
aniketpalu pushed a commit to aniketpalu/feast that referenced this pull request Jul 28, 2026
* fix: Correct Flink PyArrow dependency constraints

Signed-off-by: Zane Wang <zelinwang10@gmail.com>

* chore: regenerate lockfiles for pyarrow constraint change

Signed-off-by: Zane Wang <zelinwang10@gmail.com>

---------

Signed-off-by: Zane Wang <zelinwang10@gmail.com>
aniketpalu pushed a commit to aniketpalu/feast that referenced this pull request Jul 28, 2026
# [0.65.0](feast-dev/feast@v0.64.0...v0.65.0) (2026-07-20)

### Bug Fixes

* add debug logging for FIPS mode detection fallback ([6c1b24e](feast-dev@6c1b24e))
* Build embedded UI from local source ([feast-dev#6525](feast-dev#6525)) ([3500349](feast-dev@3500349))
* Bump decommissioned Snowflake Python UDF runtime from 3.9 to 3.10 ([feast-dev#6606](feast-dev#6606)) ([feast-dev#6608](feast-dev#6608)) ([10341e4](feast-dev@10341e4))
* configure FIPS-compliant gRPC cipher suites for offline server ([6bc80a2](feast-dev@6bc80a2))
* Correct Flink PyArrow dependency constraints ([feast-dev#6604](feast-dev#6604)) ([70a9751](feast-dev@70a9751))
* Fix ValueError in signal handling for Trino worker threads ([feast-dev#6428](feast-dev#6428)) ([506d919](feast-dev@506d919))
* Fixed monitoring page issues ([7946018](feast-dev@7946018))
* Make pytest config compatible with newer pytest ([feast-dev#5779](feast-dev#5779)) ([a57ea33](feast-dev@a57ea33))
* Replace comma with space in DynamoDB-incompatible label tag value ([51e3a16](feast-dev@51e3a16))
* Resolve UI build warnings ([feast-dev#6529](feast-dev#6529)) ([abe92af](feast-dev@abe92af))
* Unblock nightly UI build ([feast-dev#6570](feast-dev#6570)) ([f296d4b](feast-dev@f296d4b))
* Use LONGBLOB for SQL registry proto columns on MySQL ([feast-dev#6566](feast-dev#6566)) ([7e4beb2](feast-dev@7e4beb2))

### Features

* Add click-to-zoom lightbox for blog post images ([feast-dev#6575](feast-dev#6575)) ([1cb23fd](feast-dev@1cb23fd))
* Add dark mode support to website and blog ([feast-dev#6589](feast-dev#6589)) ([7358fb8](feast-dev@7358fb8))
* Add OnlineStore for Aerospike ([feast-dev#6532](feast-dev#6532)) ([9cd35e1](feast-dev@9cd35e1))
* Add OpenLineage Consumer to Feast - receive, store, and visualize cross-producer lineage ([feast-dev#6549](feast-dev#6549)) ([a834126](feast-dev@a834126))
* Add registry list feature views by updated since ([feast-dev#6092](feast-dev#6092)) ([feast-dev#6093](feast-dev#6093)) ([006c606](feast-dev@006c606))
* Add ScyllaDB online store with vector search ([feast-dev#6508](feast-dev#6508)) ([1669661](feast-dev@1669661))
* Added compute and jobs UI ([ba2c05c](feast-dev@ba2c05c))
* Added Iceberg REST Catalog data source support ([e0a8573](feast-dev@e0a8573))
* Bring Your Own Spark - SparkApplication ([feast-dev#6550](feast-dev#6550)) ([dcd496f](feast-dev@dcd496f))
* **cassandra:** Add multi-DC support via per-datacenter execution profiles ([feast-dev#6434](feast-dev#6434)) ([0de9196](feast-dev@0de9196))
* Enhanced data source creation as a visual catalog with type-specific forms ([feast-dev#6557](feast-dev#6557)) ([d6acbba](feast-dev@d6acbba))
* Enhanced datasets UI functionality ([de11152](feast-dev@de11152))
* Implement RegistryServer.Proto RPC with RBAC-filtered response ([feast-dev#6558](feast-dev#6558)) ([feast-dev#6552](feast-dev#6552)) ([0d02614](feast-dev@0d02614))
* New zoned timestamp feature type ([feast-dev#6536](feast-dev#6536)) ([feast-dev#6537](feast-dev#6537)) ([eb042f0](feast-dev@eb042f0))
* **operator:** Auto-create RBAC for spark_application batch engine ([feast-dev#6597](feast-dev#6597)) ([f487b37](feast-dev@f487b37))
* **operator:** integrate cluster TLS profile for OCP 5.0 compliance ([43263a6](feast-dev@43263a6))
* Permissions CRUD UI and OIDC auth integration in UI ([6511da1](feast-dev@6511da1))
* Retrieve historical features from BigQuery without entity_df ([feast-dev#6569](feast-dev#6569)) ([cd5f6bb](feast-dev@cd5f6bb)), closes [feast-dev#6558](feast-dev#6558) [feast-dev#6552](feast-dev#6552)
* **spark:** SparkSource query+path and pre-computed offline read for BatchFeatureView ([feast-dev#6440](feast-dev#6440)) ([4dc8757](feast-dev@4dc8757))

### BREAKING CHANGES

* total_timeout_ms is renamed to batch_total_timeout_ms. Config files using the old name must be updated. No default value change.

Docs updated (reference + perf-tuning guide) with a short explainer on the per-attempt vs total deadline distinction. Two new unit tests pin the policy wiring: socket_timeout_ms propagates to all three scopes, and is omitted (not injected as None) when unset.

Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com>

* refactor(aerospike): use MAP_KEY_ORDERED, KEY_DIGEST, and instance-scoped client

Cheap-win cleanups flagged in review, all touching the same small patch of write-path and lifecycle code.

* Map CDTs are now created with MAP_KEY_ORDERED. map_get_by_key / map_remove_by_key on an ordered map are O(log N) in the map size instead of O(N); matters on reads of wide feature views and on the update() background scan (which walks every record in the project's set).

* Writes drop POLICY_KEY_SEND and rely on the client default (POLICY_KEY_DIGEST). The serialized entity key is no longer stored alongside each record, saving per-record storage the read path never consumes (batch_operate preserves request order; results are paired back by zip in online_read).

* _client moves from a class attribute to an instance attribute (set in __init__). Previously two AerospikeOnlineStore instances could share the cached client through class state until one wrote self._client. With the instance attribute the state is always per-instance from construction.

* Drop MongoDB references from class docstrings and comments (they referred to how the storage layout was derived rather than documenting current behavior). Also rewrite the _build_batch_writes docstring to describe the policies applied on the write path.

Unit test assertions for the write-path record are updated: bw.policy is now None (client default applies) and map ops carry map_policy={'map_order': MAP_KEY_ORDERED}. All three docker-backed integration tests still pass end-to-end (cross-FV upsert, update() background scan, full feature-store round-trip), so the read/write shape survives the ordering and policy changes against a real server.

Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com>

* feat(aerospike): add per-FV namespace/set overrides and prewriting hook

Adds three configuration knobs to AerospikeOnlineStoreConfig:

- namespace_overrides: pin individual feature views to a different
  Aerospike namespace (e.g. RAM-only vs. SSD-backed) without splitting
  the project across stores.
- set_overrides: place a feature view in its own set so admin ops on
  it (truncate, scan-based deletes during `feast apply`) do not touch
  records of other views.
- prewriting_hook: import-string-resolved callable invoked once per
  online_write_batch with the rows about to be written, returning the
  rows that actually go on the wire. Resolved and cached on first use;
  returning [] short-circuits the wire call.

Read, write, update and teardown paths all honour the per-FV ns/set
resolution. update() groups dropped feature views by their resolved
(ns, set) pair and issues one background scan per group. teardown()
truncates every unique (ns, set) pair the project may have written to,
including the store-level default.

Adds 22 unit tests for the new behaviour and updates 3 existing call
sites of _build_batch_writes for the new namespace= parameter. Adds a
sample hook module under examples/online_store/aerospike_overrides_and_hooks/
and corresponding sections in docs/reference/online-stores/aerospike.md.

Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com>

* test: update aerospike image tag

Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com>

* chore: sync README template and secrets baseline after master merge

Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com>

* chore: fix secrets baseline line number for v1 operator types

Adding aerospike to the feast-operator enum shifted the allowlisted
SecretRef entry in api/v1/featurestore_types.go by one line.

Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com>

* docs: update aerospike docs

Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com>

* fix(aerospike): wire batch max_retries and fix empty projection handling

Copilot review feedback on PR feast-dev#6532:

- Add max_retries to the batch client policy (batch_operate/batch_write path)
- Treat empty projected feature maps as present FV slots (is not None)
- Return {} from _normalize_projected_features([]) instead of None
- Fix projection unit test mock/assertions
- Correct prewriting_hook config docstring

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com>

* style(aerospike): format online_read docs assignment for ruff

Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com>

* chore: update pixi.lock for aerospike optional extra

Regenerate the v6 lockfile with Pixi v0.63.1 after adding the aerospike extra to pyproject.toml.

Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com>

* fix(aerospike): add client init lock and batch chunking

Guard lazy client creation with a lock to avoid connection leaks under concurrent first use, and chunk batch reads/writes by batch_max_records so large materializations stay under Aerospike server batch limits.

Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

extra != does not work as expected

4 participants