fix(spark): Replace mapInArrow with foreachPartition and bound write memory#6441

Open

abhijeet-dhumal wants to merge 8 commits into

feast-dev:masterfrom

abhijeet-dhumal:fix/spark-foreachpartition-materialization

Commits on Jun 1, 2026

fix(spark): replace mapInArrow with foreachPartition and fix session config forwarding for vector store materialization
abhijeet-dhumal
committed
fix: resolve linting and ruff check issues
abhijeet-dhumal
committed
fix: chunk foreachPartition writes to bound Python memory — prevents MemoryError/OOMKill on large feature views
abhijeet-dhumal
committed
style: ruff format utils.py
abhijeet-dhumal
committed
fix: correct pandas_udf returnType annotation and remove tests for deleted _apply_bfv_transformations
abhijeet-dhumal
committed
ci: retrigger CI
abhijeet-dhumal
committed
fix: restore _apply_bfv_transformations in SparkOfflineStore.get_historical_features
abhijeet-dhumal
committed
fix(spark): address review feedback on foreachPartition PR

abhijeet-dhumal
and
cursoragent
committed