Skip to content

pyspark dependency breaks Snowflake offline store #3383

@beauchbum

Description

@beauchbum

This commit introduced a pyspark dependency for the Snowflake offline store that is not reflected in setup.py. This causes feast actions to fail due to:

File "/usr/local/lib/python3.9/site-packages/feast/infra/offline_stores/snowflake.py", line 26, in <module>
  from pyspark.sql import DataFrame, SparkSession
ModuleNotFoundError: No module named 'pyspark'`.

Steps to reproduce

  • Install feast = {extras = ["snowflake"], version = "^0.27.0"}
  • Set up a Feature Repository with a Snowflake offline store
  • Run feast plan

Specifications

  • Version: 0.27.0
  • Platform:
  • Subsystem:

Possible Solution

Not sure - would prefer not needing pyspark for Snowflake offline store to work, but if that's the case then add to setup.py?

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions