-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Description
Is your feature request related to a problem? Please describe.
Currently, Feast only allows configuring a single offline store for a project deployment. This forces users with heterogeneous data infrastructure to either use a single type of offline store for all their feature retrieval needs or create multiple separate Feast deployments for different data sources, which increases operational complexity. Organizations with data spread across multiple storage systems (e.g., some in BigQuery, some in Redshift, some in Snowflake) cannot efficiently use Feast without duplicating data or maintaining multiple deployments.
Describe the solution you'd like
A Hybrid Offline Store implementation that can route operations to different offline stores based on data source type. This would allow:
- Configuring multiple offline stores in a single Feast deployment
- Automatically routing feature retrieval operations to the appropriate offline store based on the data source
- Maintaining a single Feast deployment for an organization with heterogeneous data infrastructure
- Enabling feature views to reference data from any supported offline store without requiring data migration
The solution would introduce a new HybridOfflineStoreConfig type that contains a list of offline store configurations. Each data source would indicate which offline store it belongs to, and the HybridOfflineStore would delegate operations to the appropriate offline store implementation.
Describe alternatives you've considered
- Data Migration: Moving all data to a single offline store type. This is costly, time-consuming, and often not feasible due to organizational constraints.
- Multiple Feast Deployments: Maintaining separate Feast deployments for each offline store type. This increases operational complexity and makes it difficult to join features from different sources.
- Custom Data Source Implementations: Creating custom data sources that fetch from multiple stores. This would require significant custom code for each combination of data sources.
Additional context
The implementation would follow the existing patterns in Feast's offline store design, making it a natural extension of the current architecture while significantly increasing flexibility for users.