Skip to content

Feature Request: Hybrid Offline Store to support multiple offline store backends simultaneously #5541

@PrasannaKumarARDS

Description

@PrasannaKumarARDS

Is your feature request related to a problem? Please describe.
Currently, Feast only allows configuring a single offline store for a project deployment. This forces users with heterogeneous data infrastructure to either use a single type of offline store for all their feature retrieval needs or create multiple separate Feast deployments for different data sources, which increases operational complexity. Organizations with data spread across multiple storage systems (e.g., some in BigQuery, some in Redshift, some in Snowflake) cannot efficiently use Feast without duplicating data or maintaining multiple deployments.

Describe the solution you'd like
A Hybrid Offline Store implementation that can route operations to different offline stores based on data source type. This would allow:

  1. Configuring multiple offline stores in a single Feast deployment
  2. Automatically routing feature retrieval operations to the appropriate offline store based on the data source
  3. Maintaining a single Feast deployment for an organization with heterogeneous data infrastructure
  4. Enabling feature views to reference data from any supported offline store without requiring data migration

The solution would introduce a new HybridOfflineStoreConfig type that contains a list of offline store configurations. Each data source would indicate which offline store it belongs to, and the HybridOfflineStore would delegate operations to the appropriate offline store implementation.

Describe alternatives you've considered

  1. Data Migration: Moving all data to a single offline store type. This is costly, time-consuming, and often not feasible due to organizational constraints.
  2. Multiple Feast Deployments: Maintaining separate Feast deployments for each offline store type. This increases operational complexity and makes it difficult to join features from different sources.
  3. Custom Data Source Implementations: Creating custom data sources that fetch from multiple stores. This would require significant custom code for each combination of data sources.

Additional context
The implementation would follow the existing patterns in Feast's offline store design, making it a natural extension of the current architecture while significantly increasing flexibility for users.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions