Supporting multiple online feature stores #2383
Description
Is your feature request related to a problem? Please describe.
We are seeing that ML feature teams want to own their feature projections and not rely on a single global online feature store with the most common reason being reduction of blast radius.
Describe the solution you'd like
A way to create projections of a subset of feature views to specific online stores.
Describe alternatives you've considered
I was pointed towards attempting to implement the requested feature using a custom provider which I managed to get working, albeit rather hacky. I've only focused on fanning out the writing of features.
class ManyOnlineStoresPassthroughConfig(RepoConfig):
online_stores: List[dict]
class ManyOnlineStoresPassthroughProvider(PassthroughProvider):
def online_write_batch(self, config: ManyOnlineStoresPassthroughConfig, table: FeatureView, *args, **kwargs) -> None:
"""
Writes feature view to all subscribers
"""
feature_view_subscribers = [store for store in config.online_stores if table.name in store["view_subscriptions"]]
for store in feature_view_subscribers:
print(f"{table.name} online write to subscriber {store['name']}")
config.online_store = get_online_config_from_type(store["type"])(**store)
self.online_store = get_online_store_from_config(config.online_store)
return super().online_write_batch(config, table, *args, **kwargs)With the config update of
project: aaaa
registry: gs://temp-feast-dev/dev/registry.db
provider: providers.custom_provider.ManyOnlineStoresPassthroughProvider
offline_store:
type: bigquery
dataset: temp_feast-dev
project_id: abc
location: EU
online_stores:
- name: "lovely"
type: redis
redis_type: redis
connection_string: "blah"
view_subscriptions: ["feature_view_1"]
- name: "lovely"
type: redis
redis_type: redis
connection_string: "abc"
view_subscriptions: ["feature_view_1", "feature_view_2"]There's still some unknowns around retrieving the online features as we'd need a way of knowing what projection should be used.
Happy to help wherever I can!
Metadata
Metadata
Assignees
Type
Projects
Status