This issue discusses common issues users face when materializing features to the online store in Feast.
User problems
Generally, users with large datasets can face issues on reliably loading data into the online store to meet their online needs.
1. Materialization in the default provider is not scalable
As per #2071,
Currently, the materialization process loads all the data from the Offline Store to an Arrow table, then converts all the data to Protobuf, then writes all the data to the Online Store. This process requires holding the entire dataset in memory which is not practical.
2. Materialization can be slow
For users that aren't working with a small number of feature views or large number of unique entities, Feast's python based materialization works fine. However, this does not hold true for many users.
The default provider is slow to materialize data. Users can report multiple hours to do incremental materialization, or worse materialization never completes.
Users have had to build custom providers to solve this (e.g. by kicking off Dataflow or Spark jobs to more quickly materialize large amounts of data)
3. Materialization not always reliable
There are several datastore specific issues such as #2027 and #2323, where batch write transactions can time out:
File "/usr/local/lib/python3.7/dist-packages/google/api_core/grpc_helpers.py", line 69, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "", line 3, in raise_from
google.api_core.exceptions.InvalidArgument: 400 The referenced transaction has expired or is no longer valid.
In datastore, there are also contention errors (#1575):
Materializing 1 feature views from 2021-04-29 21:19:00-07:00 to 2021-04-29 21:19:05-07:00 into the datastore online store.
my_fv:
8%|████▋ | 2720/33120 [00:07<01:18, 388.10it/s]
google.api_core.exceptions.Aborted: 409 too much contention on these datastore entities. please try again. entity groups:
Describe the solution you'd like
A clear and concise description of what you want to happen.
There are multiple ways of addressing this. Some ideas
This issue discusses common issues users face when materializing features to the online store in Feast.
User problems
Generally, users with large datasets can face issues on reliably loading data into the online store to meet their online needs.
1. Materialization in the default provider is not scalable
As per #2071,
2. Materialization can be slow
For users that aren't working with a small number of feature views or large number of unique entities, Feast's python based materialization works fine. However, this does not hold true for many users.
The default provider is slow to materialize data. Users can report multiple hours to do incremental materialization, or worse materialization never completes.
Users have had to build custom providers to solve this (e.g. by kicking off Dataflow or Spark jobs to more quickly materialize large amounts of data)
3. Materialization not always reliable
There are several datastore specific issues such as #2027 and #2323, where batch write transactions can time out:
In datastore, there are also contention errors (#1575):
Describe the solution you'd like
A clear and concise description of what you want to happen.
There are multiple ways of addressing this. Some ideas