Is your feature request related to a problem? Please describe.
As of now, the Redis Value size is directly proportional to the length of the feature name strings. This becomes more significant when there are several features. As a result the memory foot print can be significantly higher than the equivalent data stored in CSV / parquet / avro.
Describe the solution you'd like
Instead of writing the byte representation of Feature Row, which has a list of Fields, it might be suffice if we simply store the list of Values instead. Feast Online Serving is already aware of the FeatureSetSpec, and hence the field names. This information can be used to reconstruct the Feature Row from the list of Values. Without storing the feature names, the Redis value size can be drastically reduced.
Describe alternatives you've considered
Another alternative would be, instead of storing the feature names in the Redis, we can store the hash code of the feature name instead. Again, the corresponding Feature Row can be reconstructed on the Feast online serving side.
Last alternative would be to apply some compression algorithm (eg Gzip/Bzip) on the byte array representation of Feature Row, though it is unlikely to surpass the above alternatives in terms of efficiency. There will also be impact in terms of latency due to additional processing required during retrieval.
Additional context
Any approach to resolve this issue will likely resulted in backward incompatible changes: the existing Redis value will not longer be interpretable once the way we stored Redis value changes. Therefore, we might have to include a Store level configuration, to toggle this feature on only when the user is ready to migrate.
Is your feature request related to a problem? Please describe.
As of now, the Redis Value size is directly proportional to the length of the feature name strings. This becomes more significant when there are several features. As a result the memory foot print can be significantly higher than the equivalent data stored in CSV / parquet / avro.
Describe the solution you'd like
Instead of writing the byte representation of Feature Row, which has a list of Fields, it might be suffice if we simply store the list of Values instead. Feast Online Serving is already aware of the FeatureSetSpec, and hence the field names. This information can be used to reconstruct the Feature Row from the list of Values. Without storing the feature names, the Redis value size can be drastically reduced.
Describe alternatives you've considered
Another alternative would be, instead of storing the feature names in the Redis, we can store the hash code of the feature name instead. Again, the corresponding Feature Row can be reconstructed on the Feast online serving side.
Last alternative would be to apply some compression algorithm (eg Gzip/Bzip) on the byte array representation of Feature Row, though it is unlikely to surpass the above alternatives in terms of efficiency. There will also be impact in terms of latency due to additional processing required during retrieval.
Additional context
Any approach to resolve this issue will likely resulted in backward incompatible changes: the existing Redis value will not longer be interpretable once the way we stored Redis value changes. Therefore, we might have to include a Store level configuration, to toggle this feature on only when the user is ready to migrate.