Skip to content

TTL for entities in the Online stores other Than Redis #1988

@vas28r13

Description

@vas28r13

Is your feature request related to a problem? Please describe.
(for context: we are using the Redis online store)
At the moment, once an entity is written to the Online store, it's there until it's manually cleaned up.

In most of our business cases, the Entity will have a natural expiration time (when it's no longer relevant for the use case).

It would be nice to be able to expire the entity in the online store, with some kind of TTL setting.

Describe the solution you'd like

  1. in our local setup we end up expiring the key bin in Redis (combination of entity + project)
def online_write_batch(...):
  redis_key_bin = _redis_key(project, entity_key)
  ...
  if table.ttl:
      redis_client_pipeline.expire(name=redis_key_bin, time=table.ttl)

the issue with this is that the ttl param in feature_view/table is overloaded (initially designed for a use case in the offline store)
Also, in this type of setup and use case, the TTL should be on the entity level not feature_view because the Redis key bin is based on entity (feature_view ttl is designed for something else also)

so the proposal:
add online_store_ttl param to the Entity object
example:

driver_entity = Entity(
 name="driver",
 description="Driver Identifier", 
 value_type=ValueType.INT64, 
 online_store_ttl=Duration(seconds=86400)
)
def online_write_batch(...):
  redis_key_bin = _redis_key(project, entity_key)
  ...
  if entity.online_store_ttl:
      redis_client_pipeline.expire(name=redis_key_bin, time=entity.online_store_ttl)

Note: implementation details may vary a bit based on the Online store type

Metadata

Metadata

Assignees

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions