Skip to content

Redis High Availability #478

@woop

Description

@woop

Is your feature request related to a problem? Please describe.

Currently we do not natively support Redis in high availability mode for online serving.

  • Redis is a single point of failure
  • Redis cannot currently scale out more than one node, leading to resource utilization problems (disk, cpu).

Additionally, there is a problem with the current implementation of the Redis client in Apache Beam where it establishes too many connections to Redis.

Describe the solution you'd like
The goal is to add support for HA Redis as part of Feast. This would require

  1. Users should be able to configure the addresses to these stores as part of their serving deployment. This may necessitate an update the Redis store in the store.proto (or the creation of a new type of store like RedisCluster.
  2. Refactoring the Feast Redis IO in the Apache Beam ingestion code to support Redis clusters.
  3. Refactoring the Feast Serving client to support Redis clusters.
  4. Ensuring that the Redis clients (both for cluster and non-cluster mode) do not create an excessive amount of connections. Ideally each dataflow or beam node should be sharing a single connection pool between all bundles.
  5. EDIT: We should try to use Lettuce as the Redis Client for both serving and ingestion.
    Why is Lettuce the default Redis client used in Spring Session Redis? spring-projects/spring-session#789
    https://github.com/lettuce-io/lettuce-core

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions