Skip to content

Remote materialization #4526

@tokoko

Description

@tokoko

Is your feature request related to a problem? Please describe.
We already have an option to run online/offline store queries remotely through feature server and offline server, respectively. This way rbac rules will be applied on operations. One piece that's missing is materialization. There are several ways to do this:

  • Keep materialization local, but rely on remote online/offline engines to apply rbac rules. This is currently impossible because remote online client doesn't implement online_write_batch method. Even if we did implement it in the feature server itself, we would essentially be using a lightweight fastapi server to transport batches and batches of potentially huge datasets.

  • Create remote materialization engine that will defer the whole materialization call to a backend server and apply rbac rules there. We can create another server component MaterializationServer that will receive these requests.

  • The same as above but instead of creating a new component, we can reuse OfflineServer to do the request handling. This is slightly awkward from the naming perspective, but probably makes the most sense in term of usage/maintenance.

I'd probably go with option 3 as a starting point.

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/featureNew feature or requestwontfixThis will not be worked on

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions