Skip to content

PostgresOnlineStore: Improve materialization #4309

@job-almekinders

Description

@job-almekinders

Is your feature request related to a problem? Please describe.
The current implementation of PostgreSQLOnlineStoreConfig.online_write_batch() is quite slow.

Describe the solution you'd like
One of the suggested options discussed in this issue is using bulk writes with adbc.

Describe alternatives you've considered
Another option could be to use COPY FROM command. This performance benchmark shows that the number of rows written per second is significantly larger compared to the batch insert we are doing now.

This image shows the comparison from the performance benchmark:
Screenshot 2024-06-24 at 11 48 25

Additional context
I think we should benchmark both solutions and pick the one that achieves the best performance.

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/featureNew feature or requestwontfixThis will not be worked on

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions