Skip to content

Control mini-batch size when writing rows to online store from bytewax job #3776

@sudohainguyen

Description

@sudohainguyen

Is your feature request related to a problem? Please describe.
Bytewax Materialization job tries to write all rows at a time from a single pyarrow parquet dataset

Describe the solution you'd like
Should break into mini-batches before writing to online store, and this number should be configurable

Describe alternatives you've considered
No

Additional context
image

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions