Skip to content

[dbsp] Use ingest rate across all levels to estimate fuel in AsyncMerger #5825

@ryzhyk

Description

@ryzhyk

Today we use arrival rate at level 0 of the LSM tree to estimate the rate of data arrival in a spine. This doesn't take into account batches that enter the spine at levels >=1. This can lead to poor merger performance.

See discussion here: #5824

Metadata

Metadata

Assignees

Labels

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions