Alternate DataLoader strategies #3721

ee-usgs · 2024-10-12T22:14:55Z

ee-usgs
Oct 12, 2024

Hi -

I'm wondering if there is a place to customize the DataLoader strategy. In my case, I would like to 'pipeline' the BatchLoader requests that are sent by the DataLoader to improve performance.

Ideally, I'd like to customize that in two ways:

Send smaller batch requests to reduce latency.
For instance, if it takes 1 second to fetch a batch of 100 records, I could have less latency if I could request 50 records instead.
Parallelize the batches that the DataLoader requests
It appears that currently the DataLoader determines that it is ready to request a batch of records from the BatchLoader and then it waits for the response. Instead, it would be more performant to keep processing to find additional records that are needed and eventually submit a BatchLoad request for those records as well, potentially before the previous one completes.

Is that possible with the current DataLoader logic?

Thank you!

Answered by bbakerman

Oct 14, 2024

For number one - see org.dataloader.DataLoaderOptions#maxBatchSize. This allows you to specify how big or small the batch size is.

For number 2

It appears that currently the DataLoader determines that it is ready to request a batch of records from the BatchLoader and then it waits for the response.

I dont think this is true.

        int maxBatchSize = loaderOptions.maxBatchSize();
        CompletableFuture<List<V>> futureList;
        if (maxBatchSize > 0 && maxBatchSize < keys.size()) {
            futureList = sliceIntoBatchesOfBatches(keys, queuedFutures, callContexts, maxBatchSize);

The code will slice the keys into the batch sized set of "batch loads". The values back here are Com…

View full answer

bbakerman · 2024-10-14T10:07:57Z

bbakerman
Oct 14, 2024
Collaborator

For number one - see org.dataloader.DataLoaderOptions#maxBatchSize. This allows you to specify how big or small the batch size is.

For number 2

It appears that currently the DataLoader determines that it is ready to request a batch of records from the BatchLoader and then it waits for the response.

I dont think this is true.

        int maxBatchSize = loaderOptions.maxBatchSize();
        CompletableFuture<List<V>> futureList;
        if (maxBatchSize > 0 && maxBatchSize < keys.size()) {
            futureList = sliceIntoBatchesOfBatches(keys, queuedFutures, callContexts, maxBatchSize);

The code will slice the keys into the batch sized set of "batch loads". The values back here are CompletableFutures and hence the parallelism is actually up to you.

It will "async combine" back that list of futures so that N keys comes back to a combined promise of N values but the sub batches can happen in parallel and hence sub batches can happen at the same time. All of this "combining" is completely async - there is no "waiting"

Perhaps this is not what you mean so if you could expand on your example more perhaps

0 replies

ee-usgs · 2024-10-21T21:13:19Z

ee-usgs
Oct 21, 2024
Author

Thank you for the response. I'm doing some testing on my system to see if that matches the behavior I'm seeing.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Alternate DataLoader strategies #3721

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Alternate DataLoader strategies #3721

Uh oh!

ee-usgs Oct 12, 2024

Replies: 2 comments

Uh oh!

bbakerman Oct 14, 2024 Collaborator

Uh oh!

ee-usgs Oct 21, 2024 Author

ee-usgs
Oct 12, 2024

bbakerman
Oct 14, 2024
Collaborator

ee-usgs
Oct 21, 2024
Author