Mass batchGetItems from Dynamo

Question

So I currently have tens of thousands of items I need to get from dynamo on a call. In my testing, with just 16K+ items, it takes around 750-1000ms for dynamo to return all the data. Not terrible, but I know they are going to be looking for more, and that is a long time for a customer event driven query.

I am running parallel tasks on the batches of 100 for batchGetItem.

        IEnumerable<string> ids;
        
        using (var client = GetDynamoDbClient())
        {
            DynamoDBContextConfig config = new DynamoDBContextConfig
            {
                TableNamePrefix = "test",
                Conversion = DynamoDBEntryConversion.V2,
                ConsistentRead = false,
            };
            
            DynamoDBContext context = new DynamoDBContext(client, config);
            var IdsBatchedLists = Split(ids.ToList(), BatchRequestSizeLimit); //100 items per batch
            var batchTasks = IdsBatchedLists.Select(async list =>
            {
                var batchRequest = context.CreateBatchGet<MyTable>();
                list.ForEach(x => batchRequest.AddKey(x));

                    await batchRequest.ExecuteAsync().ConfigureAwait(false);
                    return batchRequest.Results;
            });

            var results = await Task.WhenAll(batchTasks).ConfigureAwait(false);
        }

I think I have done everything I can to make it as fast as possible (I have tried every configuration I can think of for AmazonDynamoDBConfig).

I think Dynamo might be suffering some kind of query fatigue. In my traces, I can see large amounts of simultaneous queries, and then I will see one or two, then a group again. Image is from DataDog showing all the dynamo hits in a specific trace.

Overall, Has anyone tried this throughput level before, and is there anything I could be missing to do this in parallel faster?

Thanks

16k records in a second comes out to about .06ms / record. That's really very good. I'd look at why you need 16k records for a your event as that seems huge. — stdunbar
– stdunbar, Commented May 12 at 16:18
You may have to look at something like DAX to speed things up then. I've not used this but, in general, it sounds like you need an in memory cache. — stdunbar
– stdunbar, Commented May 12 at 16:24
Do you have the available compute to do that parallelism. If you're not being throttled by DynamoDB, I'd suspect your slowness is caused by queuing threads. — Leeroy Hannigan
– Leeroy Hannigan, Commented May 12 at 17:19
Yeah, the system has enough juice for it. Any limiting on the parallelization just slows the process down more. — Limey
– Limey, Commented May 12 at 17:58

Charles · Accepted Answer · 2025-05-13 14:51:27Z

2

I'd agree that retrieving 16K items at once from DDB isn't a good use case.

But as far as the behavior you are seeing, I suspect the issue is that you're not actually running in parallel.

Try using Parallel.ForEachAsync to spread out the tasks among multiple threads. By default 1 per CPU, but controllable with

ParallelOptions parallelOptions = new()
{
    MaxDegreeOfParallelism = 3
};

you should end up with something like so:

await Parallel.ForEachAsync(batchRequestList, parallelOptions, async (batchRequest) =>
{
    var response = await batchRequest.ExecuteAsync().ConfigureAwait(false);
    //do something to store the response 
    // note that multiple threads will be involved, 
    // so look at System.Collections.Concurrent 
    // maybe ConcurrentBag?
}
// Should have all the results at this point

answered May 13 at 14:51

Charles

24.2k3 gold badges23 silver badges50 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Limey May 20 at 18:50

What about GSIs? If the 16K items had the same one, would it be faster?

Charles May 20 at 19:37

Don't think it'd be faster, possibly a bit easier as DDB Scan has built in support for parallelization

Limey May 21 at 17:38

I gave it a shot, and yeah, its actually slower since its sequential vs in parallel. I don't want to do any of the synthetic sharding that would allowing for pulling in parallel.

Collectives™ on Stack Overflow

Mass batchGetItems from Dynamo

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related