Batching trigger Lambda on DynamoDB

Question

I'm having a table as below

Key                                 | Value
---------------------------------------------------
Client_123_UNIQUE_ID                | s3://abc.txt
Client_123_UNIQUE_ID                | s3://xyz.txt
Client_456_UNIQUE_ID                | s3://qaz.txt              
Client_456_UNIQUE_ID                | s3://qwe.txt
Client_789_UNIQUE_ID                | s3://asd.txt
Client_789_UNIQUE_ID                | s3://zxc.txt

The data will be inserted consistently to this table from a AWS Lambda function. (maybe millions items)

I have an use case which I need to have a trigger when there is 100 items available in the table to perform some batch processing. In other mean, as soon as we have 100 new items created in this table, I would like to have a trigger to Lambda function to perform a batch processing on 100 items.

When I research, it seems DynamoDB Stream can support Batch but I'm not pretty clear based on documentation.

Lambda reads records in batches and invokes your function to process records from the batch.

Lambda polls shards in your DynamoDB Streams stream for records at a base rate of 4 times per second. When records are available, Lambda invokes your function and waits for the result. If processing succeeds, Lambda resumes polling until it receives more records.

If your function returns an error, Lambda retries the batch until processing succeeds or the data expires. Until the issue is resolved, no data in the shard is processed. Handle any record processing errors in your code to avoid stalled shards and potential data loss.

Could you please help me to clarify documentation or advice me the approach which we use DynamoDB Stream is correct one for this use case?

If I'm explaining my question is not good enough, please put comments so I can clarify more.

were you able to solve this? if yes, how? Thanks!

sam
– sam

2021-01-23 19:38:05 +00:00
Commented Jan 23, 2021 at 19:38 — sam
– sam, Commented Jan 23, 2021 at 19:38
Yes , I did it already

Nghia Do
– Nghia Do

2021-01-24 16:55:29 +00:00
Commented Jan 24, 2021 at 16:55 — Nghia Do
– Nghia Do, Commented Jan 24, 2021 at 16:55

Milan Cermak · Accepted Answer · 2022-02-15 17:22:26Z

2

You can set the BatchSize when declaring the mapping between the stream and Lambda. The max size is 10000 items.

edited Feb 15, 2022 at 17:22

answered Mar 2, 2019 at 17:10

Milan Cermak

8,1043 gold badges48 silver badges60 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Nghia Do Over a year ago

Suppose, I set the batch size is 100 and then if there is no enough 100 items, is the lambda receiving the trigger with records?

Milan Cermak Over a year ago

Yes. It's the maximum size. The Lambda function can be invoked also with less than the full batch size.

Nghia Do Over a year ago

I would like to understand more about batch, suppose im having 100 as batch size and there is 140 items inserted to dynamo db, how many times lambda will get triggered?

Nghia Do Over a year ago

I have tried by myself, the batch number varies alot

Batman Over a year ago

@NghiaDo In that case, the number of batch invocations would be 2. First batch of 100 items and second batch of 40 items. Later 40 items are sent as batch to Lambda assuming timeout of batch occures.

|

Collectives™ on Stack Overflow

Batching trigger Lambda on DynamoDB

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related