404 questions
0
votes
1
answer
38
views
Counting Items and Handling Duplicates in DynamoDB Without Full Data Traversal in Node.js
The problem involves efficiently counting the total number of items in an Amazon DynamoDB table while handling duplicates, without traversing each item or returning the entire dataset. This requires a ...
-1
votes
1
answer
91
views
What guarantee does dynamoDB's strongly consistent read give?
I am aware that DynamoDB has a strongly consistent read option. See Consistency Model.
It states:
A strongly consistent read in Amazon DynamoDB returns a result that reflects all writes that received ...
- 4,948
0
votes
1
answer
82
views
How to create GSI based on timestamp? [closed]
I'm working with a DynamoDB table which stores records for a maximum of 24 hours. We're talking about tens or even hundreds of millions of records. Most of the records are removed minutes after being ...
- 501
0
votes
2
answers
339
views
Do DynamoDB Streams using Lambda Triggers guarantee no duplicates?
I'm looking into the best approach for having a stream from our Dynamo tables and there is some conflicting answers/documentation over whether using the Dynamo Stream guarantees no duplicates.
The ...
1
vote
0
answers
128
views
EventBridge Pipes with DynamoDB Stream - losing payload during target input transform
I have set up an EventBridge pipe that takes the latest insert or modify event on a DynamoDB instance's Stream, formats the event's payload to remove DynamoDB's data type notation, and pass a JSON ...
- 663
0
votes
0
answers
73
views
Are AWS Lambda functions automatically queued in a way that's aware of concurrency limit?
Suppose I have AWS Lambda functions X and Y, and X immediately issues 5000 asynchronous calls to Y before exiting. The default concurrency limit is 1000, so only 1000 instances of Y will immediately ...
- 34.2k
-1
votes
1
answer
277
views
What will happen if I insert a record into DynamoDB with the same primary key as an existing record that has an expired TTL?
I currently have a record with the primary key 389 in DynamoDB, but its TTL has expired and it hasn't been deleted yet.
If I attempt to insert a record with the same primary key but with an updated ...
0
votes
1
answer
115
views
What happens to DynamoDB Stream when a table is dropped and re-deployed?
I have an application reading DynamoDB streams (Not Kinesis). The following series of events happens:
1.) A row in table A is deleted. Presumably, this is captured in the changestream.
2.) Before my ...
- 37
1
vote
0
answers
128
views
AWS KCL worker on shutdown populates the lease table with more leases then shards exist
Currently I am in the middle of building a dynamodb streaming service using AWS KCL library specifically the dynamodb streams adapter (version 1.6.0), in my initial testing implementation I am seeing ...
0
votes
1
answer
102
views
Serverless Framework with TypeScript: Lambda Processing All DynamoDB Stream Events Despite Filters
I am using the Serverless Framework with TypeScript to process DynamoDB stream events and then publish them to an SNS topic. I have configured my Lambda function to use filters to process only ...
2
votes
1
answer
75
views
How to efficiently paginate large nested lists in DynamoDB?
I'm working on a chat application where each chat session is stored as an item in a DynamoDB table. The structure of a chat item looks like this:
{
"chatId": {
"S": "...
- 119
0
votes
1
answer
256
views
To subscribe to the dynamodb global table stream in regional aws lambdas?
Here is my regional lambda sdk configuration.
The problem is tableStreamArn it seems:
const globalTableTableName = "my-org-glb-table1"
const regionalTable = Table.fromTableAttributes(...
- 13.4k
0
votes
1
answer
68
views
AttributeError: 'DynamoHandler' object has no attribute 'export_table_to_point_in_time'
I have a function that exports dynamo data using export_table_to_point_in_time from boto3 in version 1.33.13
client = boto3.client("dynamodb")
response = client....
- 1
0
votes
0
answers
28
views
DDB table not updating all items
I am running the below python code from Linux terminal which would run to insert the "Time" and "Message" of 5K items in the dummy_table. First few executions inserted all the ...
- 1
0
votes
0
answers
64
views
Unable to insert items into DDB table if the table has 30-40k items
The purpose of the below code is to retrieves data from firebase and post duplicate check & split it would append the items to buffer. Once the buffer limit has reached 4 its would insert the ...
- 1
0
votes
1
answer
183
views
duplicate overlapping events for dynamodb lambda streams
Will a Lambda connected to a DynamoDB stream ever send 2 batches that have a different set of records in them but one or more records in common?
e.g. Let's say we have a DynamoDB table with a ...
- 181
0
votes
0
answers
85
views
Dynamodb Timestamp Range query with steps (It's possible??)
Well, my query is simple but I haven't seen (nor do I know if it exists) anything that does what I want.
I know how to make a Timestamp Query where I get a range of ordered data, the range would be ...
- 51
0
votes
1
answer
129
views
DynamoDB: Count aggregate group by PK
I understand that this is not possible in DynamoDB.
DynamoDB Streams seem to be an option, but...
My struggle involves being able to create aggregates over a specific period of time. I want to be able ...
- 49
0
votes
1
answer
887
views
AWS stream and filter patterns - When value is not null
I have a dynamodb and when one record is deleted, a lambda is called. My question is pretty simple as I'm interested only to those records that have a parameter different from null and it's weird but ...
- 382
0
votes
1
answer
322
views
Enable a DynamoDB Stream via yaml
I have a DynamoDB table and I need to enable a stream without lose the table data.
I can do it via web console, but I need to do using a yaml template and a cloud formation stack.
Is it possible ...
0
votes
1
answer
4k
views
DynamoDB Time to Live (TTL) not working, despite it is active
UPDATE: Ok, it "kind works", it took more than ~10 minutes to call the lambda. For anyone who is reading this question I do not recommend using this feature (TTL & Stream), it is so ...
- 1,049
1
vote
0
answers
86
views
Is it possible to use dyanmodb streams with encrypted data?
I am doing some investigation on whether it is possible to use DynamoDB streams with encrypted data using the latest DynamoDB sdk updates i.e. can DynamoDB streams decrypt data that was encrypted ...
- 2,339
0
votes
1
answer
851
views
How to process Dynamo DB stream updates in order across shards?
I want to process a DynamoDB stream to make updates into a different DynamoDB in another account. The schemas are different so I will be transforming the data in between as well.
I have thought of the ...
- 579
2
votes
0
answers
872
views
Eventbridge Pipes not working with DynamoDB streams source and event bus destination
I am trying to write a terraform script that creates an eventbridge pipe that reads from a dynamodb stream and directly puts it in an event bus. The end goal is to be able to invoke a rule. However, ...
- 80
0
votes
1
answer
225
views
How to add FilterCriteria to a DynamoDB Stream CloudFormation template created by AWS Amplify?
Using the AWS Amplify CLI to create a Lambda trigger for a DynamoDB Table Stream. The documentation provided by Amazon is working as expected. However, I'm wanting to add filter criteria to the stream ...
- 5,828
0
votes
1
answer
874
views
DynamoDB stream's tumbling window - Will there be only a single Lambda function used for this?
For the tumbling window concept:
This new feature introduces the concept of a tumbling window, which is a fixed-size, non-overlapping time interval of up to 15 minutes. To use this, you specify a ...
- 4,961
0
votes
1
answer
276
views
Can I create the spill bucket for dynamodb Athena data connector after it has been deleted by mistake?
I was following this document while setting the dynamoDB connector, but recently was facing some issues with the data not being displayed so I checked everything and couldnt find the spill bucket in ...
- 25
0
votes
1
answer
176
views
Lambda not discarding records from DynamoDB Stream Table if retry attempts is exceed the limit
I have a Lambda that attached to the DynamoDB stream. I configured retry attempts upto 5. I have SQS as on failure destination. In some cases lambda is timing out and It retried upto 5 times. But ...
- 712
0
votes
1
answer
357
views
Updating hash key or sort key in dynamodb table creates 2 events in dynamo streams
I am using dynamodb streams to capture changes in the table. I have set the trigger to a lambda function. Insert and delete are working as expected but while updating specifically the hash key or sort ...
1
vote
1
answer
394
views
Poll dynamodb stream nextShardIterator is never null resulting in an infinite loop
I am trying to consume a dynamodb stream and would like to understand the best way to consume it.
I run a periodic job every 2s that polls the stream. It then lifts all the new records from the stream ...
- 8,471
0
votes
2
answers
562
views
Lambda event filtering for DDB trigger
My usecase is to filter events from DDB with filter condition present on both OldImage AND NewImage.
Filter that I have added -
{
"dynamodb": {
"NewImage": {
"...
0
votes
1
answer
297
views
ElastiCache prepopulate when node starts
I've been taking a look at this AWS tutorial on building a Geospatial app on AWS. It's basically using DynamoDB to store the data, and has a Lambda function set up to write data onto Redis whenever an ...
- 628
0
votes
2
answers
667
views
What happens to DynamoDB stream when we convert regional table to global table
I am trying to convert my existing DynamoDB table to a global DynamoDB table.
In the existing DynamoDB table structure, we have configured a DynamoDB Stream that triggers a Lambda function when there ...
- 3
0
votes
1
answer
1k
views
DynamoDB Streams processing - multiple consumer receiving the same stream
I am working on a workflow app comprised mostly of microservices and batch jobs. The planned setup is this: We have a java based microservice consuming a particular DynamoDB stream via the Amazon ...
-1
votes
1
answer
462
views
Disabling DynamoDB stream leads to inconsistency between tfstate file and tf config files
I am seeing an issue with dynamodb stream where if "stream_enabled" is set to "false" but the "stream_view_type" is set to a variable, in this case it is set to "...
- 45
0
votes
0
answers
190
views
How to configure KCL worker properties for Amazon KCL client
We have a java based KCL worker consuming records from DynamoDB streams. I want to understand based on high number of partition in table and shards in streams, what values should be given while ...
1
vote
1
answer
1k
views
Dynamodb streams: small number of items per batch
I have a very large dynamodb table, and I want to use lambda function triggered by a stream. I would like to work in big batches, of at least 1000 items. But when I connect the lambda, I see it is ...
- 16.1k
1
vote
1
answer
425
views
DynamoDBEvent produces attribute value JSON
I'm writing a lambda that takes in streamed data from a DynamoDB table. After parsing out the proper record, I'm trying to convert it to JSON. Currently, I'm doing this:
func LambdaHandler(ctx context....
- 8,202
0
votes
2
answers
394
views
Hold DynamoDB Stream Events in Lambda for a Small Period
We're planning to add a few updates in Lambda Function which has been set as the trigger for DynamoDB New item Events
DynamoDB --> DynamoDB Streams --> AWS Lambda
So while performing an update, ...
- 1,263
0
votes
1
answer
540
views
How to have UI subscribe to DynamoDB events?
I have an EC2 service that writes data to a DynamoDB table. I also have an app that is totally unrelated to the service. Now I want the app to display components in its UI based off the status ...
- 99
0
votes
3
answers
1k
views
Choosing between DynamoDB Streams vs. Kinesis Streams for IoT Sensor data
I have a fleet of 250 Wifi-enabled IoT sensors streaming weight data. Each devices samples once per second. I am requesting help between choosing AWS DynamoDB Streams vs. AWS Kinesis Streams to to ...
0
votes
2
answers
487
views
Transfer Data from AWS Time Stream to DynamoDB
i am working in the IoT Space with 2 Databases. AWS Time Stream & AWS DynamoDB.
My sensor data is coming into Time Stream via AWS IoT Core and MQTT. I set up a rule, that gives permission to ...
- 131
2
votes
1
answer
3k
views
EventBridge Pipes Target Input Transformer - Formatting lists/arrays
I am using EventBridge Pipes to connect DynamoDB Streams directly to EventBridge. However, I am having difficulty transforming DynamoDB list attributes to my desired format using the Pipes Target ...
- 5,767
-1
votes
1
answer
202
views
DynamoDB Trigger Lambda Function Call Failed
I am trying to have events in a DynamoDB table trigger Lambda function that moves the events into Kinesis Data Firehose. Kinesis then batches the files and send them to an S3 bucket. The Lambda ...
0
votes
0
answers
406
views
Can DynamoDb streams eventId be used as primary key?
I have a usecase to process dynamodb streams and ingest the data in Redshift for data warehousing and analytics. I want to ensure that there exists only one record per dynamodb update and there are no ...
2
votes
3
answers
3k
views
Lambda event filtering for DynamoDB trigger
Here is a modified version of an Event type I am receiving in my handler for a lambda function with a DynamoDB someTableName table trigger that I logged using cargo lambda.
Event {
records: [
...
- 383
2
votes
1
answer
1k
views
Will inserting a duplicate item in DynamoDB trigger Lambda or send a Change Data Capture CDC event in DynamoDB streams?
If I'm writing an item that is exactly the same as another item in DynamoDB, will it trigger Lambda or send a Change Data Capture (CDC) event to DynamoDB streams?
I know that it will overwrite the ...
- 21
1
vote
1
answer
958
views
How to choose between DynamoDB streams Kinesis adapter and Lambda triggers
The AWS Docs show that DynamoDB streams can be consumed in 2 ways:
DynamoDB Streams Kinesis adapter
Lambda triggers
The docs say that "the Amazon Kinesis Adapter is the recommended way to ...
- 1,488
4
votes
1
answer
1k
views
In dynamodb, what is the difference between using dynamo streams + lambda trigger versus kinesis streams + lambda trigger?
I am looking into dynamo db streaming options. There are 2:
Dynamo streams
Kinesis streams
Dynamo streams can have a lambda trigger that will consume the stream.
Kinesis streams can also have the ...
- 9,896
0
votes
1
answer
307
views
Limit AWS Lambda to poll dynamo db streams
I currently have a dynamo db table which adds new records at 80 TPS. Now, I want to do some processing on these records and I'm planning to trigger a Lambda function from a dynamo db stream on the ...
- 201