4,304 questions
1
vote
0
answers
38
views
Using Mass Transit, is there a granular way to override the headers on error messages when there is a fault?
On errors in my service (while consuming a message) Mass Transit will attempt to send the message to an '_error' queue and will fail due to an SQS hard limit on message Attributes:
MassTransit....
1
vote
1
answer
74
views
Filter aws lambda event source on metadata
I'm trying to filter based on metadata values of incoming SQS events, not what is in their body.
From the documentation, it is mentioned that there is a messageAttributes attribute. I want to filter ...
0
votes
0
answers
39
views
Node exe not working when I use AWS SQS client
I am using Node.js and the SQS client to listen to a FIFO queue and process events. It works when I run it in Node, but I would ideally like to compile an exe with all its dependencies and run the exe....
-4
votes
1
answer
72
views
What is the difference between a key managed by SQS vs a KMS key managed by SQS?
I'm looking into encrypting an SQS Queue that I've got using the CDK, and as options, there are
Encryption type
Description
UNENCRYPTED
Messages in the queue are not encrypted.
KMS_MANAGED
Server-side ...
0
votes
0
answers
48
views
Camel aws2 sqs warns it cannot add headers to MessageAttributes, but it shouldn't
I'm upgrading from camel-aws to camel-aws2. All in all, everything seems to be working. However, I keep getting these warnings in the logs:
WARN 23652 --- [Camel (camel-1) thread #11 - seda://to-image-...
Advice
0
votes
2
replies
61
views
How to retry sending a message to AWS SQS in Spring Boot, and then send it to a DLQ if all retries fail?
I am working on a Spring Boot 3.5.0 application that publishes messages to an AWS SQS FIFO queue.
My requirement is:
When sending a message to SQS:
If it fails (network error, AWS error etc.), the ...
1
vote
0
answers
53
views
Django Celery Beat SQS slow scheduling
Beat seems to be sending the messages into SQS very slowly, about 100/minute.
Every Sunday I have a sendout to about 16k users, and they're all booked for 6.30pm. Beat starts picking it up at the ...
0
votes
0
answers
71
views
Laravel + RoadRunner + SQS — API workers blocked while publishing to SQS (70ms latency per request)
I’m running a high-concurrency Laravel application on an EC2 instance using RoadRunner (12 PHP workers).
Each incoming request simply publishes a message to AWS SQS — that’s all it does.
The problem: ...
0
votes
1
answer
41
views
WSO2 MI 4.4.0 - Unable to push data to AWS SQS Queue using AWS SQS connector via proxy server
I am trying to push incoming data to AWS SQS in WSO2 MI 4.4.0 using AWS SQS connector v3.0.1.
Integration: WSO2 MI 4.4.0
Developer Tool: WSO2 MI for VS Code extension
OS: Ubuntu 22.04
Proxy server: ...
0
votes
0
answers
64
views
Why Upon failure SQS does not place the failed message upon DLQ?
I am running this worker:
php -d memory_limit=-1 bin/console messenger:consume sqs_channel_manager -vv
And worker fails:
14:29:32 INFO [messenger] Received message App\Domain\Event\ChannelManager\...
0
votes
1
answer
80
views
How can I disable ssl upon sqs on localstack?
Upon a symfony app I try to use localstack for local sqs:
services:
php:
container_name: php
build:
context: ./php
args:
TIMEZONE: ${...
0
votes
1
answer
79
views
SQS endpoint resolution by AWS SDK
I am configuring SQS client as shown in the below snippet:
SqsClient client = SqsClient.builder()
.region(region)
.credentialsProvider(DefaultCredentialsProvider.create(...
1
vote
2
answers
75
views
SQS Messaging Design: Payload vs. Reference?
I'm working on an event-driven system using AWS SQS and facing a design decision when sending commands between services. The command triggers an operation in the receiving service, and that operation ...
0
votes
2
answers
215
views
AWS lambda destination using "Event source mapping invocation"
I have a lambda function that has an SQS queue as a trigger.
I've understood that this sort of trigger causes a synchronous invocation, and that if I want a destination, it must be set up accordingly....
0
votes
0
answers
139
views
Connection Pooling Challenges with MongoDB and AWS Lambda
I'm facing an issue related to MongoDB connections while sending push notifications to users three times a day — morning, noon, and evening.
I have a Lambda-based service that uses SQS and MongoDB. A ...
1
vote
1
answer
42
views
How to setup SAM template with permissions for AWS SQS
I'm working on an application that moves files from an sftp server to an AWS S3 bucket using an SQS message queue. Currently, the application is working but the permission policies in the template don'...
2
votes
1
answer
54
views
Are in-flight messages still deduplicated in an AWS SQS FIFO queue?
If the producer sends a message to an SQS FIFO queue while another message with the same de-duplication ID is in-flight (read by the consumer but not yet committed), will the new message be dropped as ...
0
votes
0
answers
57
views
Celery + SQS: Worker Process Delay - Task Received but Execution Starts After 30+ Seconds
Celery Worker Delay: Task Received but Execution Starts After 30+ Seconds
Environment
Python 3.9.7
Celery 5.5.3
kombu 5.5.4
billiard 4.2.1
AWS SQS as message broker
Docker container running on m1 ...
0
votes
1
answer
43
views
How to send a complex object in dynamo attribute using sdk?
I am trying to insert a record in dynamodb and then get it processed using a lambda via SQS. This record contains a complex object. When I am inserting it using the sdk, i am able to see the record in ...
2
votes
1
answer
48
views
Order of Events processing with combination of FIFO SQS and AWS Lambda
AWS Lambda documentation says:
Lambda scales your function by running additional instances of it as demand increases, and by stopping instances as demand decreases. This model leads to variations in ...
0
votes
1
answer
86
views
Is it possible to customize the SFN execution name when triggering SFN using SQS + Eventbridge Pipes?
I am trying to trigger an SFN using SQS + Pipes, and I wanted to know if it's possible to configure the name of the SFN execution based on the payload.
Example:
If the event is as follows:
{"...
0
votes
1
answer
198
views
Aws SSO credential provider auto refresh token SDK of JAVA2 not work
I meet a problem and can't sovle it even though i search in AWS docs.
I got doc from this link:
https://docs.aws.amazon.com/sdkref/latest/guide/understanding-sso.html
https://docs.aws.amazon.com/...
0
votes
1
answer
126
views
Is there any way to get the Receipt Handle(s) for messages in flight for an SQS queue?
So - we have a cloud watch alarm for the age of the oldest message in an SQS queue. Basically to alert us if consumers aren't doing their job/keeping up. It started going bonkers today because ...
1
vote
1
answer
175
views
issue with sqs/lambda event filters using terraform
I am running into the following issue using the event source mapping filter with sqs lambda trigger
in terraform the trigger with filter looks like this
resource "aws_lambda_event_source_mapping&...
-2
votes
1
answer
89
views
How is load-balancing handled between multiple SQS triggers of a lambda?
If a lambda has multiple SQS queue triggers, how the polling for events is balanced? Specifically, if one of the SQS queues (Queue A) has a batch size of 10 and the other (Queue B) has a batch size of ...
0
votes
1
answer
118
views
Set up to send a message from account A ECS task to SQS in account B
I want to send a message from the task running ECS cluster in Account A to the SQS in account B.
Referrring different documentation to do the setup, i'm little confused as it is mentioned in few cases ...
0
votes
0
answers
14
views
BatchEntryIdsNotDistinct in Apache airflow, Is my data lost which is processed in the task?
I have an Apache airflow DAG which is receiving messages from SNS-SQS with the code
task1 = SqsSensor (
task_id="SqsSensor name",
sqs queue="Queue name",
max_messages=...
0
votes
0
answers
90
views
aws sqs messages showing up after delete
Even after deleting messages from SQS queue, the messages still show up. I understand that if any other thread picks up a message which is being processed by other thread or process when the ...
0
votes
0
answers
36
views
Can we use DataLoader outside of a GraphQL context in Node.js?
For a GraphQL request, in our resolver, we often need to insert messages to an AWS SQS queue which then triggers a Lambda function.
So we wanted to batch all invocations to our util function (where we ...
0
votes
0
answers
86
views
Issue with SNS subscriptions
I wanted to mention a (supposed) issue with AWS SQS. The logic is simple: an SNS topic that has a subscription to a queue. It works with one way of creating it and not with others. The fact is that ...
0
votes
1
answer
189
views
Localstack upgrade v2 -> v3 - SQS service not available on localhost
I have a Localstack repository which starts up using Docker Compose. The relevant part of the Docker Compose file is as follows:
version: '3.4'
services:
localstack:
image: localstack/...
0
votes
0
answers
130
views
Passing parameters from AWS EventBridge Pipe to ECS task read by a python application
I am totally at a loss trying to figure out a seemingly simple thing and all searches are taking me in different directions, except offering the clear solution.
I have a python application (not flask/...
0
votes
1
answer
203
views
Java 21 SQS Listener Fails to Parse Custom Object Sent as Message
I am upgrading a microservices-based system from Java 17 to Java 21, running on Spring Boot 3 and Spring Framework 6. My application sends a custom object (Event) to an AWS SQS queue, but my message ...
1
vote
1
answer
141
views
Why isn’t my EventBridge Pipe scaling with increased SQS FIFO messages
I’m facing an issue with a webhook system built using AWS SQS FIFO queues, EventBridge Pipes, Step Functions, and Lambda. Here’s the setup:
• SQS FIFO Queue:
• Receives webhook messages containing ...
0
votes
1
answer
140
views
software.amazon.awssdk for SQS is not showing SqsClient [closed]
I'm trying to create a microservice to consume SQS messages and call an API, but after importing in the dependency for SQS via maven
<!-- https://mvnrepository.com/artifact/software.amazon.awssdk/...
0
votes
0
answers
108
views
How can I gain access to an AWS SQS queue message using argo events in Kubernetes?
I have a k3s cluster setup with the Argo Events framework. I have an AWS SQS queue set up.
The event source and sensor are configured properly (Every time I send a message to the queue manually ...
1
vote
1
answer
85
views
SQSListener with multiple IAM users
I have one spring boot project with 2 separate SQS queues. Each SQS queue is in a separate AWS account with a separate IAM user. I have two separate config classes where I configure SqsAsyncClient ...
0
votes
0
answers
92
views
Use IAM Role for Java AWS SQS queue
I have an application that is supposed to consume messages from a given SQS queue.
My method is specified as the following:
@SqsListener(value = "TEST-QUEUE.fifo", deletionPolicy = ...
0
votes
1
answer
244
views
How to Pass SQS Message Body as Environment Variables to AWS Batch Containers Using EventBridge
I am configuring a workflow where messages from an SQS queue trigger AWS Batch jobs, and the content of these messages is passed as environment variables to the container executing the job. I am using ...
0
votes
0
answers
71
views
Amazon SQS FIFO Consumer Stops Processing Events Until Restart
I am using Amazon SQS FIFO to process events received from a Salesforce subscription. Each event contains a recordId, and we use it to call the Salesforce API to fetch additional details before ...
0
votes
1
answer
49
views
Masstransit- how to enable Encryption for amazon queue without using ReceiveEndpoint function
Masstransit- how to enable Encryption for Amazon queue.
I am using AWSSQS queue/topic for messaging purposes & wanted to enable server-side encryption. I am not using ReceiveEndPoint function so ...
-2
votes
1
answer
81
views
Localstack test failing on linux ubuntu
Config for localstack setup:
localstack image:0.14.3 (I have tried other images but same result)
pom: springboot:3.3.1, spring-boot-starter-test, testcontainers-bom: 1.18.3, localstack
runtime: Java ...
0
votes
0
answers
44
views
multiple amazon sqs queues with laravel
I have setup two queues (calculator-webhook-queue & offers-queue) on Amazon SQS with the following code:-
In .env file
QUEUE_CONNECTION=sqs
AWS_DEFAULT_REGION=us-east-1
AWS_ACCESS_KEY_ID=*******
...
0
votes
1
answer
54
views
API Gateway SQS Integration - MessageAttributes not being sent
I am trying to include the API Gateway Request Id in my SQS message, so that I can log it in the consuming Lambda for tracing.
My mapping template is as follows:
Action=SendMessage&MessageBody=$...
1
vote
1
answer
204
views
AWS SQS and Spring pollTimeoutSeconds difference
I hava been reading about using pollTimeoutSeconds in @SqsListener from Spring Cloud AWS.
Look at this configuration:
@SqsListener(value = "my-queue", pollTimeoutSeconds = "10", ...
0
votes
0
answers
180
views
Lambda getting triggered by SQS but not receiving event
I am sending a message to SQS FIFO which triggers the lambda.
The message is received as shown in console, and the lambda is triggered, but the event is empty when read by the lambda function.
...
...
0
votes
0
answers
24
views
AWS Pipe- mapping between api destination request payload and response
I have a aws event bridge pipe where I am calling a 3rd party api using api destination,
payload like:
{
"request_id": 123,
"FirstName": "Launch",
"LastName": &...
0
votes
0
answers
65
views
Celery: How to add a delay to a message sending to SQS
I'm using Celery to consume messages from SQS queue.
The queue is Standard type.
There are cases [exceptions caught] when I explicitly re-enqueue tasks back to the queue.
def run(self):
...
0
votes
1
answer
71
views
Response from AWS SQS Java SDK does not match Python SDK or CLI
We are using Java to send/receive messages from SNS/SQS. When we publish the message, we are doing this:
final var req = PublishRequest.builder()
.topicArn(topicArn)
.message(payload)
....
0
votes
2
answers
194
views
AWS lambda SQS does not allow to process 1 message per 1 invocation
I am trying to set up what I thought initially was an easy pipeline.
I have 2 SQS queues and 2 lambdas that should work sequentially.
The problem is that my lambdas do a heavy job so I want 1 message ...