257 questions
1
vote
0
answers
26
views
Storing Domain Events directly in Outbox table vs mapping to Integration Events in Java/Spring (DDD based Microservices) [duplicate]
I have a Spring Boot / Java application following Domain-Driven Design and Event-Driven Architecture.
In my domain service, I create a ReservationCreatedEvent (a domain event) and publish it:
...
1
vote
2
answers
75
views
SQS Messaging Design: Payload vs. Reference?
I'm working on an event-driven system using AWS SQS and facing a design decision when sending commands between services. The command triggers an operation in the receiving service, and that operation ...
- 27
-1
votes
1
answer
46
views
How can i manage offsets in Kafka at Message Level
This is my code :
@retry(stop=stop_after_attempt(3), wait=wait_fixed(2))
def kafka_consumer(self):
"""Consume messages from Kafka with retry logic."""
...
0
votes
2
answers
152
views
How to format or compress the subject field of Event Grid events in Azure
I'm working with Azure Event Grid to receive incoming events from an external publisher. When files are uploaded to a blob container named "snlf", I want my Service Bus trigger Function App ...
- 229
1
vote
1
answer
50
views
Design Review: Handling Dynamic Event Processing in Azure Function App
I'm building a piece of software responsible for processing different types of events from my system. This component is hosted as an Azure Function App, triggered by an Azure Storage Queue. The ...
- 115
0
votes
1
answer
166
views
How to recover data from Google pub/sub topic subscription when my service is down
I have subscribed to a Google pub/sub topic from my spring boot application with subscription.What happens if my service is down and There is some data which is pushed to pub/sub topic. Will i receive ...
- 23
0
votes
0
answers
73
views
How to handle an authorization policy in both queries and commands in CQRS?
I'm implementing a restaurant management micro-service using CQRS pattern.
The read and write databases are separated and eventually become consistent by async events using an event bus.
Now I want to ...
- 131
-1
votes
1
answer
70
views
what message delivery guarantees to choose for KAFKA CQRS
need help
i am implementing CQRS using i would like to know which method i should choose when it comes to message delivery guarantees which one is suitable for CQRS
if i choose at least one i will ...
2
votes
1
answer
458
views
How to achieve distributed transaction consistency in a microservices architecture without using 2PC?
In a microservice architecture, each service typically manages its own database and executes operations independently. However, when dealing with business processes that span multiple microservices, ...
- 494
0
votes
1
answer
60
views
Decoupled communication between Stateful and a plain object
Looking for an answer from a senior architect in Flutter who knows the in and out of the SDK and the platform.
Objective is to get notified once a Stateful object comes to life or gets instantiated. ...
- 2,180
0
votes
1
answer
204
views
In an event driven architecture (websockets, pub sub, etc), do the clients and server constantly check for new messages? How does it work?
I have read a number of articles describing how websockets and pub sub architectures work at a high level.
I get that with websockets a connection is persisted between a client and a server to send ...
- 1,899
0
votes
1
answer
152
views
Event Driven Architecture: Populating User Dashboard data
Let's say we are building an application where the user has a dashboard populated with data from multiple services. For example: On the dashboard, Display pending notifications to be read, fed by ...
- 1
1
vote
1
answer
684
views
Message ordering in event driven architecture
Consider there are 3 microservices - s1, s2 and s3.
s1 sends message m1.
s2 consumes message m1, applies some business logic and then sends message m2.
The problem is that s3 receives message m2 ...
- 27
1
vote
1
answer
147
views
Dashboard in Event driven design
Lets imagine we have several bounded contexts and they are all decoupled using messaging services (queues, message buses).
Some business process, lets say Order processing spans multiple bounded ...
- 15.5k
1
vote
1
answer
57
views
EDA Choreography - keep overall state
In an event driven architecture using choreogeaphy model, how do we keep current, global state of the process?
Lets say we have a process where many services p1,...,pn transition many states s1,...,...
- 15.5k
0
votes
1
answer
67
views
Does not using log based broker break event driven architecture
If I am using a queue (SQS or RabbitMQ) to communicate events between micro-services then does it not conform to the requirements of event driven architecture? as, I am not using a durable (log based) ...
- 359
0
votes
2
answers
104
views
Axon v4.9.5 startup problem, came out of nowhere
I've built my CQRS project Core-API, where i have declared events, and commands, also the AxonConfig (just some xStream allowtypesByWildcard)
Till this point, everythings works fine
I created the ...
0
votes
2
answers
82
views
Is using SQS Lambda combination the only way here?
I have a architectural problem at work,
There are records coming into a SNS FIFO topic, each record coming in needs to be processed by 3 different modules in a sequential manner.
For sake of ...
- 41
1
vote
1
answer
100
views
How to maintain control of numerous event racing scenarios in a platform of growing complexity?
I have a Voucher Service which listens to incoming sales, and updates vouchers based on what has been purchased/returned etc.
So we consume a SaleCompletedEvent, which may put vouchers into a '...
- 4,638
0
votes
0
answers
86
views
Spark-Eventstoredb connect
I'm working on handling events coming from eventstoredb using spark put actually couldn't find a straightforward way to connect between them with pyspark.
What I aim to do is to read events as stream ...
- 11
1
vote
0
answers
111
views
How can I run a kafka consumer in a django project so that messages will passed to django project
I am running my kafka consumer (from the confluent_kafka library) in a separate django management command. (I did this because I couldn't find a way of running a kafka consumer within the runserver ...
- 11
0
votes
2
answers
690
views
Generating own AWS Trace Header on an SQS message does not prevent Recursive Loop on a Lambda
I'm designing a system solution where an SQS queue triggers a Lambda, the queue receives a lot of messages for a logical grouping say an execId. The lambda reads the messages from the queue in a batch ...
- 23
0
votes
0
answers
199
views
Sync data from one application to another
We have an application A which has its own domain model and database (SQLSERVER). Another application B has its own domain model and database. We want to create another application C which is to ...
- 49
0
votes
0
answers
140
views
How to break inifine loop in a event driven architecture?
To understand the issue I provide a simplified diagram of my system:
Let's see the Business service allows you to create an account. It generates 1. Business event.
The event is processed by the ...
0
votes
1
answer
1k
views
Event Bridge Targets
I'm investigating the use of event bridge as we are currently using Kafka and we want to confirm it's the best fit for us.
However, I haven't used event bridge before and I'm a little confused with ...
- 119
0
votes
1
answer
170
views
QM Error: ERROR F041 Failed to parse the model file
I have a QM model that has been created using QM Version 5.1.1. Now, when I attempt to open it using QM Version 5.2.5, the following error arises.
If it is not possible to open QM Modeled with older ...
- 129
0
votes
1
answer
285
views
Redis Streams, Event Driven Architecture and Schema Evolution
I'm really interested in using Redis Streams as the messaging backbone for a platform utilising event driven architecture in place of Kafka for the following reasons:
Lower cost
Drastically lower ...
- 688
0
votes
0
answers
380
views
How to achieve Service Orchestration using workflow like Airflow or Event-driven architecture
I am implementing a business process that performs 6 steps e.g.
The Service Orchestrator consumes an event from an SNS (or Standard Publish-Subscribe service) Topic
Using the message consumed from ...
- 1,364
0
votes
1
answer
119
views
EDA with EventSourcing order of events guarantee with Kafka
Let's say I have an Order and a Product.
Whenever product is created I publish PRODUCT_CREATE event and for the order it is ORDER_CREATE
If I use Kafka ( or any message broker ) should I have 1 queue ...
- 25
2
votes
2
answers
3k
views
How Event Driven Architecture Know When Request is Complete?
Let's say there's a service called "CreateOrder". Let's say as well that the service depends on "Inventory" service to check whether the items in the order are available, and also &...
- 551
1
vote
2
answers
616
views
Microservices streaming data vs events
In microservices architecture, you can follow an event driven architecture where you want to publish events, doubt it’s about size of the payload.
What is a reasonable amount of data to be there?
You ...
- 2,189
0
votes
1
answer
466
views
How to handle time-related events in Event Driven Architecture?
I have a distributed platform which allows customers to make purchases, and the items which are purchased are stored in an inventory:
Sales app -> PurchaseEvent -> Inventory app
The Sales app ...
- 4,638
1
vote
0
answers
1k
views
Creating an event driven architecture using Django
I would like to create an event driven architecture in Django. Every time a database record is created, updated or deleted, i would like to create and event, posting it to a queue ( like kafka, sqs, ...
- 8,446
1
vote
1
answer
502
views
How will Kafka manage order of events if 1 Topic and 1 consumer in 1 consumer group but 1 consumer is running on multiple machine
Recently I was given a situation:
Events are being published to 1 Topic
Given we have 1 consumer in 1 consumer group.
In order to keep the pace of produced message and consumeed message we have ...
- 435
1
vote
1
answer
336
views
How to "replay" past integration events to replicate current state in a new context
I'm quite new to DDD and Event Driven architectures. And after searching this issue I did not manage to get a clear answer by myself, so I'll appreciate if someone can clarify this.
Suppose I ...
- 73
3
votes
1
answer
373
views
Guarantying eventual consistency without a message broker (utilising an in memory message bus)
I am thinking if there could be an easy way to guarantee eventual consistency in an event driven modular monolith, which is utilising an in process message bus instead of an actual external message ...
- 773
1
vote
1
answer
120
views
Migrating an asynchronous businness flow to an event-driven system
In the effort to redesign an asynchronous flow based functional service to an event driven one, we have come up with changes on different part of this system. The service receives various statuses ...
- 2,932
0
votes
1
answer
774
views
Should each microservice contain event consumer itself or separate from it in event-driven MSA?
Which one is better between
(1) having event consumer(e.g. KafkaListener) in server application of a microservice
(2) having another application only for consuming the events and calling API of the ...
- 461
1
vote
1
answer
1k
views
How multiple Kafka Consumers in the same consumer group read messages from one partition in the topic?
I would like to know about how the consumers in the same consumer group read the messages from one topic which has only one partition.
For example, I have 3 consumers in one consumer group and that ...
- 45
1
vote
1
answer
512
views
What is the correct streaming pattern to replace database table polling?
I am trying to architect an event streaming system to replace our existing database table polling mechanism. We currently have a process where Application ABC will query/scan the entire XYZ (MySQL) ...
- 659
2
votes
1
answer
195
views
Retry logic when a partial update failed
I think that it is a general problem and it is not related with the using technologies. So, please consider the problem itself.
I am storing data in Couchbase like below format
productId is document ...
- 471
0
votes
2
answers
1k
views
Consuming messages in a Kafka topic ASAP
Imagine a scenario in which a producer is producing 100 messages per second, and we're working on a system that consuming messages ASAP matters a lot, even 5 seconds delay might result in a decision ...
- 1,859
0
votes
1
answer
2k
views
Zephyr Project Event Handling
I am working on an event driven portion of a project utilizing the use of the Zephyr Project. I am relatively new to Zephyr so I apologize if I am going about this wrong. My goal is to have a forever ...
0
votes
1
answer
448
views
A simple event driven batch processing
First of all, I wanted to gather some information one how a simple event driven architecture works in real life. If this question is not suitable in here. please kindly let me know where I can ask ...
- 1,662
0
votes
2
answers
309
views
How to do versioning of Custom Serdes objects in Kafka Stream while using processor API during Rolling Upgrade
How to achieve handling multiple versions of UserDetailDto while processing it from Topic-A to Topic-B with Kafka stream using processor API.
Existing instance/replica of aggregation service should ...
- 81
0
votes
1
answer
380
views
Kafka client and aggregated events
In event-driven design we strive to find out events that we interested of. Using Kafka we can easily subscribe (a new group.id) to a topic and start consuming events. If retention policy is default ...
- 11
0
votes
1
answer
970
views
Webhook (Push) vs Asynchronous Queue (Push) Architecture [closed]
When building webhooks, it's a best practice for the consumer of the webhook (e.g. the receiver of the webhook) to immediately drop any messages received into a queue to prevent it from "backing ...
- 1,462
0
votes
1
answer
160
views
ddd: remove entity (microservice) dependent of other (microservice)
We've two domains: Departments and Employees.
Both of them are microservices and we're using an event bus (kafka) to populate events.
As usual, department can have several employees.
Currently, we've ...
- 23.7k
4
votes
3
answers
2k
views
Flutter BLoC - Bloc vs Cubit event driven state management advantages
What are the actual advantage(s) of Bloc over Cubit?
In addition to traceability (which you can also achieve with appropriate logging in Cubit), and advanced event transformations (I can't think of ...
- 2,526
-1
votes
1
answer
433
views
How to create a event source system with nodejs and mongodb
I am quite new to event source concept but got glimpse that i need to divide my each task or operation to event and save those, but my basic do but is if for each request if you save each step as ...
- 57