0

In Kafka, it's generally recommended to map one table to one topic. However, what if the order of events across multiple tables is important for a specific key, such as CompanyId?

For example, consider the following scenario: events for a CompanyId must be processed (consumed) in order, even though they originate from different tables. Partitioning is configured such that each partition corresponds to a specific CompanyId:

enter image description here

Is it possible to ensure that the order of events (transactions) for each CompanyId is reliably preserved across all relevant tables? For instance, achieving something like this:

enter image description here

I use Avro for schema control.

Improve this question

1 Answer 1

Reset to default
0

I had to solve this problem so I could maintain atomic apply of messages in Kafka that represented replicated database transactions. My requirements were stricter in that I needed total ordering of the original source sequence of messages for my customers and to maintain the context of where transaction boundaries occurred.

I designed the following technology to do this and it is in use at a large number of our financial industry customers who need to ensure exactly once characteristics and perfect ordering, along with maintaining transaction atomicity.

https://www.ibm.com/docs/en/idr/11.4.0?topic=kafka-transactionally-consistent-consumer

I gave a talk at the Kafka conference describing the internals of the approach… I think Confluent has it posted somewhere….

Ah here…

https://www.confluent.io/kafka-summit-sf18/a-solution-for-leveraging-kafka-to-provide-end-to-end-acid-transactions/

It is essentially a two phase commit approach to the problem utilizing the information in callbacks to record which exact message is the next to return, and injecting into the meta topic information about when commits occurred on the source.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.