Downloaded 284 times
Uploaded byconfluent
Exactly-once Semantics in Apache Kafka
The document discusses the introduction of exactly-once semantics in Apache Kafka, outlining its significance for reliable stream processing. It explains the challenges and existing semantics, highlighting improvements through idempotent producers and transactional writes. The presentation concludes with insights on performance gains and future developments in Kafka's capabilities.
More Related Content
Similar to Exactly-once Semantics in Apache Kafka
Exactly-once Semantics in Apache Kafka
- 1. 1 Introducing Exactly Once Semanticsin Apache Kafka™ Apurva Mehta, Software Engineer, Gehrig Kunz, Technical Product Marketing Manager
- 2. 2 Agenda • Why exactly-once? •An overview of messaging semantics • Why are duplicates introduced? • What is exactly-once semantics? • Exactly-once semantics in Kafka: Is it Practical? • Next Steps
- 3. 3 Exactly Once Semanticsis a hard problem
- 4. 4 An overview ofmessaging semantics • At-most once • At-least once • Exactly-once
- 5. 5 Why exactly-once? • Streamprocessing is becoming the norm; it’s more natural. • Apache Kafka is the most popular streaming platform. • Mission critical applications require stronger guarantees.
- 6. 6 Why exactly-once? • Streamprocessing is becoming the norm; it’s more natural. • Apache Kafka is the most popular streaming platform. • Mission critical applications require stronger guarantees. In other words: make stream processing easy, simple, and reliable enough for everyone.
- 7. 7 Apache Kafka’s existingsemantics At Least Once
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14. 14 What do wedo now??? Kafka’s Existing Semantics
- 15. 15 Kafka’s Existing Semantics:At Least Once
- 16. 16 Kafka’s Existing Semantics:At Least Once
- 17. 17 Kafka’s Existing Semantics:At Least Once
- 18. 18 Why are duplicatesintroduced? Various failures must be handled correctly: • Broker can fail • Producer-to-Broker RPC can fail • Producer or Consumer client can fail
- 19. 19 TL;DR – Whatwe have today • At least once in order delivery per partition. • Producer retries can introduce duplicates and headaches.
- 20. 20 The age oldengineering question Before we make this work, are we sure we should?
- 21. 21 KafkaCash: A Peerto Peer Lending App A peer-to-peer lending platform.
- 22. 22 Help Bob reach$1000, send him $10
- 23.
- 24.
- 25.
- 26.
- 27. 27 How did Kafkaadd exactly once semantics?
- 28. 28 Exactly-once semantics inKafka, explained Apache Kafka’s guarantees are stronger in 3 ways: • Idempotent producer: Exactly-once, in-order, delivery per partition. • Transactions: Atomic writes across partitions. • Exactly-once stream processing across read-process- write tasks.
- 29. 29 Part 1/3 :Idempotent Producer Exactly-once, in-order, delivery per partition
- 30. 30 Idempotent Producer Semantics Asingle --successful!-- producer.send will result in exactly one copy of the message in the log in all circumstances.
- 31. 31 Producer Configs • enable.idempotence= true • max.inflight.requests.per.connection=1 • acks = “all” • retries > 0 (preferably MAX_INT)
- 32.
- 33.
- 34.
- 35.
- 36.
- 37.
- 38.
- 39.
- 40. 40 TL;DR: idempotent producer •Works transparently -- only one config change. • Sequence numbers and producer ids are in the log. • Resilient to broker failures, producer retries, etc.
- 41. 41 Part 2/3 :Transactions Atomic writes across multiple partitions.
- 42. 42 Transactions semantics • Atomicwrites across multiple partitions. • All messages in a transaction are made visible together, or none are. • Consumers must be configured to skip uncommitted messages.
- 43. 43 Producer config fortransactions • transactional.id = ‘some string’ • Typically based on the partition identifier in a partitioned, stateful, app. • Enables transaction recovery across producer sessions.
- 44. 44 The transaction API producer.initTransactions(); try{ producer.beginTransaction(); producer.send(record0); producer.send(record1); producer.commitTransaction(); } catch (KafkaException e) { producer.abortTransaction(); }
- 45.
- 46. 46 1. Initialize theproducer producer.initTransactions(); try { producer.beginTransaction(); producer.send(record0); producer.send(record1); producer.commitTransaction(); } catch (KafkaException e) { producer.abortTransaction(); }
- 47.
- 48. 48 2. Begin transactionsand send data producer.initTransactions(); try { producer.beginTransaction(); producer.send(record0); producer.send(record1); producer.commitTransaction(); } catch (KafkaException e) { producer.abortTransaction(); }
- 49.
- 50.
- 51. 51 3. Commit transaction producer.initTransactions(); try{ producer.beginTransaction(); producer.send(record0); producer.send(record1); producer.commitTransaction(); } catch (KafkaException e) { producer.abortTransaction(); }
- 52.
- 53.
- 54.
- 55.
- 56. 56 Consumer configs • isolation.level: •“read_committed”, or • “read_uncommitted”
- 57. 57 What do youget with isolation levels? • read_committed: consumers read to the point where there are no open transactions. • read_uncommitted: will read everything. • Messages read in offset order.
- 58. 58 TL;DR: Transactions • Atomic,multi-partition, writes. • Use the new producer APIs for transactions. • Consumers can filter out uncommitted or aborted transactional messages.
- 59. 59 Part 3/3 :Stream Processing Stream Processing with Exactly Once Semantics
- 60. 60 Streams config • processing.mode= “exactly_once”
- 61. 61 End-to-end exactly-once semantics •The read-process-write operation is atomic. • Thus streams tasks produce valid answers even when failures happen.
- 62.
- 63. 63 Exactly Once Semanticsin Kafka Is it practical?
- 64. 64 Performance boost forApache Kafka 0.11! • Up to +20% producer throughput • Up to +50% consumer throughput • Up to -20% disk utilization • Details: https://bit.ly/kafka-eos-perf
- 65. 65 Gains due tomore efficient message format
- 66. 66 What about theidempotent producer and transactions? • Transactions: 3-5% overhead for 100ms transactions, 1KB messages. • Longer transactions and better batching result in better performance. • 20% overhead relative to at-most once delivery without ordering guarantees. • Idempotent producer alone has negligible overhead.
- 67. 67 Putting it together •We talked through an idempotent producer • How we added transactions with atomic writes • The impact it has on stream processing
- 68. 68 When is itavailable? Available to use in Kafka 0.11, June 2017.
- 69. 69 Where we’ve come 2007 Highthroughput messaging broker 2008 Highly available replicated log 2012 Top Level Apache Project 2016 Streams API Connect API 2017 Exactly Once Semantics
- 70. 70 San Francisco August 28,2017 Organized by Confluent
- 71. 71 What’s next foryou slackpass.io/ confluentcommunity v Try it v v Join the Community Let us know what you think @ConfluentDownload Confluent Open Source
- 72.