Downloaded 14 times
Uploaded byJyrki Pulliainen
Taming Pythons with ZooKeeper (Pyconfi edition)
Apache ZooKeeper is a distributed coordination service that allows distributed applications to synchronize data and configuration information and maintain consistency. It provides a hierarchical name space and simple primitives like counters and change notifications. ZooKeeper elects a leader automatically and guarantees consistency by requiring a quorum for all operations. It is useful for applications that need to maintain consistency but also remain available, as it allows choosing consistency and availability over partition tolerance. Spotify uses ZooKeeper to orchestrate distributed tasks across its infrastructure while maintaining a consistent order and remaining available within data centers.
More Related Content
Similar to Taming Pythons with ZooKeeper (Pyconfi edition)
Taming Pythons with ZooKeeper (Pyconfi edition)
- 1. Taming Pythons withZooKeeper PyCon Finland 2012
- 2.
- 3.
- 4. Day to dayoperations • Consume boatloads of XML • Build database out ofit • Transcode audio • Enrich data • Build indexes • Ship indexes
- 5.
- 6.
- 7. The naïve solution •Central orchestrator • Assume all machines are running • Run remote commands • We need to know all the quirks • Fail fast
- 8. Guess what? Itbreaks
- 9. ...and replacing isn'tsimple
- 10. New tool isneeded • No central orchestrator • Assume some machines are always down • Run locally • Shift responsibility to system owners • Fail gracefully
- 11.
- 12.
- 13. Pick two. Anytwo will do.
- 14.
- 15. Why? • Users aremostly contained in single DC • Inside a single DC connections are quite robust • Remember the order? We need consistency • Availability is everything
- 16.
- 18.
- 19. What? • A distributedtree-like data structure • Simple primitives • Automatic leader elections • Guaranteed hard consistency • Ephemeral nodes
- 20. Tree-like structure / /dir/ /dir2/ /subdir/
- 21. Simple primitives ● Guaranteed atomic operations ● Counters ● Change notifications
- 22. Automatic leader election ● Nodes know who is the most up to date ● If no leader can be picked, ZooKeeper refuses to work
- 23. Guaranteed hard consistency ● Every change is sent to every node! ● Quorum for all operations is always required
- 24. Ephemeral nodes ● Node is present only if the client is alive
- 25.
- 26. The Good: ● Thin ● Comes with ZooKeeper ● Maintained by the Apache ZooKeeper project
- 27. The bad: ● Thin ● C bindings only, no PyPy for you
- 28. The ugly: ● No documentation :(
- 29. There are others:Kazoo
- 30. The Good: ● Pure Python ● Recipes implemented ● Used by many (Quora, Mozilla, reddit, Zope)
- 31. The bad: ● Not much recipes done ● Not owned by the mainline
- 32. The ugly: ● Own implementation of the protocol
- 33.
- 34. Don't ship largechunks
- 35.
- 36. Don't write thereall the time
- 37.
- 38.
- 39.
- 40.
- 41.
- 42. No partitions? GoZooKeeper!
- 43. Pick your weaponlibrary
- 44.
- 45.