Meetup on Apache Zookeeper
Download as PPTX, PDF1 like697 views
Zookeeper is a distributed coordination service that provides naming, configuration, synchronization, and group services. It allows distributed processes to coordinate with each other through a shared hierarchical namespace of data registers called znodes. Zookeeper follows a leader-elected consensus protocol to guarantee atomic broadcast of state updates from the leader to followers. It uses a hierarchical namespace of znodes similar to a file system to store configuration data and other application-defined metadata. Zookeeper provides services like leader election, group membership, synchronization, and configuration management that are essential for distributed systems.
![Zookeeper Future Releases - 3.5.X
• Dynamic Reconfiguration w/o Rolling restarts
• Separate Dynamic configuration file
• Adding/Removing Servers
• Modifying server parameters
• Rebalance Client Connections
• Backward Compatible
• Modifying Server Roles[Obersver/Participants]](https://image.slidesharecdn.com/apachezookeepermeetupv1-161009172809/85/Meetup-on-Apache-Zookeeper-28-320.jpg)
Meetup on Apache Zookeeper
- 2. What is distributed system ? • A distributed systems provides single coherent view of collection of independent computing machines. • Multiple software components on multiple computers, which run as a single system. • It can be locally or geographically connected.
- 3. Characteristics of Distributed System • Concurrency • Fault Tolerance • Scalability • Transparency • Heterogeneity
- 4. Delusion of Distributed System • Reliable Network • Zero Latency • Infinite Bandwidth • Secure Network • Topology Doesn’t change • Homogeneous Network • There is one administrator
- 5. Introducing Zookeeper • Zookeeper provides coordination between distributed process via shared hierarchical namespace.
- 6. Why Zookeeper ? • Naming Service • Locking • Synchronization • Configuration Management • Leader Election
- 8. Zookeeper Architecture • Zookeeper follows client server architecture • Collection of Zookeeper servers is called an ensemble • At given time, a Zookeeper client is connected to one Zookeeper server • Client read requests are catered by connected Zookeeper server • Client write requests are forwarded to Zookeeper leader. • Write is considered successful, if quorum is achieved.
- 9. Zookeeper Architecture • Client updates are sequentially consistent • Updates are atomic • Updates are reliable • Timeliness
- 10. ZAB Protocol • Zookeeper Atomic Broadcast • Propagates state changes from leader to followers • Guarantees the state change
- 11. Znodes • Organized similar to standard Unix and Linux file system • Similar to files and directories • Both directory and leaf znode holds data • Stores data <= 1MB
- 12. Znode - Hierarchical Namespace
- 13. Time in Zookeeper • Zxid : Every change to ZK state receives a stamp in form of zxid • Version Number : Change to Znode will cause change in Version • Ticks : Ticks define timing of events • Real Time : To put timestamp of stat structure and znode modification
- 14. Znode Stat Structure • Czxid : Change that caused Znode to be created • Mzxid : Change that last modified Znode • Pzxid : Change that last modified children of the Znode • Ctime : Millisecond from epoch, when this Znode was created • Mtime : Millisecond from epoch, when this Znode was last modified • Version : Number of changes to data of Znode • Cversion : Number of changes to children of Znode • Aversion : Number of changes to ACL of Znode • EphemeralOwner : SessionId of owner of this Znode • DataLength : Length of data field • Numchildren : Number of children
- 15. Zookeeper Operations create Creates Znode delete Deletes Znode exists Does Znode exist? getACL Get ACL permissions setACL Set ACL permissions
- 16. Zookeeper Operations getChildren Gets the childrens of Znode getData Gets Data of Znode setData Sets Data of Znode Sync Syncs the client View
- 17. Types of Znodes • Ephemeral Znode : Deleted with termination of client session, also can’t have any child znode. • Persistent Znode : Can have child znode, It is explicitly required to delete it. • Sequential Znode : Append a monotonicly increasing counter to the end of path. It is unique to parent znode.
- 18. Znode Watchers • One time triggers • Sent to client which sets the watch • Maintained locally at zookeeper • Client won’t receive watches after getting disconnected • Two lists of watches – Data watches : create(),setData(), delete() – Child watches : create(), delete()
- 19. Zookeeper ACL • ACL Permission – Create – Read – Write – Delete – Admin
- 20. Zookeeper ACL • ACL Scheme – World – Auth – Digest – Host – IP
- 21. Zookeeper Recipes/Higher Order Functions/Apache Curator • Naming Service • Service Discovery • Configuration Management • Locks • 2-phase commit • Leader Election • Barriers • Queues
- 22. Netflix/Soabase Exhibitor • Zookeeper server is fail fast • Zookeeper is self-healing, (i.e. crashed/failed server if brought online, will automatically join the cluster) • Instance Monitoring • Log Cleanup • Backup/Restore • Cluster-wide configuration • Rolling Ensemble Change • Rich REST API
- 23. Backup/Exhibitor • Online Backup - Use Netflix Exhibitor • Offline Backup - Bring down the old ensemble cluster and copy the zookeeper data directory to the new ensemble cluster
- 24. Zookeeper Monitoring • 4 Letter Words • JMX
- 25. Zookeeper Monitoring - 4 Letter Words conf srst cons srvr dump stat envi dirs (3.5.1) ruok mntr
- 26. Zookeeper Monitoring - JMX • Mbeans Quorum ServerCnxn Leader LocalPeer Follower LeaderElection DataTree
- 27. Data Directory Cleanup • autopurge.snapRetainCount : Maintains n number of recent snapshots along with transaction log • autopurge.purgeInterval : Time Interval at which purge event will trigger
- 28. Zookeeper Future Releases - 3.5.X • Dynamic Reconfiguration w/o Rolling restarts • Separate Dynamic configuration file • Adding/Removing Servers • Modifying server parameters • Rebalance Client Connections • Backward Compatible • Modifying Server Roles[Obersver/Participants]
- 29. Zookeeper in Real World • Apache Kafka • Apache Storm • Apache HBase • Apache Solr • Apache Mesos • Apache Hive