SlideShare a Scribd company logo
June 29, 2013
Taming Pythons with
ZooKeeper
Wednesday, 3 July 13
$ whoami
Wednesday, 3 July 13
Jyrki Pulliainen
@nailor
jyrki@spotify.com
Wednesday, 3 July 13
Wednesday, 3 July 13
ZooKeeper?
Wednesday, 3 July 13
- This talk is about ZooKeeper.
- Preaching Java software in Python conference. I can think of more healthier things to do too.
- On side note, our Product Owner is really damn good at this game
ZooKeeper!
Wednesday, 3 July 13
- Apache project, Yahoo 2007
- Consistency & Partition tolerance
- Filesystem like, actually can be viewed as a trie, store data in directories too
- In memory, limits the dataset you can store. Maximum zookeeper node, znode, data size 1M
The Tao of
ZooKeeper
7
Wednesday, 3 July 13
Orderly, Reliable, Efficient, Timely, Contention free, ambition free
Wednesday, 3 July 13
June 29, 2013
One library for the enterprise
Curator, from Netflix
Wednesday, 3 July 13
June 29, 2013
Seven for the pythonistas
gevent-zookeeper
zkpython
zc.zk
pykeeper
twitter’s zookeeper library
zoop
txzookeeper
Wednesday, 3 July 13
- gevent-zookeeper -> Spotify
- zkpython segfaults
- Others have somewhat OK implementations, but lack core features
One Library to
rule them all*
* unless you are running twisted
Wednesday, 3 July 13
- txzookeeper still valid for twisted
Kazoo
Wednesday, 3 July 13
- Origins from the Nimbus project
- Ben Bangert as the Sauron of ZooKeeper
- Not Frodo, that bastard wanted to destroy the perfectly good ring.
- All Python, including the protocol. No more segfaults!
CRUD
ZooKeeper CRUD
Let’s talk about KaZoo
13
Wednesday, 3 July 13
Create, Read, Update, Delete
All basic operations available as async too, but we’ll focus on the synchronous use
First we need to connect!
Text
from kazoo.client import KazooClient
zk = KazooClient(hosts='127.0.0.1:2181')
zk.start()
# ...
zk.stop()
TextText
Wednesday, 3 July 13
Easy to connect to one host, multiple host, with namespace....
ZooKeeper supports connection namespacing!
Bonus: get notified when the connection state changes
First we need to connect!
Text
from kazoo.client import KazooClient
zk = KazooClient(hosts='127.0.0.1:2181')
zk.start()
# ...
zk.stop()
TextText
zk = KazooClient(hosts='127.0.0.1:2181,127.0.0.2:2181')
Wednesday, 3 July 13
Easy to connect to one host, multiple host, with namespace....
ZooKeeper supports connection namespacing!
Bonus: get notified when the connection state changes
First we need to connect!
Text
from kazoo.client import KazooClient
zk = KazooClient(hosts='127.0.0.1:2181')
zk.start()
# ...
zk.stop()
Text
zk = KazooClient(hosts='127.0.0.1:2181/namespace,127.0.0.2:2181')
Text
zk = KazooClient(hosts='127.0.0.1:2181,127.0.0.2:2181')
Wednesday, 3 July 13
Easy to connect to one host, multiple host, with namespace....
ZooKeeper supports connection namespacing!
Bonus: get notified when the connection state changes
Create
zk.create('/europython', b'2013')
Wednesday, 3 July 13
Adding nodes easy, helpers exist for recursive creation
Ephemeral ZK feature, session + heartbeats, can’t have children!
Incremental: guaranteed ever increasing 10 digit number in node name
Create
zk.create('/europython', b'2013')
zk.create('/europython/jyrki', ephemeral=True)
Wednesday, 3 July 13
Adding nodes easy, helpers exist for recursive creation
Ephemeral ZK feature, session + heartbeats, can’t have children!
Incremental: guaranteed ever increasing 10 digit number in node name
Create
zk.create('/europython', b'2013')
zk.create('/europython/jyrki', ephemeral=True)
zk.create('/europython/sequential', sequence=True)
Wednesday, 3 July 13
Adding nodes easy, helpers exist for recursive creation
Ephemeral ZK feature, session + heartbeats, can’t have children!
Incremental: guaranteed ever increasing 10 digit number in node name
Read
Text
zk.get('/europython')
Wednesday, 3 July 13
Read
Text
zk.get('/europython')
zk.exists('/europython/jyrki')
Wednesday, 3 July 13
Read
Text
zk.get('/europython')
zk.exists('/europython/jyrki')
zk.get_children('/europython')
Wednesday, 3 July 13
Update & Delete
zk.set_data('/europython/jyrki', b'nervous')
Wednesday, 3 July 13
Update & Delete
zk.set_data('/europython/jyrki', b'nervous')
zk.delete('/europython/jyrki')
Wednesday, 3 July 13
Wednesday, 3 July 13
Distributed locks?
Barrier?
Semaphores?
Counters?
Elections?
Wednesday, 3 July 13
Textfrom kazoo.recipe import <your-favourite-thing>
Wednesday, 3 July 13
Of course it does not have everything from curator
Want to know when things change?
Wednesday, 3 July 13
watchers
Wednesday, 3 July 13
Text
zk.exists('/europython/wine', watch=callback)
zk.get('/europython/wine', watch=callback)
zk.get_children('/europython/dinners', watch=callback)
Wednesday, 3 July 13
from kazoo.recipe.watchers import DataWatch, ChildWatch
@DataWatch('/path/to/node')
def data_callback(data, stat):
# ...
do_something
@ChildWatch('/path/to/node')
def child_callback(children):
# ...
do_something
Wednesday, 3 July 13
TESTS
Wednesday, 3 July 13
What if something goes
WRONG?
Wednesday, 3 July 13
sys.exit()Wednesday, 3 July 13
Stand back!
It’s time for real life example
Wednesday, 3 July 13
Wednesday, 3 July 13
SEARCHED FOR THE NEWEST
JUSTIN BIEBER
Encryption keys were
not available to play it.
Wednesday, 3 July 13
SUMMARY
Wednesday, 3 July 13
Guess what, we’re hiring
Wednesday, 3 July 13
Thank You!
jyrki@spotify.com
@nailor
Wednesday, 3 July 13

More Related Content

What's hot (20)

PDF
Alta performance com Python
Bruno Barbosa
 
PDF
Intel IoT webinar #1 - Tedison presentation
BeMyApp
 
PPTX
Voldemort collections library
Jason Ko
 
KEY
Data Processing @ bit.ly - Posscon 2011
jehiah
 
PDF
CIALUG: Encrypt all the things
Andrew Denner
 
PDF
Managing Windows Systems with Puppet - PuppetConf 2013
Puppet
 
KEY
Active support事始め
Jun Yokoyama
 
PPTX
5 Time Saving Bash Tricks
Nikhil Mungel
 
PDF
gemdiff
teeparham
 
PDF
Videos on Android - Stuff What I Learned
Mark Hemmings
 
PDF
Puppet without Root - PuppetConf 2013
Puppet
 
PPTX
A Brief Introduction to Writing and Understanding Puppet Modules
David Phillips
 
PPTX
Creating beautiful puppet modules with puppet-lint
Spencer Owen
 
PDF
pam_container -- jeszcze lżejsza wirtualizacja
gnosek
 
TXT
Readme
bxccaxxt
 
PDF
(Fun clojure)
Timo Sulg
 
PDF
Json perl example
Ashoka Vanjare
 
PPTX
The Art, Joy, and Power of Creating Musical Programs (JFugue at SXSW Interact...
David Koelle
 
PDF
Surf iOS版 中文用户指南
yarshure Kong
 
PPTX
JFugue, Music, and the Future of Java [JavaOne 2016, CON1851]
David Koelle
 
Alta performance com Python
Bruno Barbosa
 
Intel IoT webinar #1 - Tedison presentation
BeMyApp
 
Voldemort collections library
Jason Ko
 
Data Processing @ bit.ly - Posscon 2011
jehiah
 
CIALUG: Encrypt all the things
Andrew Denner
 
Managing Windows Systems with Puppet - PuppetConf 2013
Puppet
 
Active support事始め
Jun Yokoyama
 
5 Time Saving Bash Tricks
Nikhil Mungel
 
gemdiff
teeparham
 
Videos on Android - Stuff What I Learned
Mark Hemmings
 
Puppet without Root - PuppetConf 2013
Puppet
 
A Brief Introduction to Writing and Understanding Puppet Modules
David Phillips
 
Creating beautiful puppet modules with puppet-lint
Spencer Owen
 
pam_container -- jeszcze lżejsza wirtualizacja
gnosek
 
Readme
bxccaxxt
 
(Fun clojure)
Timo Sulg
 
Json perl example
Ashoka Vanjare
 
The Art, Joy, and Power of Creating Musical Programs (JFugue at SXSW Interact...
David Koelle
 
Surf iOS版 中文用户指南
yarshure Kong
 
JFugue, Music, and the Future of Java [JavaOne 2016, CON1851]
David Koelle
 

Viewers also liked (20)

PPTX
ZooKeeper (and other things)
Jonathan Halterman
 
PDF
ZooKeeper Futures
Cloudera, Inc.
 
PPT
Zookeeper Introduce
jhao niu
 
PDF
Apache ZooKeeper TechTuesday
Andrei Savu
 
PDF
Zookeeper
ltsllc
 
PDF
Taming Pythons with ZooKeeper (Pyconfi edition)
Jyrki Pulliainen
 
PDF
Zookeeper In Action
juvenxu
 
PDF
ZooKeeper and Embedded ZooKeeper Support for IBM InfoSphere Streams V4.0
lisanl
 
PDF
Distributed system coordination by zookeeper and introduction to kazoo python...
Jimmy Lai
 
PPTX
Winter is coming? Not if ZooKeeper is there!
Joydeep Banik Roy
 
PDF
Overview of Zookeeper, Helix and Kafka (Oakjug)
Chris Richardson
 
PDF
ZooKeeper - wait free protocol for coordinating processes
Julia Proskurnia
 
PDF
Jcconf 2016 zookeeper
Matt Ho
 
PDF
Zookeeper
Geng-Dian Huang
 
PDF
Dynamic Reconfiguration of Apache ZooKeeper
DataWorks Summit
 
PPTX
Centralized Application Configuration with Spring and Apache Zookeeper
Ryan Gardner
 
PDF
Apache Zookeeper 分布式服务框架
Cabin WJ
 
PDF
Introduction to Apache ZooKeeper
knowbigdata
 
KEY
Curator intro
Jordan Zimmerman
 
ZooKeeper (and other things)
Jonathan Halterman
 
ZooKeeper Futures
Cloudera, Inc.
 
Zookeeper Introduce
jhao niu
 
Apache ZooKeeper TechTuesday
Andrei Savu
 
Zookeeper
ltsllc
 
Taming Pythons with ZooKeeper (Pyconfi edition)
Jyrki Pulliainen
 
Zookeeper In Action
juvenxu
 
ZooKeeper and Embedded ZooKeeper Support for IBM InfoSphere Streams V4.0
lisanl
 
Distributed system coordination by zookeeper and introduction to kazoo python...
Jimmy Lai
 
Winter is coming? Not if ZooKeeper is there!
Joydeep Banik Roy
 
Overview of Zookeeper, Helix and Kafka (Oakjug)
Chris Richardson
 
ZooKeeper - wait free protocol for coordinating processes
Julia Proskurnia
 
Jcconf 2016 zookeeper
Matt Ho
 
Zookeeper
Geng-Dian Huang
 
Dynamic Reconfiguration of Apache ZooKeeper
DataWorks Summit
 
Centralized Application Configuration with Spring and Apache Zookeeper
Ryan Gardner
 
Apache Zookeeper 分布式服务框架
Cabin WJ
 
Introduction to Apache ZooKeeper
knowbigdata
 
Curator intro
Jordan Zimmerman
 
Ad

Similar to Taming Pythons with ZooKeeper (20)

PDF
A Python Petting Zoo
devondjones
 
PDF
zookeeperProgrammers
Hiroshi Ono
 
PPTX
Apache zookeeper seminar_trinh_viet_dung_03_2016
Viet-Dung TRINH
 
PDF
Apache Zookeeper
Nguyen Quang
 
PPTX
Apache zookeeper 101
Quach Tung
 
PDF
Introduction to Apache ZooKeeper | Big Data Hadoop Spark Tutorial | CloudxLab
CloudxLab
 
PDF
Distributed Coordination with Python
OSCON Byrum
 
PDF
Introduction to ZooKeeper - TriHUG May 22, 2012
mumrah
 
PPTX
Zookeeper Tutorial for beginners
jeetendra mandal
 
PPTX
Introduction to apache zoo keeper
Omid Vahdaty
 
PPTX
Zookeeper
SatyaHadoop
 
PPTX
So we're running Apache ZooKeeper. Now What? By Camille Fournier
Hakka Labs
 
PDF
ZooKeeper Partitioning - A project report
pramodbiligiri
 
PPTX
Leo's Notes about Apache Kafka
Léopold Gault
 
PDF
Malcon2017
Andriy Brukhovetskyy
 
PPTX
Zookeeper big sonata
Anh Le
 
PPTX
Meetup on Apache Zookeeper
Anshul Patel
 
PDF
Tech Talks_25.04.15_Session 3_Tibor Sulyan_Distributed coordination with zook...
EPAM_Systems_Bulgaria
 
PPTX
Distributed Applications with Apache Zookeeper
Alex Ehrnschwender
 
PDF
Базы данных. ZooKeeper
Vadim Tsesko
 
A Python Petting Zoo
devondjones
 
zookeeperProgrammers
Hiroshi Ono
 
Apache zookeeper seminar_trinh_viet_dung_03_2016
Viet-Dung TRINH
 
Apache Zookeeper
Nguyen Quang
 
Apache zookeeper 101
Quach Tung
 
Introduction to Apache ZooKeeper | Big Data Hadoop Spark Tutorial | CloudxLab
CloudxLab
 
Distributed Coordination with Python
OSCON Byrum
 
Introduction to ZooKeeper - TriHUG May 22, 2012
mumrah
 
Zookeeper Tutorial for beginners
jeetendra mandal
 
Introduction to apache zoo keeper
Omid Vahdaty
 
Zookeeper
SatyaHadoop
 
So we're running Apache ZooKeeper. Now What? By Camille Fournier
Hakka Labs
 
ZooKeeper Partitioning - A project report
pramodbiligiri
 
Leo's Notes about Apache Kafka
Léopold Gault
 
Zookeeper big sonata
Anh Le
 
Meetup on Apache Zookeeper
Anshul Patel
 
Tech Talks_25.04.15_Session 3_Tibor Sulyan_Distributed coordination with zook...
EPAM_Systems_Bulgaria
 
Distributed Applications with Apache Zookeeper
Alex Ehrnschwender
 
Базы данных. ZooKeeper
Vadim Tsesko
 
Ad

Recently uploaded (20)

PDF
CIFDAQ Market Wrap for the week of 4th July 2025
CIFDAQ
 
PPTX
WooCommerce Workshop: Bring Your Laptop
Laura Hartwig
 
PDF
Agentic AI lifecycle for Enterprise Hyper-Automation
Debmalya Biswas
 
PDF
Exolore The Essential AI Tools in 2025.pdf
Srinivasan M
 
PDF
From Code to Challenge: Crafting Skill-Based Games That Engage and Reward
aiyshauae
 
PDF
NewMind AI - Journal 100 Insights After The 100th Issue
NewMind AI
 
PDF
July Patch Tuesday
Ivanti
 
PDF
New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
PDF
What Makes Contify’s News API Stand Out: Key Features at a Glance
Contify
 
PDF
HubSpot Main Hub: A Unified Growth Platform
Jaswinder Singh
 
PDF
The Rise of AI and IoT in Mobile App Tech.pdf
IMG Global Infotech
 
PPTX
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
PPTX
Building Search Using OpenSearch: Limitations and Workarounds
Sease
 
PDF
POV_ Why Enterprises Need to Find Value in ZERO.pdf
darshakparmar
 
PDF
Presentation - Vibe Coding The Future of Tech
yanuarsinggih1
 
PDF
Biography of Daniel Podor.pdf
Daniel Podor
 
PPTX
COMPARISON OF RASTER ANALYSIS TOOLS OF QGIS AND ARCGIS
Sharanya Sarkar
 
PDF
Blockchain Transactions Explained For Everyone
CIFDAQ
 
PPTX
OpenID AuthZEN - Analyst Briefing July 2025
David Brossard
 
PPTX
AUTOMATION AND ROBOTICS IN PHARMA INDUSTRY.pptx
sameeraaabegumm
 
CIFDAQ Market Wrap for the week of 4th July 2025
CIFDAQ
 
WooCommerce Workshop: Bring Your Laptop
Laura Hartwig
 
Agentic AI lifecycle for Enterprise Hyper-Automation
Debmalya Biswas
 
Exolore The Essential AI Tools in 2025.pdf
Srinivasan M
 
From Code to Challenge: Crafting Skill-Based Games That Engage and Reward
aiyshauae
 
NewMind AI - Journal 100 Insights After The 100th Issue
NewMind AI
 
July Patch Tuesday
Ivanti
 
New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
What Makes Contify’s News API Stand Out: Key Features at a Glance
Contify
 
HubSpot Main Hub: A Unified Growth Platform
Jaswinder Singh
 
The Rise of AI and IoT in Mobile App Tech.pdf
IMG Global Infotech
 
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
Building Search Using OpenSearch: Limitations and Workarounds
Sease
 
POV_ Why Enterprises Need to Find Value in ZERO.pdf
darshakparmar
 
Presentation - Vibe Coding The Future of Tech
yanuarsinggih1
 
Biography of Daniel Podor.pdf
Daniel Podor
 
COMPARISON OF RASTER ANALYSIS TOOLS OF QGIS AND ARCGIS
Sharanya Sarkar
 
Blockchain Transactions Explained For Everyone
CIFDAQ
 
OpenID AuthZEN - Analyst Briefing July 2025
David Brossard
 
AUTOMATION AND ROBOTICS IN PHARMA INDUSTRY.pptx
sameeraaabegumm
 

Taming Pythons with ZooKeeper