SlideShare a Scribd company logo
#DevoxxFR
Hand’s on
Kafka : http://kafka.apache.org/downloads.html
Hand’s on : https://github.com/mblanc/hands_on_kafka.git
#DevoxxFR
@matthieublanc
@slequeux
Matthieu Blanc
Sylvain Lequeux
#DevoxxFR
Messaging System?
#DevoxxFR
Jay Kreps
Neha Narkhede
Jun Rao
History
#DevoxxFR
WebAppRelational
DB
NoSQL
DB
DWH
Hadoop
Monitoring Logs
#DevoxxFR
WebApp
Relational
DB
NoSQL
DB
DWH
Hadoop
ActiveMQ
WebApp
Logs
Monitoring
WebApp
Search
Big Data?
#DevoxxFR
WebApp
Relational
DB
NoSQL
DB
DWH
Hadoop
ActiveMQ
WebApp
Logs
Monitoring
WebApp
Search
BIG
MESS!
#DevoxxFR
Stream Data Platform
#DevoxxFR
● Decoupling Systems
● High throughput
● Distributed - Horizontal scaling
● Multi consumers
● Persistence
● Automatic recovery from broker failure
Features
#DevoxxFR
● Cost
● Persistence
● Batch system -> perfs down
● Large scale stream processing
● Ordering guarantees
RabbitMQ/ActiveMQ?
#DevoxxFR
Consumer
Broker
Consumer
Consumer
Kafka Cluster
Broker Broker
Broker Broker Broker
Zookeeper
Producer
Producer
Producer
Architecture
#DevoxxFR
Distributed Commit Logs
10 11 12 13 14 15 16 17 18987654321 19
1st record
Next record
Written
Reads
(sequential access = high performance)
#DevoxxFR
Producer
10 11 12 13 14 15 16 17 18987654321
10 11 12 13 14987654321 15
10 11 12 13 14 15987654321 16
Partition #1
Partition #2
Partition #3
ProducerProducer
19
16
17
offset
Old New
Writes
Writes
Writes
message :
(key bytes[ ], value bytes[ ])
#DevoxxFR
Topic storage
10 11 12 13 14 15 16 17 18987654321
Partition #1
directory segment = file
#DevoxxFR
Fast
● Sequential Access
● PageCache
● Linux : sendfile()
● Compression
Source : http://queue.acm.org/detail.cfm?id=1563874
#DevoxxFR
Fast
#DevoxxFR
Consumer Group
10 11 12 13 14 15 16 17 18987654321
10 11 12 13 14987654321 15
10 11 12 13 14 15987654321 16
19
16
17
Producer
Consumer
Group A
Consumer
Group A
Consumer
Group A
Consumer
Group B
Consumer
Group B
Partition #1
Partition #2
Partition #3
Writes
Consumption
#DevoxxFR
Fault tolerant consumption
10 11 12 13 14 15 16 17 18987654321
10 11 12 13 14987654321 15
10 11 12 13 14 15987654321 16
19
16
17
Producer
Consumer
Group A
Consumer
Group A
Consumer
Group A
Consumer
Group B
Consumer
Group B
Partition #1
Partition #2
Partition #3
Writes
Automatic
rebalancing
on failure
#DevoxxFR
Consumer group
10 11 12 13 14 15 16 17 18987654321
10 11 12 13 14987654321 15
10 11 12 13 14 15987654321 16
Partition #1
Partition #2
Partition #3
Group Topic # Offset
1 log 1 18
1 log 2 12
1 log 3 14
2 log 1 1
2 log 2 0
2 log 3 3
Consumer group 2 Consumer group 1
Old New
#DevoxxFR
Replicas/ISRs
Partition #0
Partition #2
Topic : foo
Partitions : 3
Replicas : 3
Partition #1
Partition #0
Partition #2
Producer
Broker #0 Broker #1 Broker #2
Writes
Consumer
Leader
Leader
Leader
Partition #1
Partition #2
Partition #0
Partition #1
#DevoxxFR
Ka ka 0.9 - New Consumer
● Unified consumer API
● Much simpler and thinner
● Allows for larger groups with far faster
rebalancing
● Decouple Kafka clients from Zookeeper!!!
#DevoxxFR
Security
● Authentication : Kerberos / TLS certificate
● Authorization : unix-like permissions
system
● Encryption on the wire : SSL
● Encryption at rest : encrypting individual
fields / filesystem security features
● User defined quota
#DevoxxFR
Ka ka Connect
KafkaConnect
Data
Source
KafkaConnect
Data
Sink
Kafka
#DevoxxFR
Ka ka Streams
KafkaConnect
Data
Source
KafkaConnect
Data
Sink
Kafka
Kafka
Streams
#DevoxxFR
Jay Kreps
Neha Narkhede
Jun Rao
Ka ka Enterprise Ready
2011 2012
2014
#DevoxxFR
● User behaviour, click stream analysis
● Infrastructure monitoring and security
● Telemetry data from mobile/sensors
● IoT
● Log analysis
● ...
Use cases
#DevoxxFR
Used by
● LinkedIn : activity stream, metrics
● Netflix : Real-time Monitoring
● Twitter : Real-time data pipeline
● Spotify : log delivery
● Loggly : log collection and processing
● Mozilla : telemetry data
● Microsoft : Ads, Bing, Office
● Airbnb, Square, Uber, Criteo, OVH ...
#DevoxxFR
GL HF !
● Download Kafka : http://kafka.apache.org/downloads.html
● Git Clone : https://github.com/mblanc/hands_on_kafka.git
● Open : reveal.js/index_java.html

More Related Content

What's hot (20)

PDF
Fundamentals of Apache Kafka
Chhavi Parasher
 
PDF
A la rencontre de Kafka, le log distribué par Florian GARCIA
La Cuisine du Web
 
PDF
Stranger Things: The Forces that Disrupt Netflix
C4Media
 
PPTX
Luca Relandini - Microservices and containers networking: Contiv, deep dive a...
Codemotion
 
PDF
The Easiest Way to Configure Security for Clients AND Servers (Dani Traphagen...
confluent
 
PDF
Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...
HostedbyConfluent
 
PDF
KubeCon EU 2016: Kubernetes meets Finagle for Resilient Microservices
KubeAcademy
 
PDF
Mattia Gandolfi - Improving utilization and portability with Containers and C...
Codemotion
 
PPTX
Gatekeeper: API gateway
ChengHui Weng
 
PDF
Crafting Kubernetes Operators
Red Hat Developers
 
PDF
Power of the Log: LSM & Append Only Data Structures
confluent
 
PDF
Load Balancing in the Cloud using Nginx & Kubernetes
Lee Calcote
 
PPTX
Streaming in Practice - Putting Apache Kafka in Production
confluent
 
PDF
Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...
confluent
 
PDF
Single tenant software to multi-tenant SaaS using K8S
CloudLinux
 
PDF
Deploying Flink on Kubernetes - David Anderson
Ververica
 
PDF
Fabric8 - Being devOps doesn't suck anymore
Henryk Konsek
 
PDF
Spring Kafka beyond the basics - Lessons learned on our Kafka journey (Tim va...
confluent
 
PPTX
Kubernetes 1.16 and rancher 2.3 enhancements
Saiyam Pathak
 
PDF
What's New in Docker 1.12 (June 20, 2016) by Mike Goelzer & Andrea Luzzardi
Mike Goelzer
 
Fundamentals of Apache Kafka
Chhavi Parasher
 
A la rencontre de Kafka, le log distribué par Florian GARCIA
La Cuisine du Web
 
Stranger Things: The Forces that Disrupt Netflix
C4Media
 
Luca Relandini - Microservices and containers networking: Contiv, deep dive a...
Codemotion
 
The Easiest Way to Configure Security for Clients AND Servers (Dani Traphagen...
confluent
 
Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...
HostedbyConfluent
 
KubeCon EU 2016: Kubernetes meets Finagle for Resilient Microservices
KubeAcademy
 
Mattia Gandolfi - Improving utilization and portability with Containers and C...
Codemotion
 
Gatekeeper: API gateway
ChengHui Weng
 
Crafting Kubernetes Operators
Red Hat Developers
 
Power of the Log: LSM & Append Only Data Structures
confluent
 
Load Balancing in the Cloud using Nginx & Kubernetes
Lee Calcote
 
Streaming in Practice - Putting Apache Kafka in Production
confluent
 
Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...
confluent
 
Single tenant software to multi-tenant SaaS using K8S
CloudLinux
 
Deploying Flink on Kubernetes - David Anderson
Ververica
 
Fabric8 - Being devOps doesn't suck anymore
Henryk Konsek
 
Spring Kafka beyond the basics - Lessons learned on our Kafka journey (Tim va...
confluent
 
Kubernetes 1.16 and rancher 2.3 enhancements
Saiyam Pathak
 
What's New in Docker 1.12 (June 20, 2016) by Mike Goelzer & Andrea Luzzardi
Mike Goelzer
 

Similar to Devoxx fr 2016 - Apache Kafka - Stream Data Platform (20)

PDF
Demystifying Kafka
nklmish
 
PDF
DevoxxFR 2016 - 3 degrees of MoM
Guillaume Arnaud
 
PPTX
How is Kafka so Fast?
Ricardo Paiva
 
PPTX
Apache kafka
Degendra Sivakoti
 
PDF
14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...
Athens Big Data
 
PDF
2015-04-15 | Apache Kafka (Vienna Scala User Group)
Dominik Gruber
 
PPTX
Apache kafka
Kumar Shivam
 
PPTX
Building Event-Driven Systems with Apache Kafka
Brian Ritchie
 
PPTX
Westpac Bank Tech Talk 1: Dive into Apache Kafka
confluent
 
PPTX
Large scale, distributed and reliable messaging with Kafka
Rafał Hryniewski
 
PPTX
Copy of Kafka-Camus
Deep Shah
 
PDF
Show Me Kafka Tools That Will Increase My Productivity! (Stephane Maarek, Dat...
confluent
 
PPTX
Apache Kafka with Spark Streaming: Real-time Analytics Redefined
Edureka!
 
PPTX
Current and Future of Apache Kafka
Joe Stein
 
PDF
Apache Kafka - Event Sourcing, Monitoring, Librdkafka, Scaling & Partitioning
Guido Schmutz
 
PDF
Enabling Data Scientists to easily create and own Kafka Consumers | Stefan Kr...
HostedbyConfluent
 
PDF
Enabling Data Scientists to easily create and own Kafka Consumers
Stefan Krawczyk
 
PPTX
Apache Kafka 0.8 basic training - Verisign
Michael Noll
 
PPTX
Distributed messaging through Kafka
Dileep Kalidindi
 
PDF
Apache Kafka - Scalable Message Processing and more!
Guido Schmutz
 
Demystifying Kafka
nklmish
 
DevoxxFR 2016 - 3 degrees of MoM
Guillaume Arnaud
 
How is Kafka so Fast?
Ricardo Paiva
 
Apache kafka
Degendra Sivakoti
 
14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...
Athens Big Data
 
2015-04-15 | Apache Kafka (Vienna Scala User Group)
Dominik Gruber
 
Apache kafka
Kumar Shivam
 
Building Event-Driven Systems with Apache Kafka
Brian Ritchie
 
Westpac Bank Tech Talk 1: Dive into Apache Kafka
confluent
 
Large scale, distributed and reliable messaging with Kafka
Rafał Hryniewski
 
Copy of Kafka-Camus
Deep Shah
 
Show Me Kafka Tools That Will Increase My Productivity! (Stephane Maarek, Dat...
confluent
 
Apache Kafka with Spark Streaming: Real-time Analytics Redefined
Edureka!
 
Current and Future of Apache Kafka
Joe Stein
 
Apache Kafka - Event Sourcing, Monitoring, Librdkafka, Scaling & Partitioning
Guido Schmutz
 
Enabling Data Scientists to easily create and own Kafka Consumers | Stefan Kr...
HostedbyConfluent
 
Enabling Data Scientists to easily create and own Kafka Consumers
Stefan Krawczyk
 
Apache Kafka 0.8 basic training - Verisign
Michael Noll
 
Distributed messaging through Kafka
Dileep Kalidindi
 
Apache Kafka - Scalable Message Processing and more!
Guido Schmutz
 
Ad

More from Publicis Sapient Engineering (20)

PDF
XebiCon'18 - L'algorithme de reconnaissance de formes par le cerveau humain
Publicis Sapient Engineering
 
PDF
Xebicon'18 - IoT: From Edge to Cloud
Publicis Sapient Engineering
 
PDF
Xebicon'18 - Spark in jail : conteneurisez vos traitements data sans serveur
Publicis Sapient Engineering
 
PDF
XebiCon'18 - Modern Infrastructure
Publicis Sapient Engineering
 
PDF
XebiCon'18 - La Web App d'aujourd'hui et de demain : état de l'art et bleedin...
Publicis Sapient Engineering
 
PDF
XebiCon'18 - Des notebook pour le monitoring avec Zeppelin
Publicis Sapient Engineering
 
PDF
XebiCon'18 - Event Sourcing et RGPD, incompatibles ?
Publicis Sapient Engineering
 
PDF
XebiCon'18 - Deno, le nouveau NodeJS qui inverse la tendance ?
Publicis Sapient Engineering
 
PDF
XebiCon'18 - Boostez vos modèles avec du Deep Learning distribué
Publicis Sapient Engineering
 
PDF
XebiCon'18 - Comment j'ai développé un jeu vidéo avec des outils de développe...
Publicis Sapient Engineering
 
PDF
XebiCon'18 - Les utilisateurs finaux, les oubliés de nos produits !
Publicis Sapient Engineering
 
PDF
XebiCon'18 - Comment fausser l'interprétation de vos résultats avec des dataviz
Publicis Sapient Engineering
 
PDF
XebiCon'18 - Le développeur dans la Pop Culture
Publicis Sapient Engineering
 
PDF
XebiCon'18 - Architecturer son application mobile pour la durabilité
Publicis Sapient Engineering
 
PDF
XebiCon'18 - Sécuriser son API avec OpenID Connect
Publicis Sapient Engineering
 
PDF
XebiCon'18 - Structuration du Temps et Dynamique de Groupes, Théorie organisa...
Publicis Sapient Engineering
 
PDF
XebiCon'18 - Spark NLP, un an après
Publicis Sapient Engineering
 
PDF
XebiCon'18 - La sécurité, douce illusion même en 2018
Publicis Sapient Engineering
 
PDF
XebiCon'18 - Utiliser Hyperledger Fabric pour la création d'une blockchain pr...
Publicis Sapient Engineering
 
PDF
XebiCon'18 - Ce que l'histoire du métro Parisien m'a enseigné sur la création...
Publicis Sapient Engineering
 
XebiCon'18 - L'algorithme de reconnaissance de formes par le cerveau humain
Publicis Sapient Engineering
 
Xebicon'18 - IoT: From Edge to Cloud
Publicis Sapient Engineering
 
Xebicon'18 - Spark in jail : conteneurisez vos traitements data sans serveur
Publicis Sapient Engineering
 
XebiCon'18 - Modern Infrastructure
Publicis Sapient Engineering
 
XebiCon'18 - La Web App d'aujourd'hui et de demain : état de l'art et bleedin...
Publicis Sapient Engineering
 
XebiCon'18 - Des notebook pour le monitoring avec Zeppelin
Publicis Sapient Engineering
 
XebiCon'18 - Event Sourcing et RGPD, incompatibles ?
Publicis Sapient Engineering
 
XebiCon'18 - Deno, le nouveau NodeJS qui inverse la tendance ?
Publicis Sapient Engineering
 
XebiCon'18 - Boostez vos modèles avec du Deep Learning distribué
Publicis Sapient Engineering
 
XebiCon'18 - Comment j'ai développé un jeu vidéo avec des outils de développe...
Publicis Sapient Engineering
 
XebiCon'18 - Les utilisateurs finaux, les oubliés de nos produits !
Publicis Sapient Engineering
 
XebiCon'18 - Comment fausser l'interprétation de vos résultats avec des dataviz
Publicis Sapient Engineering
 
XebiCon'18 - Le développeur dans la Pop Culture
Publicis Sapient Engineering
 
XebiCon'18 - Architecturer son application mobile pour la durabilité
Publicis Sapient Engineering
 
XebiCon'18 - Sécuriser son API avec OpenID Connect
Publicis Sapient Engineering
 
XebiCon'18 - Structuration du Temps et Dynamique de Groupes, Théorie organisa...
Publicis Sapient Engineering
 
XebiCon'18 - Spark NLP, un an après
Publicis Sapient Engineering
 
XebiCon'18 - La sécurité, douce illusion même en 2018
Publicis Sapient Engineering
 
XebiCon'18 - Utiliser Hyperledger Fabric pour la création d'une blockchain pr...
Publicis Sapient Engineering
 
XebiCon'18 - Ce que l'histoire du métro Parisien m'a enseigné sur la création...
Publicis Sapient Engineering
 
Ad

Recently uploaded (20)

PDF
Salesforce CRM Services.VALiNTRY360
VALiNTRY360
 
PDF
Alexander Marshalov - How to use AI Assistants with your Monitoring system Q2...
VictoriaMetrics
 
PDF
Capcut Pro Crack For PC Latest Version {Fully Unlocked} 2025
hashhshs786
 
PDF
Understanding the Need for Systemic Change in Open Source Through Intersectio...
Imma Valls Bernaus
 
PDF
Mobile CMMS Solutions Empowering the Frontline Workforce
CryotosCMMSSoftware
 
PDF
Beyond Binaries: Understanding Diversity and Allyship in a Global Workplace -...
Imma Valls Bernaus
 
PPTX
Comprehensive Guide: Shoviv Exchange to Office 365 Migration Tool 2025
Shoviv Software
 
PDF
Unlock Efficiency with Insurance Policy Administration Systems
Insurance Tech Services
 
PPTX
Platform for Enterprise Solution - Java EE5
abhishekoza1981
 
PPTX
Tally_Basic_Operations_Presentation.pptx
AditiBansal54083
 
PPTX
3uTools Full Crack Free Version Download [Latest] 2025
muhammadgurbazkhan
 
PPTX
The Role of a PHP Development Company in Modern Web Development
SEO Company for School in Delhi NCR
 
PDF
Automate Cybersecurity Tasks with Python
VICTOR MAESTRE RAMIREZ
 
PDF
Powering GIS with FME and VertiGIS - Peak of Data & AI 2025
Safe Software
 
PPTX
Agentic Automation Journey Session 1/5: Context Grounding and Autopilot for E...
klpathrudu
 
PPTX
Fundamentals_of_Microservices_Architecture.pptx
MuhammadUzair504018
 
PDF
Streamline Contractor Lifecycle- TECH EHS Solution
TECH EHS Solution
 
DOCX
Import Data Form Excel to Tally Services
Tally xperts
 
PDF
Odoo CRM vs Zoho CRM: Honest Comparison 2025
Odiware Technologies Private Limited
 
PPT
MergeSortfbsjbjsfk sdfik k
RafishaikIT02044
 
Salesforce CRM Services.VALiNTRY360
VALiNTRY360
 
Alexander Marshalov - How to use AI Assistants with your Monitoring system Q2...
VictoriaMetrics
 
Capcut Pro Crack For PC Latest Version {Fully Unlocked} 2025
hashhshs786
 
Understanding the Need for Systemic Change in Open Source Through Intersectio...
Imma Valls Bernaus
 
Mobile CMMS Solutions Empowering the Frontline Workforce
CryotosCMMSSoftware
 
Beyond Binaries: Understanding Diversity and Allyship in a Global Workplace -...
Imma Valls Bernaus
 
Comprehensive Guide: Shoviv Exchange to Office 365 Migration Tool 2025
Shoviv Software
 
Unlock Efficiency with Insurance Policy Administration Systems
Insurance Tech Services
 
Platform for Enterprise Solution - Java EE5
abhishekoza1981
 
Tally_Basic_Operations_Presentation.pptx
AditiBansal54083
 
3uTools Full Crack Free Version Download [Latest] 2025
muhammadgurbazkhan
 
The Role of a PHP Development Company in Modern Web Development
SEO Company for School in Delhi NCR
 
Automate Cybersecurity Tasks with Python
VICTOR MAESTRE RAMIREZ
 
Powering GIS with FME and VertiGIS - Peak of Data & AI 2025
Safe Software
 
Agentic Automation Journey Session 1/5: Context Grounding and Autopilot for E...
klpathrudu
 
Fundamentals_of_Microservices_Architecture.pptx
MuhammadUzair504018
 
Streamline Contractor Lifecycle- TECH EHS Solution
TECH EHS Solution
 
Import Data Form Excel to Tally Services
Tally xperts
 
Odoo CRM vs Zoho CRM: Honest Comparison 2025
Odiware Technologies Private Limited
 
MergeSortfbsjbjsfk sdfik k
RafishaikIT02044
 

Devoxx fr 2016 - Apache Kafka - Stream Data Platform