SlideShare a Scribd company logo

Building a Data Subscription Service with Kafka Connect (Danica Fine & Ajay Vyasapeetam, Bloomberg) Kafka Summit 2020

confluent
confluent

The document discusses building a data subscription service using Apache Kafka Connect. It describes motivating factors like easy subscription to multiple data sources with consistent interfaces. An ideal system architecture is presented using a subscription manager and custom source connectors. Details are provided on building a source connector in Kafka Connect, including components, source connector tasks, and monitoring threads. Challenges around reconfiguration and testing connectors are also noted.

Technology
1 of 18
Downloaded 19 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
© 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Building a Data Subscription Service with Kafka Connect Kafka Summit 2020 24 August 2020 Danica Fine, Software Engineer Ajay Vyasapeetam, Software Engineer
© 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Use Case
© 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Non-Ideal Architecture
© 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Ideal Architecture
© 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Motivation • Easy subscription to multiple data sources • Consistent interfaces § Subscribing § Receiving data • Optimize reuse across applications • Fault-tolerant and scalable
© 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. System Architecture
© 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Subscription Manager
© 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Side Input Data Connector
© 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Building a Custom Source Connector
© 2018 Bloomberg Finance L.P. All rights reserved. Kafka Connect Basics
© 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Connect Components • Connector • Tasks • Monitor Thread
© 2018 Bloomberg Finance L.P. All rights reserved. Source Connector • Defines task breakdown • Kicks off monitor thread
© 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Source Connector Task • Defines: § Where to collect data § How to format it • Client for external data call
© 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Monitoring Thread • Consumes from Kafka topics • Triggers task reconfiguration
© 2018 Bloomberg Finance L.P. All rights reserved. Final Workflow
© 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Pitfalls • Kafka Connect stop-the-world reconfiguration • Consumers in monitor thread vs tasks • Testing connectors
© 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Questions? Danica Fine, dfine17@bloomberg.net Ajay Vyasapeetam, avyasapeeta1@bloomberg.net
© 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. We are hiring! Thank you! https://www.bloomberg.com/careers

More Related Content

What's hot (20)

PPTX
Kafka Connect and KSQL: Useful Tools in Migrating from a Legacy System to Kaf...
confluent
 
PDF
Building Stateful applications on Streaming Platforms | Premjit Mishra, Dell ...
HostedbyConfluent
 
PDF
Cisco’s E-Commerce Transformation Using Kafka
confluent
 
PDF
Kafka for connected vehicle research | Pavle Bujanovic, Federal Highway Admin...
HostedbyConfluent
 
PPTX
Apache Kafka Architectures and Fundamentals
confluent
 
PPTX
Comparing three data ingestion approaches where Apache Kafka integrates with ...
HostedbyConfluent
 
PDF
Death of the dumb pipes: Using Apache Kafka® for Integration projects
HostedbyConfluent
 
PDF
Transform Your Mainframe and IBM i Data for the Cloud with Precisely and Apac...
HostedbyConfluent
 
PPTX
Should we manage events like APIs? | Kim Clark, IBM
HostedbyConfluent
 
PDF
Elastically Scaling Kafka Using Confluent
confluent
 
PDF
Mesh-ing around with Streams across the Enterprise | Phil Scanlon, Solace
HostedbyConfluent
 
PDF
Application Modernization Using Event Streaming Architecture (David Wadden, V...
HostedbyConfluent
 
PDF
Government Track Welcome Address
HostedbyConfluent
 
PDF
Why Kafka Works the Way It Does (And Not Some Other Way) | Tim Berglund, Conf...
HostedbyConfluent
 
PPTX
Using Kafka Streams to Analyze Live Trading Activity for Crypto Exchanges (Lu...
confluent
 
PDF
Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...
confluent
 
PDF
Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...
HostedbyConfluent
 
PDF
Apache Kafka and the Data Mesh | Michael Noll, Confluent
HostedbyConfluent
 
PPTX
Confluent Private Cloud | Rohit Bakhshi, Staff Product Manager
HostedbyConfluent
 
PDF
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
HostedbyConfluent
 
Kafka Connect and KSQL: Useful Tools in Migrating from a Legacy System to Kaf...
confluent
 
Building Stateful applications on Streaming Platforms | Premjit Mishra, Dell ...
HostedbyConfluent
 
Cisco’s E-Commerce Transformation Using Kafka
confluent
 
Kafka for connected vehicle research | Pavle Bujanovic, Federal Highway Admin...
HostedbyConfluent
 
Apache Kafka Architectures and Fundamentals
confluent
 
Comparing three data ingestion approaches where Apache Kafka integrates with ...
HostedbyConfluent
 
Death of the dumb pipes: Using Apache Kafka® for Integration projects
HostedbyConfluent
 
Transform Your Mainframe and IBM i Data for the Cloud with Precisely and Apac...
HostedbyConfluent
 
Should we manage events like APIs? | Kim Clark, IBM
HostedbyConfluent
 
Elastically Scaling Kafka Using Confluent
confluent
 
Mesh-ing around with Streams across the Enterprise | Phil Scanlon, Solace
HostedbyConfluent
 
Application Modernization Using Event Streaming Architecture (David Wadden, V...
HostedbyConfluent
 
Government Track Welcome Address
HostedbyConfluent
 
Why Kafka Works the Way It Does (And Not Some Other Way) | Tim Berglund, Conf...
HostedbyConfluent
 
Using Kafka Streams to Analyze Live Trading Activity for Crypto Exchanges (Lu...
confluent
 
Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...
confluent
 
Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...
HostedbyConfluent
 
Apache Kafka and the Data Mesh | Michael Noll, Confluent
HostedbyConfluent
 
Confluent Private Cloud | Rohit Bakhshi, Staff Product Manager
HostedbyConfluent
 
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
HostedbyConfluent
 

Similar to Building a Data Subscription Service with Kafka Connect (Danica Fine & Ajay Vyasapeetam, Bloomberg) Kafka Summit 2020 (16)

PDF
The Power of Event Driven Caches (Brendan Powers, Bloomberg L.P) Kafka Summit...
confluent
 
PDF
Real-time Adaptation of Financial Market Events with Kafka | Cliff Cheng and ...
HostedbyConfluent
 
PPTX
VOGIN-IP-lezing-edgar_meij
voginip
 
PDF
Verifying Apache Kafka-Based Data Pipelines With Subhangi Agarwala | Current ...
HostedbyConfluent
 
PDF
HBase Internals And Operations
Biju Nair
 
PPT
Bloomberg LLC
bechina
 
PDF
Eventos y Microservicios - Santander TechTalk
confluent
 
PDF
Confluent & GSI Webinars series: Session 2
confluent
 
PDF
Confluent & GSI Webinars series - Session 3
confluent
 
PDF
Mesoscon 2015
Skand Gupta
 
PDF
Apache Kafka® Use Cases for Financial Services
confluent
 
PDF
Slides: Why You Need End-to-End Data Quality to Build Trust in Kafka
DATAVERSITY
 
PPTX
Bloomberg
Pramod Jindal
 
PPTX
Dynamic Rule-based Real-time Market Data Alerts
Flink Forward
 
PPTX
Brooklin Mirror Maker - How and why we moved away from Kafka Mirror Maker
Shun-ping Chiu
 
PDF
Spark and Online Analytics: Spark Summit East talky by Shubham Chopra
Spark Summit
 
The Power of Event Driven Caches (Brendan Powers, Bloomberg L.P) Kafka Summit...
confluent
 
Real-time Adaptation of Financial Market Events with Kafka | Cliff Cheng and ...
HostedbyConfluent
 
VOGIN-IP-lezing-edgar_meij
voginip
 
Verifying Apache Kafka-Based Data Pipelines With Subhangi Agarwala | Current ...
HostedbyConfluent
 
HBase Internals And Operations
Biju Nair
 
Bloomberg LLC
bechina
 
Eventos y Microservicios - Santander TechTalk
confluent
 
Confluent & GSI Webinars series: Session 2
confluent
 
Confluent & GSI Webinars series - Session 3
confluent
 
Mesoscon 2015
Skand Gupta
 
Apache Kafka® Use Cases for Financial Services
confluent
 
Slides: Why You Need End-to-End Data Quality to Build Trust in Kafka
DATAVERSITY
 
Bloomberg
Pramod Jindal
 
Dynamic Rule-based Real-time Market Data Alerts
Flink Forward
 
Brooklin Mirror Maker - How and why we moved away from Kafka Mirror Maker
Shun-ping Chiu
 
Spark and Online Analytics: Spark Summit East talky by Shubham Chopra
Spark Summit
 
Ad

More from confluent (20)

PDF
Stream Processing Handson Workshop - Flink SQL Hands-on Workshop (Korean)
confluent
 
PPTX
Webinar Think Right - Shift Left - 19-03-2025.pptx
confluent
 
PDF
Migration, backup and restore made easy using Kannika
confluent
 
PDF
Five Things You Need to Know About Data Streaming in 2025
confluent
 
PDF
Data in Motion Tour Seoul 2024 - Keynote
confluent
 
PDF
Data in Motion Tour Seoul 2024 - Roadmap Demo
confluent
 
PDF
From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...
confluent
 
PDF
Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...
confluent
 
PDF
Data in Motion Tour 2024 Riyadh, Saudi Arabia
confluent
 
PDF
Build a Real-Time Decision Support Application for Financial Market Traders w...
confluent
 
PDF
Strumenti e Strategie di Stream Governance con Confluent Platform
confluent
 
PDF
Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeks
confluent
 
PDF
Building Real-Time Gen AI Applications with SingleStore and Confluent
confluent
 
PDF
Unlocking value with event-driven architecture by Confluent
confluent
 
PDF
Il Data Streaming per un’AI real-time di nuova generazione
confluent
 
PDF
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
confluent
 
PDF
Break data silos with real-time connectivity using Confluent Cloud Connectors
confluent
 
PDF
Building API data products on top of your real-time data infrastructure
confluent
 
PDF
Speed Wins: From Kafka to APIs in Minutes
confluent
 
PDF
Evolving Data Governance for the Real-time Streaming and AI Era
confluent
 
Stream Processing Handson Workshop - Flink SQL Hands-on Workshop (Korean)
confluent
 
Webinar Think Right - Shift Left - 19-03-2025.pptx
confluent
 
Migration, backup and restore made easy using Kannika
confluent
 
Five Things You Need to Know About Data Streaming in 2025
confluent
 
Data in Motion Tour Seoul 2024 - Keynote
confluent
 
Data in Motion Tour Seoul 2024 - Roadmap Demo
confluent
 
From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...
confluent
 
Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...
confluent
 
Data in Motion Tour 2024 Riyadh, Saudi Arabia
confluent
 
Build a Real-Time Decision Support Application for Financial Market Traders w...
confluent
 
Strumenti e Strategie di Stream Governance con Confluent Platform
confluent
 
Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeks
confluent
 
Building Real-Time Gen AI Applications with SingleStore and Confluent
confluent
 
Unlocking value with event-driven architecture by Confluent
confluent
 
Il Data Streaming per un’AI real-time di nuova generazione
confluent
 
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
confluent
 
Break data silos with real-time connectivity using Confluent Cloud Connectors
confluent
 
Building API data products on top of your real-time data infrastructure
confluent
 
Speed Wins: From Kafka to APIs in Minutes
confluent
 
Evolving Data Governance for the Real-time Streaming and AI Era
confluent
 
Ad

Recently uploaded (20)

PDF
Wojciech Ciemski for Top Cyber News MAGAZINE. June 2025
Dr. Ludmila Morozova-Buss
 
PDF
CIFDAQ Token Spotlight for 9th July 2025
CIFDAQ
 
PDF
Exolore The Essential AI Tools in 2025.pdf
Srinivasan M
 
PDF
Building Real-Time Digital Twins with IBM Maximo & ArcGIS Indoors
Safe Software
 
PDF
Fl Studio 24.2.2 Build 4597 Crack for Windows Free Download 2025
faizk77g
 
PDF
The Builder’s Playbook - 2025 State of AI Report.pdf
jeroen339954
 
PDF
HubSpot Main Hub: A Unified Growth Platform
Jaswinder Singh
 
PDF
NewMind AI Journal - Weekly Chronicles - July'25 Week II
NewMind AI
 
PDF
LLMs.txt: Easily Control How AI Crawls Your Site
Keploy
 
PDF
Presentation - Vibe Coding The Future of Tech
yanuarsinggih1
 
PDF
July Patch Tuesday
Ivanti
 
PDF
New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
PDF
Log-Based Anomaly Detection: Enhancing System Reliability with Machine Learning
Mohammed BEKKOUCHE
 
PDF
Achieving Consistent and Reliable AI Code Generation - Medusa AI
medusaaico
 
PDF
Building Resilience with Digital Twins : Lessons from Korea
SANGHEE SHIN
 
PDF
SFWelly Summer 25 Release Highlights July 2025
Anna Loughnan Colquhoun
 
PDF
Impact of IEEE Computer Society in Advancing Emerging Technologies including ...
Hironori Washizaki
 
PDF
Complete JavaScript Notes: From Basics to Advanced Concepts.pdf
haydendavispro
 
PDF
How Startups Are Growing Faster with App Developers in Australia.pdf
India App Developer
 
PDF
Chris Elwell Woburn, MA - Passionate About IT Innovation
Chris Elwell Woburn, MA
 
Wojciech Ciemski for Top Cyber News MAGAZINE. June 2025
Dr. Ludmila Morozova-Buss
 
CIFDAQ Token Spotlight for 9th July 2025
CIFDAQ
 
Exolore The Essential AI Tools in 2025.pdf
Srinivasan M
 
Building Real-Time Digital Twins with IBM Maximo & ArcGIS Indoors
Safe Software
 
Fl Studio 24.2.2 Build 4597 Crack for Windows Free Download 2025
faizk77g
 
The Builder’s Playbook - 2025 State of AI Report.pdf
jeroen339954
 
HubSpot Main Hub: A Unified Growth Platform
Jaswinder Singh
 
NewMind AI Journal - Weekly Chronicles - July'25 Week II
NewMind AI
 
LLMs.txt: Easily Control How AI Crawls Your Site
Keploy
 
Presentation - Vibe Coding The Future of Tech
yanuarsinggih1
 
July Patch Tuesday
Ivanti
 
New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
Log-Based Anomaly Detection: Enhancing System Reliability with Machine Learning
Mohammed BEKKOUCHE
 
Achieving Consistent and Reliable AI Code Generation - Medusa AI
medusaaico
 
Building Resilience with Digital Twins : Lessons from Korea
SANGHEE SHIN
 
SFWelly Summer 25 Release Highlights July 2025
Anna Loughnan Colquhoun
 
Impact of IEEE Computer Society in Advancing Emerging Technologies including ...
Hironori Washizaki
 
Complete JavaScript Notes: From Basics to Advanced Concepts.pdf
haydendavispro
 
How Startups Are Growing Faster with App Developers in Australia.pdf
India App Developer
 
Chris Elwell Woburn, MA - Passionate About IT Innovation
Chris Elwell Woburn, MA
 

Building a Data Subscription Service with Kafka Connect (Danica Fine & Ajay Vyasapeetam, Bloomberg) Kafka Summit 2020

  • 1. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Building a Data Subscription Service with Kafka Connect Kafka Summit 2020 24 August 2020 Danica Fine, Software Engineer Ajay Vyasapeetam, Software Engineer
  • 2. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Use Case
  • 3. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Non-Ideal Architecture
  • 4. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Ideal Architecture
  • 5. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Motivation • Easy subscription to multiple data sources • Consistent interfaces § Subscribing § Receiving data • Optimize reuse across applications • Fault-tolerant and scalable
  • 6. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. System Architecture
  • 7. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Subscription Manager
  • 8. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Side Input Data Connector
  • 9. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Building a Custom Source Connector
  • 10. © 2018 Bloomberg Finance L.P. All rights reserved. Kafka Connect Basics
  • 11. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Connect Components • Connector • Tasks • Monitor Thread
  • 12. © 2018 Bloomberg Finance L.P. All rights reserved. Source Connector • Defines task breakdown • Kicks off monitor thread
  • 13. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Source Connector Task • Defines: § Where to collect data § How to format it • Client for external data call
  • 14. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Monitoring Thread • Consumes from Kafka topics • Triggers task reconfiguration
  • 15. © 2018 Bloomberg Finance L.P. All rights reserved. Final Workflow
  • 16. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Pitfalls • Kafka Connect stop-the-world reconfiguration • Consumers in monitor thread vs tasks • Testing connectors
  • 17. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Questions? Danica Fine, [email protected] Ajay Vyasapeetam, [email protected]
  • 18. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. We are hiring! Thank you! https://www.bloomberg.com/careers