SlideShare a Scribd company logo
Apache Kafka Meetup
From Postgres to Event-Driven: using
docker-compose to build CDC pipelines
into Kafka
Mark Teehan
Singapore
16-Mar-2021
developer.confluent.io
Learn Kafka.
Start building with Apache
Kafka at Confluent Developer.
Discover The World Of Event Streaming
Kafka Summit is the premier event for developers, architects, data engineers, devops professionals, and
anyone else who wants to learn about streaming data. It brings the Apache Kafka community together to
share best practices, learn how to build next-generation systems, and discuss the future of streaming
technologies.
Kafka Summit APAC 2021 will be hosted virtually, and is free to charge to attend.
● Register now free of charge: https://www.kafka-summit.org/
● Submit an abstract via the Call for Papers (closes April 5th):
Watch short video from Tim Berglund on how to write a great CFP
https://www.youtube.com/watch?v=N0g3QoCuqH4
cnfl.io/ask-the-community
Ask the
Community!
Ask questions, share
knowledge and chat
with your fellow
community members!
Free eBooks Designing Event-Driven Systems
Ben Stopford
Kafka: The Definitive Guide
Neha Narkhede, Gwen Shapira, Todd
Palino
Making Sense of Stream Processing
Martin Kleppmann
I ❤ Logs
Jay Kreps
http://cnfl.io/book-bundle
confluent.io/blog
confluent.io/blog
Apache Kafka Meetup
From Postgres to Event-Driven: using
docker-compose to build CDC pipelines
into Kafka
Mark Teehan
Singapore
16-Mar-2021
Mark Teehan
Principal
Solution
Engineer
Confluent
(Singapore)
https://github.com/markteehan/
CDC_Postgres_Kafka
From Postgres
to Event-Driven:
using
docker-compose
to build CDC
pipelines into
Kafka
Use the Debezium CDC connector to
capture database changes from a
Postgres database (or MySQL or
Oracle); streaming into Kafka topics
and onwards to an external data store.
.
I’m Live
But I’ve
recorded some
videos
(They are short)
Ask me questions in the chat at
any time
D1 - lets get started
D2
D3
D4
……
Discover The World Of Event Streaming
Kafka Summit is the premier event for developers, architects, data engineers, devops professionals, and
anyone else who wants to learn about streaming data. It brings the Apache Kafka community together to
share best practices, learn how to build next-generation systems, and discuss the future of streaming
technologies.
Kafka Summit APAC 2021 will be hosted virtually, and is free to charge to attend.
● Register now free of charge: https://www.kafka-summit.org/
● Submit an abstract via the Call for Papers (closes April 5th):
https://sessionize.com/kafka-summit-apac-2021/
● Watch short video from Tim Berglund on how to write a great CFP
https://www.youtube.com/watch?v=N0g3QoCuqH4
developer.confluent.io
Learn Kafka.
Start building with Apache
Kafka at Confluent Developer.
developer.confluent.io
Apache Kafka Meetup
Thank You!
https://github.com/markteehan/CDC_Postgres_Kafka
Mark Teehan
Singapore
16-Mar-2021
……

More Related Content

What's hot (20)

PPTX
Introduction to Storm
Chandler Huang
 
PDF
The Top Five Mistakes Made When Writing Streaming Applications with Mark Grov...
Databricks
 
PDF
Running Apache Spark on Kubernetes: Best Practices and Pitfalls
Databricks
 
PPTX
Data Pipelines with Kafka Connect
Kaufman Ng
 
PDF
Apache kafka
NexThoughts Technologies
 
PPTX
Apache Spark Architecture
Alexey Grishchenko
 
PPTX
Big data and Hadoop
Rahul Agarwal
 
PPTX
Apache Tez - A New Chapter in Hadoop Data Processing
DataWorks Summit
 
PDF
How to Automate Performance Tuning for Apache Spark
Databricks
 
PDF
Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...
Databricks
 
PDF
Introduction to Spark with Python
Gokhan Atil
 
PDF
Simplify and Scale Data Engineering Pipelines with Delta Lake
Databricks
 
PPTX
Tuning Apache Kafka Connectors for Flink.pptx
Flink Forward
 
PDF
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
Databricks
 
PDF
Performant Streaming in Production: Preventing Common Pitfalls when Productio...
Databricks
 
PDF
Introducing Change Data Capture with Debezium
ChengKuan Gan
 
PPTX
Apache Tez: Accelerating Hadoop Query Processing
DataWorks Summit
 
PDF
Deep Dive: Memory Management in Apache Spark
Databricks
 
PDF
Understanding Query Plans and Spark UIs
Databricks
 
PDF
Kafka Streams: What it is, and how to use it?
confluent
 
Introduction to Storm
Chandler Huang
 
The Top Five Mistakes Made When Writing Streaming Applications with Mark Grov...
Databricks
 
Running Apache Spark on Kubernetes: Best Practices and Pitfalls
Databricks
 
Data Pipelines with Kafka Connect
Kaufman Ng
 
Apache Spark Architecture
Alexey Grishchenko
 
Big data and Hadoop
Rahul Agarwal
 
Apache Tez - A New Chapter in Hadoop Data Processing
DataWorks Summit
 
How to Automate Performance Tuning for Apache Spark
Databricks
 
Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...
Databricks
 
Introduction to Spark with Python
Gokhan Atil
 
Simplify and Scale Data Engineering Pipelines with Delta Lake
Databricks
 
Tuning Apache Kafka Connectors for Flink.pptx
Flink Forward
 
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
Databricks
 
Performant Streaming in Production: Preventing Common Pitfalls when Productio...
Databricks
 
Introducing Change Data Capture with Debezium
ChengKuan Gan
 
Apache Tez: Accelerating Hadoop Query Processing
DataWorks Summit
 
Deep Dive: Memory Management in Apache Spark
Databricks
 
Understanding Query Plans and Spark UIs
Databricks
 
Kafka Streams: What it is, and how to use it?
confluent
 

Similar to From Postgres to Event-Driven: using docker-compose to build CDC pipelines into Apache Kafka® (20)

PDF
Apache Kafka and KSQL in Action: Let's Build a Streaming Data Pipeline!
confluent
 
PDF
JHipster conf 2019 - Kafka Ecosystem
Florent Ramiere
 
PDF
Introduction to Apache Kafka and why it matters - Madrid
Paolo Castagna
 
PPTX
Data Stream Processing for Beginners with Kafka and CDC
Abhijit Kumar
 
PPTX
Mule soft meetup_chandigarh_#7_25_sept_2021
Lalit Panwar
 
PPTX
Streaming Data Ingest and Processing with Apache Kafka
Attunity
 
PDF
Apache Kafka and KSQL in Action: Let's Build a Streaming Data Pipeline!
confluent
 
PDF
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
HostedbyConfluent
 
PDF
What's new in Confluent 3.2 and Apache Kafka 0.10.2
confluent
 
PPTX
Building streaming data applications using Kafka*[Connect + Core + Streams] b...
Data Con LA
 
PDF
Data integration with Apache Kafka
confluent
 
PDF
Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...
confluent
 
PPTX
Streaming Data and Stream Processing with Apache Kafka
confluent
 
PDF
Apache Kafka as Event-Driven Open Source Streaming Platform (Prague Meetup)
Kai Wähner
 
PDF
Learn Apache Kafka Online | Comprehensive Kafka Course & Training
Accentfuture
 
PDF
Concepts and Patterns for Streaming Services with Kafka
QAware GmbH
 
PDF
Welcome to Kafka, We’re Glad You’re Here With Dave Klein | Current 2022
HostedbyConfluent
 
PPTX
Learn Apache Kafka Online | Comprehensive Kafka Course & Training
Accentfuture
 
PDF
Building Streaming Data Applications Using Apache Kafka
Slim Baltagi
 
PDF
Apache Kafka Use Cases_ When To Use It_ When Not To Use_.pdf
Noman Shaikh
 
Apache Kafka and KSQL in Action: Let's Build a Streaming Data Pipeline!
confluent
 
JHipster conf 2019 - Kafka Ecosystem
Florent Ramiere
 
Introduction to Apache Kafka and why it matters - Madrid
Paolo Castagna
 
Data Stream Processing for Beginners with Kafka and CDC
Abhijit Kumar
 
Mule soft meetup_chandigarh_#7_25_sept_2021
Lalit Panwar
 
Streaming Data Ingest and Processing with Apache Kafka
Attunity
 
Apache Kafka and KSQL in Action: Let's Build a Streaming Data Pipeline!
confluent
 
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
HostedbyConfluent
 
What's new in Confluent 3.2 and Apache Kafka 0.10.2
confluent
 
Building streaming data applications using Kafka*[Connect + Core + Streams] b...
Data Con LA
 
Data integration with Apache Kafka
confluent
 
Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...
confluent
 
Streaming Data and Stream Processing with Apache Kafka
confluent
 
Apache Kafka as Event-Driven Open Source Streaming Platform (Prague Meetup)
Kai Wähner
 
Learn Apache Kafka Online | Comprehensive Kafka Course & Training
Accentfuture
 
Concepts and Patterns for Streaming Services with Kafka
QAware GmbH
 
Welcome to Kafka, We’re Glad You’re Here With Dave Klein | Current 2022
HostedbyConfluent
 
Learn Apache Kafka Online | Comprehensive Kafka Course & Training
Accentfuture
 
Building Streaming Data Applications Using Apache Kafka
Slim Baltagi
 
Apache Kafka Use Cases_ When To Use It_ When Not To Use_.pdf
Noman Shaikh
 
Ad

More from confluent (20)

PDF
Stream Processing Handson Workshop - Flink SQL Hands-on Workshop (Korean)
confluent
 
PPTX
Webinar Think Right - Shift Left - 19-03-2025.pptx
confluent
 
PDF
Migration, backup and restore made easy using Kannika
confluent
 
PDF
Five Things You Need to Know About Data Streaming in 2025
confluent
 
PDF
Data in Motion Tour Seoul 2024 - Keynote
confluent
 
PDF
Data in Motion Tour Seoul 2024 - Roadmap Demo
confluent
 
PDF
From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...
confluent
 
PDF
Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...
confluent
 
PDF
Data in Motion Tour 2024 Riyadh, Saudi Arabia
confluent
 
PDF
Build a Real-Time Decision Support Application for Financial Market Traders w...
confluent
 
PDF
Strumenti e Strategie di Stream Governance con Confluent Platform
confluent
 
PDF
Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeks
confluent
 
PDF
Building Real-Time Gen AI Applications with SingleStore and Confluent
confluent
 
PDF
Unlocking value with event-driven architecture by Confluent
confluent
 
PDF
Il Data Streaming per un’AI real-time di nuova generazione
confluent
 
PDF
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
confluent
 
PDF
Break data silos with real-time connectivity using Confluent Cloud Connectors
confluent
 
PDF
Building API data products on top of your real-time data infrastructure
confluent
 
PDF
Speed Wins: From Kafka to APIs in Minutes
confluent
 
PDF
Evolving Data Governance for the Real-time Streaming and AI Era
confluent
 
Stream Processing Handson Workshop - Flink SQL Hands-on Workshop (Korean)
confluent
 
Webinar Think Right - Shift Left - 19-03-2025.pptx
confluent
 
Migration, backup and restore made easy using Kannika
confluent
 
Five Things You Need to Know About Data Streaming in 2025
confluent
 
Data in Motion Tour Seoul 2024 - Keynote
confluent
 
Data in Motion Tour Seoul 2024 - Roadmap Demo
confluent
 
From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...
confluent
 
Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...
confluent
 
Data in Motion Tour 2024 Riyadh, Saudi Arabia
confluent
 
Build a Real-Time Decision Support Application for Financial Market Traders w...
confluent
 
Strumenti e Strategie di Stream Governance con Confluent Platform
confluent
 
Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeks
confluent
 
Building Real-Time Gen AI Applications with SingleStore and Confluent
confluent
 
Unlocking value with event-driven architecture by Confluent
confluent
 
Il Data Streaming per un’AI real-time di nuova generazione
confluent
 
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
confluent
 
Break data silos with real-time connectivity using Confluent Cloud Connectors
confluent
 
Building API data products on top of your real-time data infrastructure
confluent
 
Speed Wins: From Kafka to APIs in Minutes
confluent
 
Evolving Data Governance for the Real-time Streaming and AI Era
confluent
 
Ad

Recently uploaded (20)

PDF
[Newgen] NewgenONE Marvin Brochure 1.pdf
darshakparmar
 
PDF
Newgen Beyond Frankenstein_Build vs Buy_Digital_version.pdf
darshakparmar
 
PDF
POV_ Why Enterprises Need to Find Value in ZERO.pdf
darshakparmar
 
PDF
CIFDAQ Market Insights for July 7th 2025
CIFDAQ
 
PDF
Smart Trailers 2025 Update with History and Overview
Paul Menig
 
PDF
Using FME to Develop Self-Service CAD Applications for a Major UK Police Force
Safe Software
 
PDF
The Rise of AI and IoT in Mobile App Tech.pdf
IMG Global Infotech
 
PDF
New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
PDF
Achieving Consistent and Reliable AI Code Generation - Medusa AI
medusaaico
 
PDF
"AI Transformation: Directions and Challenges", Pavlo Shaternik
Fwdays
 
PPTX
OpenID AuthZEN - Analyst Briefing July 2025
David Brossard
 
PDF
CIFDAQ Market Wrap for the week of 4th July 2025
CIFDAQ
 
PDF
Empower Inclusion Through Accessible Java Applications
Ana-Maria Mihalceanu
 
PDF
Exolore The Essential AI Tools in 2025.pdf
Srinivasan M
 
PDF
IoT-Powered Industrial Transformation – Smart Manufacturing to Connected Heal...
Rejig Digital
 
PDF
Reverse Engineering of Security Products: Developing an Advanced Microsoft De...
nwbxhhcyjv
 
PDF
DevBcn - Building 10x Organizations Using Modern Productivity Metrics
Justin Reock
 
PDF
Biography of Daniel Podor.pdf
Daniel Podor
 
DOCX
Python coding for beginners !! Start now!#
Rajni Bhardwaj Grover
 
PPTX
Building Search Using OpenSearch: Limitations and Workarounds
Sease
 
[Newgen] NewgenONE Marvin Brochure 1.pdf
darshakparmar
 
Newgen Beyond Frankenstein_Build vs Buy_Digital_version.pdf
darshakparmar
 
POV_ Why Enterprises Need to Find Value in ZERO.pdf
darshakparmar
 
CIFDAQ Market Insights for July 7th 2025
CIFDAQ
 
Smart Trailers 2025 Update with History and Overview
Paul Menig
 
Using FME to Develop Self-Service CAD Applications for a Major UK Police Force
Safe Software
 
The Rise of AI and IoT in Mobile App Tech.pdf
IMG Global Infotech
 
New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
Achieving Consistent and Reliable AI Code Generation - Medusa AI
medusaaico
 
"AI Transformation: Directions and Challenges", Pavlo Shaternik
Fwdays
 
OpenID AuthZEN - Analyst Briefing July 2025
David Brossard
 
CIFDAQ Market Wrap for the week of 4th July 2025
CIFDAQ
 
Empower Inclusion Through Accessible Java Applications
Ana-Maria Mihalceanu
 
Exolore The Essential AI Tools in 2025.pdf
Srinivasan M
 
IoT-Powered Industrial Transformation – Smart Manufacturing to Connected Heal...
Rejig Digital
 
Reverse Engineering of Security Products: Developing an Advanced Microsoft De...
nwbxhhcyjv
 
DevBcn - Building 10x Organizations Using Modern Productivity Metrics
Justin Reock
 
Biography of Daniel Podor.pdf
Daniel Podor
 
Python coding for beginners !! Start now!#
Rajni Bhardwaj Grover
 
Building Search Using OpenSearch: Limitations and Workarounds
Sease
 

From Postgres to Event-Driven: using docker-compose to build CDC pipelines into Apache Kafka®

  • 1. Apache Kafka Meetup From Postgres to Event-Driven: using docker-compose to build CDC pipelines into Kafka Mark Teehan Singapore 16-Mar-2021
  • 2. developer.confluent.io Learn Kafka. Start building with Apache Kafka at Confluent Developer.
  • 3. Discover The World Of Event Streaming Kafka Summit is the premier event for developers, architects, data engineers, devops professionals, and anyone else who wants to learn about streaming data. It brings the Apache Kafka community together to share best practices, learn how to build next-generation systems, and discuss the future of streaming technologies. Kafka Summit APAC 2021 will be hosted virtually, and is free to charge to attend. ● Register now free of charge: https://www.kafka-summit.org/ ● Submit an abstract via the Call for Papers (closes April 5th): Watch short video from Tim Berglund on how to write a great CFP https://www.youtube.com/watch?v=N0g3QoCuqH4
  • 4. cnfl.io/ask-the-community Ask the Community! Ask questions, share knowledge and chat with your fellow community members!
  • 5. Free eBooks Designing Event-Driven Systems Ben Stopford Kafka: The Definitive Guide Neha Narkhede, Gwen Shapira, Todd Palino Making Sense of Stream Processing Martin Kleppmann I ❤ Logs Jay Kreps http://cnfl.io/book-bundle
  • 8. Apache Kafka Meetup From Postgres to Event-Driven: using docker-compose to build CDC pipelines into Kafka Mark Teehan Singapore 16-Mar-2021
  • 10. From Postgres to Event-Driven: using docker-compose to build CDC pipelines into Kafka Use the Debezium CDC connector to capture database changes from a Postgres database (or MySQL or Oracle); streaming into Kafka topics and onwards to an external data store. .
  • 11. I’m Live But I’ve recorded some videos (They are short) Ask me questions in the chat at any time
  • 12. D1 - lets get started D2 D3 D4
  • 14. Discover The World Of Event Streaming Kafka Summit is the premier event for developers, architects, data engineers, devops professionals, and anyone else who wants to learn about streaming data. It brings the Apache Kafka community together to share best practices, learn how to build next-generation systems, and discuss the future of streaming technologies. Kafka Summit APAC 2021 will be hosted virtually, and is free to charge to attend. ● Register now free of charge: https://www.kafka-summit.org/ ● Submit an abstract via the Call for Papers (closes April 5th): https://sessionize.com/kafka-summit-apac-2021/ ● Watch short video from Tim Berglund on how to write a great CFP https://www.youtube.com/watch?v=N0g3QoCuqH4
  • 15. developer.confluent.io Learn Kafka. Start building with Apache Kafka at Confluent Developer.
  • 17. Apache Kafka Meetup Thank You! https://github.com/markteehan/CDC_Postgres_Kafka Mark Teehan Singapore 16-Mar-2021