Serverless stream processing of Debezium data change events with Knative | DevNation Tech Talk
0 likes3,324 views
The document discusses the implementation of serverless stream processing for Debezium data change events using Knative, focusing on consuming temperature measurements, enriching the data stream, routing, and visualizing it. Key components include Apache Kafka, Kafka Connect, Debezium for change data capture, Strimzi for managing Kafka on Kubernetes, and Knative for serverless management. It provides an overview of the solution architecture and links for further exploration and demos.
Serverless stream processing of Debezium data change events with Knative | DevNation Tech Talk
- 1. Serverless Stream Processing of Debezium Data Change Events with Knative Gunnar Morling @gunnarmorling Principal Software Engineer, Red Hat Matthias Wessendorf @mwessendorf Principal Software Engineer, Red Hat
- 2. Gunnar Morling 2 Gunnar Morling ● Open source software engineer at Red Hat ● Debezium ● Hibernate ● Spec Lead for Bean Validation 2.0 ● Java Champion ● @gunnarmorling
- 3. Matthias Wessendorf Matthias Wessendorf ● Open source software engineer at Red Hat ● Knative Eventing ● @mwessendorf
- 4. Today’sObjective 4 Today’s Objective Consume a stream of temperature measurements Enrich the stream with master data Route to different different consumers based on metadata Visualize contents of the stream on a map
- 6. 6 Solution Overview | | | | | | | | | | | | | | | | | | | | | | | Apache Kafka Kafka Connect DBZ PG Sensors Solution Overview Kafka Streams Knative Eventing Broker Kafka Event Source
- 7. Solution Overview 7 Sensor Data | | | | | | | Apache Kafka Sensors
- 8. Solution Overview 8 Database and Debezium | | | | | | | | | | | | | | | Apache Kafka Kafka Connect DBZ PG Sensors
- 9. Solution Overview 9 Apache Kafka and Strimzi | | | | | | | | | | | | | | | Apache Kafka Kafka Connect DBZ PG Sensors
- 10. Solution Overview 10 Kafka Streams and Quarkus | | | | | | | | | | | | | | | Apache Kafka Kafka Connect DBZ PG Sensors Kafka Streams
- 11. Solution Overview 11 Kafka Streams and Quarkus | | | | | | | | | | | | | | | Apache Kafka Kafka Connect DBZ PG Sensors | | | | | | | | Kafka Streams
- 12. Solution Overview 12 Knative Eventing | | | | | | | | | | | | | | | Apache Kafka Kafka Connect DBZ PG Sensors | | | | | | | | Knative Eventing Broker Kafka Streams Kafka Event Source
- 13. 13 Maps Application | | | | | | | | | | | | | | | | | | | | | | | Apache Kafka Kafka Connect DBZ PG Sensors Solution Overview Kafka Streams Knative Eventing Broker Kafka Event Source
- 14. 14 Kubernetes | | | | | | | | | | | | | | | | | | | | | | | Apache Kafka Kafka Connect DBZ PG Sensors Solution Overview Kafka Streams Knative Eventing Broker Kafka Event Source
- 15. SolutionElements 15 Solution Elements Debezium for Change Data Capture Strimzi for running Apache Kafka on Kubernetes Quarkus and Kafka Streams for stream processing Knative for serverless application management
- 16. Debezium Open-Source Change Data Capture Platform ■ Log-based CDC for a variety of databases ■ MySQL, Postgres, SQL Server, MongoDB etc. ■ Transparent to writing apps ■ Snapshotting, Filtering etc. ■ Fully open-source, very active community ■ Based on Apache Kafka (Connect) ■ Use Cases: replication, streaming queries, microservices data exchange etc. 16
- 17. Strimzi Managing your Apache Kafka Cluster on Kubernetes 17 Topic & User operators Zookeeper cluster Kafka cluster Cluster operator Kafka Custom Resource
- 18. Supersonic Subatomic Java Cloud Native Microservices Serverless 18
- 19. Supersonic Subatomic Java Quarkus Benefits Developer Joy Supersonic Subatomic Java Unifies imperative and reactive Best of breed libraries and standards 19
- 20. 20 Knative
- 21. Knative Kubernetes-based Serverless Platform 21 Serving A request-driven model that serves the container with your application and can "scale to zero". Eventing Common infrastructure for consuming and producing events that will stimulate applications.
- 22. 22 Knative Eventing Source → Service: Broker & Trigger
- 23. Demo 23 Demo
- 24. 24 Summary
- 25. 25 Kubernetes | | | | | | | | | | | | | | | | | | | | | | | Apache Kafka Kafka Connect DBZ PG Sensors Solution Overview Kafka Streams Knative Eventing Broker Kafka Event Source
- 26. Call to Action Resources 26 ● Learn more about Debezium ○ https://debezium.io/ ● Explore the demo ○ https://github.com/gunnarmorling/kafka-knative-demo/ ● Debezium GA in Red Hat Integration: ○ https://access.redhat.com/documentation/en-us/red_hat _integration/2020-04/html/ ● Install from resource descriptors on Kubernetes Cluster ○ https://knative.dev/docs/install/ ● Red Hat OpenShift Serverless - Technical Preview ○ https://www.openshift.com/learn/topics/serverless