Confluent Kafka and KSQL: Streaming Data Pipelines Made Easy
Download as PPTX, PDF3 likes435 views
The document outlines the Confluent Kafka platform and its role in simplifying the creation of streaming data pipelines. It discusses the challenges associated with these pipelines and introduces ksql as a tool for building real-time applications using SQL-like semantics. Additionally, it provides an overview of key features and components of the Confluent platform, including management, deployment, and stream processing capabilities.
Confluent Kafka and KSQL: Streaming Data Pipelines Made Easy
- 1. Confluent Kafka and KSQL: Streaming Data Pipelines Made Easy Kairo Tavares [email protected]
- 2. Goals How Confluent Kafka Platform can solve problems in company What are streaming data pipelines and what are its challenges How KSQL can make easy your streaming data pipeline 2
- 3. Agenda – Kafka 101 – Confluent Kafka – Streaming Data Pipeline – KSQL – Demo 3
- 4. Kafka 101 4
- 5. Common problems that we face Extract - Transform – Load (ETL) 5
- 6. Common problems that we face Microservices 6
- 8. Kafka 101 Apache Kafka Kafka® is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault- tolerant, wicked fast, and runs in production in thousands of companies. 8 https://kafka.apache.org/
- 9. Kafka 101 9
- 11. Kafka 101 Consuming Data 11 Simple Consumer Consumer Groups
- 13. Confluent Platform 13 Founded by the team that built Apache Kafka®, Confluent builds an event streaming platform that enables companies to easily access data as real-time streams Development and Connectivity Stream Processing Management and Operations https://www.confluent.io/
- 14. Confluent Platform Deployment and Connectivity – Kafka Connect & Connectors – Schema Registry – Kafka Clients – REST Proxy – MQTT Proxy 14
- 15. Confluent Platform Avro Format – Open Source Data Format – Choice for a number of reasons: – Direct mapping to and from JSON – Compact format – Very fast to serialize and deserialize – Support to several programming languages – It has a rich, extensible schema language defined in pure JSON – It has the best notion of compatibility for evolving your data over time – Built-in documentation 15
- 16. Confluent Platform Management and Operations – Control Center – Manage key operations and monitor the health and performance of Kafka clusters and data streams with curated dashboards directly from a GUI. – Replicator – Replicate Kafka topics across data centers and public clouds to ensure disaster recovery and build distributed data pipelines. – Auto Data Balancer – Optimize your resource utilization by invoking a rack-aware algorithm that automatically rebalances partitions across a Kafka cluster. – Security Controls – Enable pass-through client credentials from REST Proxy / Schema Registry to Kafka broker. Map AD/LDAP groups to Kafka ACLs. – Operator – Automate deployment of the complete Confluent Platform, including Kafka, as a cloud-native application on Kubernetes. 16
- 17. Confluent Platform Stream Processing – Kafka Streams (Library) – Build mission-critical real-time applications with a simple Java library and a Kafka cluster - no additional framework or cluster needed. – KSQL (Service) – Build real-time stream processing applications against Apache Kafka using simple SQL-like semantics. 17
- 20. Streaming Data Pipeline But what is a stream? 20 – A stream is an unbounded, continuous flow of records – Data is real-time – Immutable events – Records are key-value pairs (Kafka)
- 21. Streaming Data Pipeline Stream Processing 21 Per-record millisecond delay Data filtering Data transformation and conversions Data enrichment with joins Data manipulation with scalar functions Stateful processing Data Aggregation Windowing processing
- 23. KSQL 23
- 26. KSQL Kafka Streams Library vs KSQL 26
- 29. KSQL Aggregation 29 How many pageviews regions by user?
- 30. KSQL Aggregation Functions 30UDF and UDAF
- 32. KSQL Click Stream - ETL 32 Stream (clickstream) {userid, page, action, usage_time} Table (users) {user_id, level, gender, age}
- 33. KSQL Credit Card Fraud – Anomaly detection 33
- 36. KSQL Machine Learning Prediction 36 CREATE STREAM AnomalyDetectionWithFilter SELECT rowtime, eventid, anomaly(sensorinput) AS Anomaly FROM carsensor WHERE anomaly(sensorinput) > 5;
Editor's Notes
- #9: Apache Kafka was originally developed by LinkedIn, and was subsequently open sourced in early 2011. Graduation from the Apache Incubator occurred on 23 October 2012. In 2014, Jun Rao, Jay Kreps, and Neha Narkhede, who had worked on Kafka at LinkedIn, created a new company named Confluent with a focus on Kafka.[5] According to a Quora post from 2014, Kreps chose to name the software after the author Franz Kafka because it is "a system optimized for writing", and he liked Kafka's work