Building a Streaming Pipeline on Kubernetes Using Kafka Connect, KSQLDB & Apache Pinot | Mateus Oliveira, One Way Solution & Luan Moreno Mederios, Pythian
0 likes778 views
The document discusses the challenges and solutions of real-time analytics in big data infrastructure, particularly using microservices and Kubernetes. It highlights the efficiencies gained by deploying containerized applications on Kubernetes, including cost reduction and scalability. Key tools mentioned include Apache Kafka for data transportation and Apache Pinot for real-time insights, emphasizing the importance of selecting the right technology stack to address business needs effectively.
Building a Streaming Pipeline on Kubernetes Using Kafka Connect, KSQLDB & Apache Pinot | Mateus Oliveira, One Way Solution & Luan Moreno Mederios, Pythian
- 1. Luan Moreno Big Data Engineer at Pythian [email protected] Mateus Oliveira Data In-Motion Specialist at One Way Solution [email protected]
- 2. Real-Time Analytics at Scale is Challenging - The Business Problem Microservices • Microservices & Datastores Problem • K8S as De-Facto Orchestration Service • Containerized Stateless & Stateful Applications Toolset • Modern Market Urges for Real-Time • Immutable Infrastructure with CI • Distributed Engines of Big Data • Scale-Up & Down Resources Real-Time Analytics • Challenge for Querying Data In-Motion • QPS in a Low Latency Fashion • Answers in Real-Time for Business Level • Scalable Data Pipelines • Rapidly Reaction of a Business Change Event Sourcing Engine apache kafka as datahub and event sourcing for real-time data transportation Querying Data ability to retrieve & store data from apache kafka in real-time fashion efficiently
- 3. Microservices Kubernetes as Cornerstone Solution for Big Data Infrastructure applications are now containerized and, the microservices age gives the power to scale up resources as needed in a timely fashion. Statefulsets in your prior days, K8s didn't support stateful applications (2017), now it’s possible, and big data solutions can be implemented and deployed efficiently. Current Gen of Big Data we've seen more and more big data applications being converted to work at k8s or born in k8s, e.g., strimzi and confluent operator. Cost-wise containerized applications are designed to consume fewer resources; you can deploy many applications at once in k8s; imagine having ingestion, processing, and serving layer for a price of 1.
- 4. Real-Time Data Pipeline on Kubernetes using Kafka, KafkaConnect, KSQLDB & Apache Pinot Event Sourcing Engine Strimzi Operator as Datahub and Event Sourcing System for Real-Time Data Transport Stream Processor KSQLDB as SQL Processing Engine, Select Data & Creating New Enriched Topics Data Sources Datastore Systems ~ Relational & NoSQL Databases Data Serving Apache Pinot OLAP System for Real-Time Insights at Scale Infrastructure Backbone Big Data Products Deployed on Kubernetes Cluster as Containers KafkaConnect Extraction of Events using a YAML Configuration File
- 5. • Blazing Fast OLAP Engine • LinkedIN, Uber, Target, Slack, Stripe • Designed for Real-Time Analytics Realtime Distributed OLAP DataStore, Designed for Answering OLAP Queries with Low-Latency Apache Pinot Apache Pinot Advantages for Real-Time Analytics Characteristics History • Pinot Noir ~ Name of Grape for Wine • Developed Internally at LinkedIN in 2014 • Open-Sourced in 2015 • Entered in Apache Incubator in 2018 Key Capabilities • Columnar-Oriented Storage • Pluggable Indexing Technologies • Horizontally Scale & Fault-Tolerant • Performs Anomaly Detection using ThirdEye • Joins using Presto & Trino • Controller (Helix & Zookeeper) ~ Manage State & Health • Broker ~ Route Queries • Server ~ Host Segments (Data) ~ Realtime & Offline Tables • Minion ~ Purge Data Core Components • PQL ~ SQL using Apache Calcite • API ~ RestAPIs for Broker & Controller • External Clients ~ JDBC, Java, Python & Go Users • 100K + Queries per Second at ms Latency • Ingestion of Million of Events per Second • 50+ User-Facing Analytics Performance
- 6. Building Data Pipeline using KafkaConnect, KSQLDB & Apache Pinot Demonstration Kafka & KafkaConnect KSQLDB Apache Pinot
- 7. The Summary Real-Time Analytics is Impactful not easy to build and manage, need to choose the right toolset and strategy K8s Simplifies Infrastructure reduce it ops and increases the agility of the team by offering containerized applications management Business Oriented when we take out the heavy lifting of infrastructure and choose the right toolset of big data, we can focus on what matters: the business problem Cost Reduction Benefit working with both kubernetes and open-source tools to deliver big data pipelines, we can see how cost could be dramatically reduced
- 8. Thank You