Overview of Kafka connect
0 likes76 views
The document discusses Kafka connectors and how they allow Kafka to connect to external systems. It defines Kafka connectors as a framework that runs in a Kafka Connect cluster to read data from Kafka topics and write it to other systems like databases. It describes how connectors work, including source and sink connectors, and highlights use cases like processing data from old to new systems. The architecture of Kafka Connect clusters and connectors is covered along with features like scalability, REST interface, and offset management.
Overview of Kafka connect
- 1. Presented By: Shreya Saxena & Chiranjeev Kumar Overview of kafka Connectors & Connector
- 2. Lack of etiquette and manners is a huge turn off. KnolX Etiquettes Punctuality Respect Knolx session timings, you are requested not to join sessions after a 5 minutes threshold post the session start time. Feedback Make sure to submit a constructive feedback for all sessions as it is very helpful for the presenter. Silent Mode Keep your mobile devices in silent mode, feel free to move out of session in case you need to attend an urgent call. Avoid Disturbance Avoid unwanted chit chat during the session.
- 3. Agenda 02 How Kafka Connect/Connector works? Kafka Connect/Connector Introduction 05 Use cases of Kafka Connect/Connector 06 Kafka Connect/Connector features Kafka Connect/Connector Architecture 07 DEMO 002 2 02 Why Kafka Connect/Connector? 03 04 01
- 4. What is Kafka Connect/Connector? ● Kafka connect/connector is a framework for connecting kafka with external systems including databases. ● The Kafka connect/connector is designed to run in a Kafka Connect cluster to read data from Kafka topics and write the data into Snowflake tables ● The Kafka Connect/Connector cluster supports running and scaling out connectors.
- 5. How is Kafka Connect/Connector works? ● Kafka Connect act as a standalone process that runs jobs on a single machine (for example, log collection), or as a distributed, scalable, fault-tolerant service supporting an entire organization ● Two types 1 Source connector 2 Sink connector
- 6. Why Kafka Connect/Connector ? ● Kafka connect/connector is a tool that provides scalably and reliably streaming data between Apache Kafka® and other data systems. ● With the help of kafka connector we can import data from external systems into Kafka topics. ● Also, export data from Kafka topics into external systems. ● There are connectors that help to move huge data sets into and out of the Kafka system.
- 7. Kafka Connect/Connector Architecture ● Kafka Connect is a separate Cluster. ● Each worker contains one or many connector task. ● A cluster can have multiple workers and worker runs on the cluster only. ● Task are automatically load-balanced if there is any failure. ● Task in Kafka Connect act as a Producer or Consumer depending on the type of Connector. ● Kafka Connect cluster has a multiple loaded connectors.
- 8. Use Cases of Kafka Connect/Connector? ● kafka connect/connectors is generally use for synced operation. ● Kafka connect/connectors is designed for connecting Kafka with other data systems. ● With the help of Kafka connect we can Process a data from outdated application to the new systems.
- 9. Kafka Connect/Connector Features ● A common framework for connecting external system with kafka ● Distributed and standalone modes ● REST interface ● Automatic offset management ● Distributed and scalable by default
- 10. DEMO (Installation of Confluent Control center)
- 11. Steps for installing and setup ● Linux System ● Java installation ● download confluent bundle curl -O http://packages.confluent.io/archive/7.1/confluent-7.1.1.zip ● unzip the file unzip confluent-7.1.1.zip ● export CONFLUENT_HOME=~/confluent-7.1.1 ● export PATH=$CONFLUENT_HOME/bin:$PATH ● confluent local services start