Introducing Kafka Connect and Implementing Custom Connectors
1 like650 views
The document introduces Kafka Connect, a tool for integrating data into and out of Kafka, designed to facilitate the management of data across various technologies. It outlines the architecture of Kafka Connect, emphasizing its fault tolerance and scalability, particularly in situations where direct integration is impractical. Additionally, it details the implementation of custom connectors and the setting up of necessary infrastructure to utilize Kafka Connect effectively.
Introducing Kafka Connect and Implementing Custom Connectors
- 1. Introducing Kafka Connect and ImplementingCustom Connectors Kobi Hikri Kafka Connect
- 2. Agenda What (is Kafka Connect) The World of Data Why (do we need it) Kafka Connect When (will we consider using) Kafka Connect How Kafka Connect Setting up the infrastructure Interacting with Kafka Connect Implementing a custom source connector Working with connectors What will not be covered in this talk (will be covered per request) Kafka ConnectTransformations Working with custom schemas
- 3. What is Kafka Connect A Kafka integration tool Allows importing data into Kafka And exporting data out of Kafka Runs as a cluster of one or more system processes Each such process is referred to as a Worker Workers are grouped together in Worker Groups Connector Instances are managed by theConnect cluster Tasks are configured by connectors and are distributed across workers Data in being pulled by SourceTasks (configured by Source Connectors) Or Pushed by Sink Tasks (configured by Sink Connectors) Worker Worker Worker Worker Worker Group Worker Group
- 5. Why KafkaConnect In one word Integration Of different data oriented technologies Of a multitude of changing APIs and communication protocols In more than one word Fault tolerant, scalable Integration with various technologies
- 6. When Kafka Connect Implementing a Kafka Producer or Consumer within the desired data source / target is impossible or expensive (e.g. the source code is unavailable) Even when we can modify the source code of the source / target system – but prefer to have a scalable and fault tolerant integration with it (e.g. not to be limited in bandwidth by the direct connection between the technology and our Kafka cluster).