Integrating Apache Kafka and Elastic Using the Connect Framework
4 likes4,612 views
The document outlines the integration of Apache Kafka and Elasticsearch using the Kafka Connect framework, emphasizing the high scalability and throughput of Kafka. It details the configuration of Kafka Connect's Elasticsearch sink and the advantages of using event-centric thinking for real-time data processing. Additionally, it discusses schema management and the use of single message transformations for ETL operations.
![17
Integrating Elastic with Kafka - Beats, Logstash
output.kafka:
hosts: ["localhost:9092"]
topic: 'logs'
required_acks: 1
output {
kafka {
topic_id => "logstash_logs_json"
bootstrap_servers => "localhost:9092"
codec => json
}
}
Beats
Logstash](https://image.slidesharecdn.com/integratingapachekafkawithelasticsearch-180307171018/85/Integrating-Apache-Kafka-and-Elastic-Using-the-Connect-Framework-17-320.jpg)
![26
Schemas in Kafka Connect - JSON
{"schema":
{"type":"struct",
"fields":[{"type":"int32","optional":true,"field":"c1"},
{"type":"string","optional":true,"field":"c2"},
{"type":"int64","optional":false,
"name":"org.apache.kafka.connect.data.Timestamp","field":"create_ts"},
{"type":"int64","optional":false,
"name":"org.apache.kafka.connect.data.Timestamp","field":"update_ts"}],
"optional":false,
"name":"foobar"
},
"payload":{ "c1":100,
"c2":"bar",
"create_ts":1516747629000,
"update_ts":1516747629000}
}](https://image.slidesharecdn.com/integratingapachekafkawithelasticsearch-180307171018/85/Integrating-Apache-Kafka-and-Elastic-Using-the-Connect-Framework-26-320.jpg)
Integrating Apache Kafka and Elastic Using the Connect Framework
- 1. 1 Integrating Apache Kafka and Elastic using the Connect Framework
- 2. 2 I ❤ Elastic 😁
- 3. Kafka Cluster 3 Apache Kafka® Kafka A Distributed Commit Log. Publish and subscribe to streams of records. Highly scalable, high throughput. Supports transactions. Persisted data. Reads are a single seek & Writes are append only
- 4. 4 Apache Kafka® Kafka Streams API Write standard Java applications & microservices to process your data in real-time Kafka Connect API Reliable and scalable integration of Kafka with other systems – no coding required. Orders Table Customers Kafka Streams API
- 5. 5 Many Systems are a bit of a mess…
- 8. 8 Why Kafka & Elastic?
- 9. Event-Centric Thinking Streaming Platform “A product was viewed” Elasticsear ch web app
- 10. Event-Centric Thinking Streaming Platform “A product was viewed” web app mobile app APIs Elasticsear ch
- 11. mobile app web app APIs Streaming Platform Hadoop Security Monitoring Elastic search “A product was Event-Centric Thinking
- 12. System Availability and Event Buffering Producer Elasticsearch
- 13. System Availability and Event Buffering Producer Elasticsearch
- 15. Visualise & Analyse data from Kafka
- 16. 16 Integrating Elastic and Kafka
- 17. 17 Integrating Elastic with Kafka - Beats, Logstash output.kafka: hosts: ["localhost:9092"] topic: 'logs' required_acks: 1 output { kafka { topic_id => "logstash_logs_json" bootstrap_servers => "localhost:9092" codec => json } } Beats Logstash
- 18. 18
- 19. 19 Kafka Connect Kafka Brokers Kafka Connect Tasks Workers Sources Sinks Amazon S3 syslog flat file
- 21. 21 Kafka Connect's Elasticsearch Sink { "name": "es-sink", "config": { "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector", "connection.url": "http://localhost:9200", "type.name": "type.name=kafka-connect", "topics": "foobar" } }
- 22. 22 Kafka Connect to stream Kafka Topics to Elasticsearch
- 23. 23 Kafka Connect Elasticsearch Sink Properties https://docs.confluent.io/current/connect/connect-elasticsearch/docs/configuration_options.html
- 24. 24 Sink properties : Converters • Json, Avro, String, Protobuf, etc • Specify the converter in the Kafka Connect configuration, e.g. key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter • Kafka Connect uses pluggable converters for both message key and value deserialisation
- 25. 25 Schemas & Document Mappings
- 26. 26 Schemas in Kafka Connect - JSON {"schema": {"type":"struct", "fields":[{"type":"int32","optional":true,"field":"c1"}, {"type":"string","optional":true,"field":"c2"}, {"type":"int64","optional":false, "name":"org.apache.kafka.connect.data.Timestamp","field":"create_ts"}, {"type":"int64","optional":false, "name":"org.apache.kafka.connect.data.Timestamp","field":"update_ts"}], "optional":false, "name":"foobar" }, "payload":{ "c1":100, "c2":"bar", "create_ts":1516747629000, "update_ts":1516747629000} }
- 27. 27 Kafka Connect + Schema Registry = WIN Avro Messag e Schema Registry Avro Schema Kafka Connect
- 28. 28 Single Message Transform (SMT) -- Extract, TRANSFORM, Load… • Modify events before storing in Kafka: • Mask/drop sensitive information • Set partitioning key • Store lineage • Cast data types • Modify events going out of Kafka: • Direct events to different Elasticsearch indexes • Mask/drop sensitive information • Cast data types to match destination
- 29. 29 Confluent Platform: Enterprise Streaming based on Apache Kafka® Database Changes Log Events loT Data Web Events … CRM Data Warehouse Database Hadoop Data Integration … Monitoring Analytics Custom Apps Transformations Real-time Applications … Apache Open Source Confluent Open Source Confluent Enterprise Confluent Platform Confluent Platform Apache Kafka® Core | Connect API | Streams API Data Compatibility Schema Registry Monitoring & Administration Confluent Control Center | Security Operations Replicator | Auto Data Balancing Development and Connectivity Clients | Connectors | REST Proxy | CLI Apache Open Source Confluent Open Source Confluent Enterprise SQL Stream Processing KSQL
- 30. 30 https://www.confluent.io/download/ Streaming ETL, powered by Apache Kafka and Confluent Platform https://www.confluent.io/blog/simplest-useful-kafka-connect-data-pipeline-world-thereabouts-part-1/ https://docs.confluent.io/current/connect/connect-elasticsearch/docs/