Apache Kafka & Kafka Connectを に使ったデータ連携パターン(改めETLの実装)
10 likes4,433 views
This document discusses Apache Kafka and Kafka Connect. It provides an overview of Kafka Connect and how it can be used for ETL processes. Kafka Connect allows data to be exported from or imported to Kafka and integrated with other systems through customizable connectors. The document describes how to run Kafka Connect in standalone and distributed modes and highlights some popular connectors available for integrating Kafka with other data sources and sinks.
Apache Kafka & Kafka Connectを に使ったデータ連携パターン(改めETLの実装)
- 1. Apache Kafka & Kafka Connect ( ETL ) Future Architect ,Inc Keigo Suda 2016/11/25 D&S Data Night vol.04
- 2. 12 ! Kafka Connect ETL ( ) ( ) /
- 3. 13 ! Future Architect ,Inc ( ) ! ! IoT
- 4. 14
- 6. 16 ! Kafka ver 0.9 ! Kafka ! ! ! / Kafka Connect
- 11. 21
- 12. 22 ! ! ! Kafka Connect ( )
- 13. 23 ETL Kakfa Connect( ) P T // / / AC KM I M KM
- 14. Kafka Connect
- 17. 27 Stream & Partition(RDB ) http://www.slideshare.net/KaufmanNg/data-pipelines-with-kafka-connect
- 18. 28 N:1 Stream & Partition(RDB ) http://www.slideshare.net/KaufmanNg/data-pipelines-with-kafka-connect
- 22. 32 $ bin/connect-standalone.sh config/connect-standalone.properties connector1.properties (Standalone mode) ! ! ! Worker ! Connector Worker Connector
- 25. 35 ! ! ! ETL
- 26. 36 ! Kafka Connect Source Sink ! Sink/Source Sink Kafka Connect Source Sink
- 27. 37 ! Connector/Task OK ! API OK ( ) ( )
- 28. 38 Connector( )
- 35. 45 ! Confluent Confuluent Platform Kafka Connect
- 36. 46 ! Kakfa ( ) ! Confluent….