Building Realtim Data Pipelines with Kafka Connect and Spark Streaming
5 likes2,206 views
This document discusses building real-time data pipelines using Kafka Connect and Spark Streaming, highlighting the need for efficient stream processing and various integration options. It details Kafka's data model, delivery guarantees, and the functionality of Kafka Connect including connector APIs. The presentation also emphasizes Confluent's offerings, including open-source connectors and training resources.
Building Realtim Data Pipelines with Kafka Connect and Spark Streaming
- 1. BUILDING REALTIME DATA PIPELINES WITH KAFKA CONNECT AND SPARK STREAMING Guozhang Wang Confluent
- 2. About Me: Guozhang Wang • Engineer @ Confluent. • Apache Kafka Committer, PMC Member. • Before: Engineer @ LinkedIn, Kafka and Samza.
- 3. What do you REALLY need for Stream Processing?
- 4. Spark Streaming! Is that All?
- 5. Spark Streaming! Is that All?
- 6. Spark Streaming! Is that All?
- 7. Data can Comes from / Goes to..
- 9. Real-time Data Integration: getting data to all the right places
- 10. Option #1: One-off Tools • Tools for each specific data systems • Examples: • jdbcRDD, Cassandra-Spark connector, etc.. • Sqoop, logstash to Kafka, etc..
- 11. Option #2: Kitchen Sink Tools • Generic point-to-point data copy / ETL tools • Examples: • Enterprise application integration tools
- 12. Option #3: Streaming as Copying • Use stream processing frameworks to copy data • Examples: • Spark Streaming: MyRDDWriter (forEachPartition) • Storm, Samza, Flink, etc..
- 13. Real-time Integration: E, T & L
- 15. Example: LinkedIn back in 2010
- 16. Example: LinkedIn with Kafka Apache Kafka
- 19. Large-scale streaming data import/export for Kafka Kafka Connect
- 22. Data Model
- 23. Data Model
- 29. Delivery Guarantees • Offsets automatically committed and restored • On restart: task checks offsets & rewinds • At least once delivery – flush data, then commit • Exactly once for connectors that support it (e.g. HDFS)
- 30. Format Converters • Abstract serialization agnostic to connectors • Convert between Kafka Connect Data API (Connectors) and serialized bytes • JSON and Avro currently supported
- 31. Connector Developer APIs class Connector { abstract void start(props); abstract void stop(); abstract Class<? extends Task> taskClass(); abstract List<Map<…>> taskConfigs(maxTasks); … } class Source/SinkTask { abstract void start(props); abstract void stop(); abstract List<SourceRecord> poll(); abstract void put(records); abstract void commit(); … }
- 32. Kafka Connect & Spark Streaming
- 33. Kafka Connect Today • Confluent open source: HDFS, JDBC • Connector Hub: connectors.confluent.io • Examples: MySQL, MongoDB, Twitter, Solr, S3, MQTT, Counchbase, Vertica, Cassandra, Elastic Search, HBase, Kudu, Attunity, JustOne, Striim, Bloomberg .. • Improved connector control (0.10.0)
- 34. THANK YOU! Guozhang Wang | [email protected] | @guozhangwang Confluent – Afternoon Break Sponsor for Spark Summit • Jay Kreps – I Heart Logs book signing and giveaway • 3:45pm – 4:15pm in Golden Gate Kafka Training with Confluent University • Kafka Developer and Operations Courses • Visit www.confluent.io/training Want more Kafka? • Download Confluent Platform Enterprise (incl. Kafka Connect) at http://www.confluent.io/product • Apache Kafka 0.10 upgrade documentation at http://docs.confluent.io/3.0.0/upgrade.html