Running Apache NiFi with Apache Spark : Integration Options
19 likes6,390 views
The document outlines the integration of Apache NiFi with Apache Spark and Kafka, focusing on data flow management, streaming analytics, and security features. It highlights various integration options and architectures that enhance real-time data processing and event management. Apache Livy is introduced as a REST interface for interacting with Spark, emphasizing its role in executing batch and interactive jobs securely.
Running Apache NiFi with Apache Spark : Integration Options
- 1. 1 © Hortonworks Inc. 2011 – 2018 All Rights Reserved Apache NiFi Integration with Apache Spark Timothy Spann, Solutions Engineer
- 2. 2 © Hortonworks Inc. 2011 – 2018 All Rights Reserved Disclaimer à This document may contain product features and technology directions that are under development, may be under development in the future or may ultimately not be developed. à Technical feasibility, market demand, user feedback, and the Apache Software Foundation community development process can all effect timing and final delivery. à This document’s description of these features and technology directions does not represent a contractual commitment, promise or obligation from Hortonworks to deliver these features in any generally available product. à Product features and technology directions are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. à Since this document contains an outline of general product development plans, customers should not rely upon it when making a purchase decision.
- 3. 3 © Hortonworks Inc. 2011 – 2018 All Rights Reserved Integration Options § Apache Spark Integration via Kafka and Spark Streaming (1.6+) § Apache Spark Integration via Kafka and Spark Structured Streaming (2.2+) § Apache Spark Integration via Apache Livy
- 4. 4 Apache Kafka and Apache NiFi Integration +
- 5. 5 © Hortonworks Inc. 2011 – 2018 All Rights Reserved NiFi and Kafka Are Complementary NiFi Provide dataflow solution • Centralized management, from edge to core • Great traceability, event level data provenance starting when data is born • Interactive command and control – real time operational visibility • Dataflow management, including prioritization, back pressure, and edge intelligence • Visual representation of global dataflow Kafka Provide durable stream store • Low latency • Distributed data durability • Decentralized management of producers & consumers +
- 6. 6 © Hortonworks Inc. 2011 – 2018 All Rights Reserved Integrated Provisioning and Security Kafka 1.0 Support To enhance data governance and lineage, users can now manage access control policies using resource or tag-based security in Ranger for Kafka 1.0 clusters. Users can now install, configure, manage, upgrade, monitor, and secure Kafka 1.0 clusters with Ambari. New processors in NiFi and Streaming Analytics Manager support Kafka 1.0 features including message headers and transactions.
- 7. 7 © Hortonworks Inc. 2011 – 2018 All Rights Reserved Apache NiFi and Kafka 1.0 – Use Case for Kafka Message Headers
- 8. 8 Apache Spark – Apache Kafka – Apache NiFi Architecture
- 9. 9 © Hortonworks Inc. 2011 – 2018 All Rights Reserved Join Architecture Example Acquire/Move Routing & Filtering Parse Analyze Model Topic 1 Topic 2 AggregateCorrolate Pattern Matching JSON Data AVRO Data Windowing Aggregations Spark Processing Flow Management Stream Analysis ++
- 10. 10 © Hortonworks Inc. 2011 – 2018 All Rights Reserved Stream Processing Streaming Analytics Manager Machine Learning Distributed queue Buffering Process decoupling Structured Streaming with SQL Orchestration Queueing Simple Event Processing Data Definition Between Environments Schema Versioning
- 11. 11 © Hortonworks Inc. 2011 – 2018 All Rights Reserved Key Integration Points – NiFi & Kafka NiFi MiNiFi MiNiFi MiNiFi Kafka Consumer 1 Consumer 2 Consumer N • Producer Processors (Main) • PublishKafka_0_11 (0.10 Kafka Client) • PublishKafka_1_0 (1.0 Kafka Client) • PublishKafkaRecord_0_11 (0.11 Kafka Client) • PublishKafkaRecord_1_0 (1.0 Kafka Client) +
- 12. 12 © Hortonworks Inc. 2011 – 2018 All Rights Reserved Key Integration Points – NiFi & Kafka Kafka Producer 1 Producer 2 Producer N NiFi Destination 1 Destination 2 Destination 3 • Consumer Processors (Main) • ConsumeKafka_0_11 (0.11 Kafka Client) • ConsumeKafka_1_0 (1.0 Kafka Client) • ConsumeKafkaRecord_0_11 (0.11 Kafka Client) • ConsumeKafkaRecord_1_0 (1.0 Kafka Client) +
- 13. 13 © Hortonworks Inc. 2011 – 2018 All Rights Reserved Better Together NiFiMiNiFi Kafka Spark Incoming Topic Results Topic PublishKafka ConsumeKafka Destinations MiNiFi • MiNiFi – Collection, filtering, and prioritization at the edge • NiFi - Central data flow management, routing, enriching, and transformation • Kafka - Central messaging bus for subscription by downstream consumers • Spark - Streaming analytics focused on complex event processing + +SR
- 14. 14 © Hortonworks Inc. 2011 – 2018 All Rights Reserved NiFi PublishKafkaRecord_1_0 Apache NiFi - Node 1 Apache Kafka Topic 1 - Partition 1 Topic 1 - Partition 2 PublishKafka Apache NiFi – Node 2 PublishKafka = Concurrent Task • Each NiFi node runs an instance of PublishKafkaRecord_1_0 • Each instance has one or more concurrent tasks (threads) • Each concurrent task is an independent producer, sends data round-robin to partitions of a topic • Records with Schemas for Performance +
- 15. 15 Apache Spark Streaming – Apache Kafka – Apache NiFi Architecture
- 16. 16 © Hortonworks Inc. 2011 – 2018 All Rights Reserved Spark Streaming à Spark Streaming is an extension of Spark-core API that supports scalable, high throughput and fault-tolerant streaming applications. à Data can be ingested from various data sources like Kafka, Flume, Twitter, ZeroMQ or TCP sockets à Data is processed using the now-familiar API: map, filter, reduce, join and window à Processed data can be stored in databases, filesystems, or live dashboards
- 17. 17 © Hortonworks Inc. 2011 – 2018 All Rights Reserved Apache Spark Streaming Integration via Kafka https://community.hortonworks.com/content/kbentry/173818/hdp-264-hdf-31-apache-spark-streaming-integration.html
- 18. 18 © Hortonworks Inc. 2011 – 2018 All Rights Reserved Apache Spark Streaming Integration via Kafka
- 19. 19 Apache Spark Structured Streaming – Apache Kafka – Apache NiFi Architecture
- 20. 20 © Hortonworks Inc. 2011 – 2018 All Rights Reserved Apache Spark Structured Streaming Integration via Kafka https://community.hortonworks.com/articles/91379/spark-structured-streaming-with-nifi-and-kafka-usi.html https://jaceklaskowski.gitbooks.io/spark-structured-streaming/spark-sql-streaming-KafkaSource.html https://community.hortonworks.com/content/kbentry/174105/hdp-264-hdf-31-apache-spark-structured-streaming- i.html val records = spark. readStream. format("kafka"). option("subscribe", "smartPlug2"). option("kafka.bootstrap.servers", "mykafkabroker:6667").load
- 21. 21 © Hortonworks Inc. 2011 – 2018 All Rights Reserved Apache NiFi – Apache Kafka – Apache Spark
- 22. 22 Apache Spark – Apache Livy
- 23. 23 © Hortonworks Inc. 2011 – 2018 All Rights Reserved Introducing Apache Livy à Apache Livy is the open source REST interface for interacting with Apache Spark from anywhere à Installed as Spark2 Ambari Service Livy Client HTTP HTTP (RPC) Spark Interactive Session SparkContext Spark Batch Session SparkContext Livy Server https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_spark-component- guide/content/ch_submit-spark-apps-livy.html
- 24. 24 © Hortonworks Inc. 2011 – 2018 All Rights Reserved Livy Server as a Session Management Service Livy Server Remote Spark Driver Session Remote Context Interactive REST API Batch REST API Standard Spark Batch Job Spark Executor Spark Executor Spark Executor Spark Executor https://livy.incubator.apache.org/docs/latest/rest-api.html
- 25. 25 Apache Spark – Apache Livy – Apache NiFi Integration
- 26. 26 © Hortonworks Inc. 2011 – 2018 All Rights Reserved SQL Architecture Example Routing & Filtering Parse Analyze Session 1 Session 1 AggregateSQL JSON Data Spark Processing Flow Management Analytics
- 27. 27 © Hortonworks Inc. 2011 – 2018 All Rights Reserved NiFi to Spark Processing Streaming Analytics Manager Machine Learning REST API Enterprise Tested Secure Structured Streaming with SQL Orchestration Queueing Simple Event Processing Data Definition Between Environments Schema Versioning
- 28. 28 © Hortonworks Inc. 2011 – 2018 All Rights Reserved Key Integration Points – NiFi & Spark NiFi MiNiFi MiNiFi MiNiFi Livy Spark Spark 2 Spark N • Processor and Controller • ExecuteSparkInteractive – setup job and code to Livy Session Service • LivySessionService – manages Spark Livy connection pool + +
- 29. 29 © Hortonworks Inc. 2011 – 2018 All Rights Reserved Better Together NiFiMiNiFi Livy Spark Session Batch ExecuteSpark Interactive MiNiFi • MiNiFi – Collection, filtering, and prioritization at the edge • NiFi - Central data flow management, routing, enriching, and transformation • Livy – Secure HTTPS connection to running Spark batch and sessions jobs with cached RDD sharing and a live Spark context. • Spark - Streaming analytics focused on complex event processing + + LivySessionService
- 30. 30 Apache Spark – Apache Livy – Apache NiFi Architecture
- 31. 31 © Hortonworks Inc. 2011 – 2018 All Rights Reserved Apache Spark Integration via Apache Livy
- 32. 32 © Hortonworks Inc. 2011 – 2018 All Rights Reserved Apache Spark Integration via Apache Livy https://community.hortonworks.com/articles/171787/hdf-31-executing-apache-spark-via-executesparkinte.html https://community.hortonworks.com/articles/171893/hdf-31-executing-apache-spark-via-executesparkinte-1.html
- 33. 33 © Hortonworks Inc. 2011 – 2018 All Rights Reserved
- 34. 34 © Hortonworks Inc. 2011 – 2018 All Rights Reserved Questions? Hortonworks Community Connection: Data Ingestion and Streaming https://community.hortonworks.com/
- 35. 35 © Hortonworks Inc. 2011 – 2018 All Rights Reserved Contact https://community.hortonworks.com/users/9304/tspann.html https://dzone.com/users/297029/bunkertor.html https://www.meetup.com/futureofdata-princeton/ https://twitter.com/PaaSDev https://community.hortonworks.com/articles/174105/hdp-264-hdf-31-apache-spark-structured-streaming-i.html
- 36. 36 © Hortonworks Inc. 2011 – 2018 All Rights Reserved Hortonworks Community Connection Read access for everyone, join to participate and be recognized • Full Q&A Platform (like StackOverflow) • Knowledge Base Articles • Code Samples and Repositories
- 37. 37 © Hortonworks Inc. 2011 – 2018 All Rights Reserved Community Engagement Participate now at: community.hortonworks.com© Hortonworks Inc. 2011 – 2015. All Rights Reserved 4,000+ Registered Users 10,000+ Answers 15,000+ Technical Assets One Website!
- 38. 38 © Hortonworks Inc. 2011 – 2018 All Rights Reserved Register at dataworkssummit.com #DWS18 Berlin, Germany San Jose, California APRIL 16-19, 2018 | ESTREL HOTEL JUNE 17-21, 2018 | MCENERY CONVENTION CENTER