Dataflow with Apache NiFi - Apache NiFi Meetup - 2016 Hadoop Summit - San Jose
Download as PPTX, PDF10 likes2,404 views
The document discusses dataflow challenges and the architecture of Apache NiFi, highlighting its key features such as guaranteed delivery and data buffering. It outlines the complexities involved in effectively moving data across systems, especially within global enterprises, and introduces NiFi's capabilities for managing dataflow. The presentation also includes a case study and encourages community engagement through upcoming events and collaboration opportunities.
![8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Why is moving data effectively hard?
Standards
Formats
“Exactly Once” Delivery
Protocols
Veracity of Information
Validity of Information
Ensuring Security
Overcoming Security
Compliance
Schemas
Consumers Change
Credential Management
“That [person|team|group]”
Network
“Exactly Once” Delivery](https://image.slidesharecdn.com/2016-160626191921/85/Dataflow-with-Apache-NiFi-Apache-NiFi-Meetup-2016-Hadoop-Summit-San-Jose-8-320.jpg)
![11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Why is moving data effectively hard when scoped internally?
Standards
Formats
“Exactly Once” Delivery
Protocols
Veracity of Information
Validity of Information
Ensuring Security
Overcoming Security
Compliance
Schemas
Consumers Change
Credential Management
“That [person|team|group]”
Network
“Exactly Once” Delivery](https://image.slidesharecdn.com/2016-160626191921/85/Dataflow-with-Apache-NiFi-Apache-NiFi-Meetup-2016-Hadoop-Summit-San-Jose-11-320.jpg)
![13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Why is moving data effectively hard when scoped globally?
Standards
Formats
“Exactly Once” Delivery
Protocols
Veracity of Information
Validity of Information
Ensuring Security
Overcoming Security
Compliance
Schemas
Consumers Change
Credential Management
“That [person|team|group]”
Network
“Exactly Once” Delivery](https://image.slidesharecdn.com/2016-160626191921/85/Dataflow-with-Apache-NiFi-Apache-NiFi-Meetup-2016-Hadoop-Summit-San-Jose-13-320.jpg)
Dataflow with Apache NiFi - Apache NiFi Meetup - 2016 Hadoop Summit - San Jose
- 1. Dataflow with Apache NiFi Aldrin Piri - @aldrinpiri Apache NiFi Meetup Hadoop Summit – San Jose 27 June 2016
- 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Slides available at the conclusion of the talk: http://slideshare.net/aldrinpiri/
- 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Key: 'Apache NiFi’ Value: 'PMC Member' Key: 'Work’ Value: ’Sr. Member of Technical Staff @ Hortonworks' Key: 'Working with NiFi Since’ Value: '2010’
- 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda What is dataflow and what are the challenges? Apache NiFi Architecture Live Demo Community
- 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda What is dataflow and what are the challenges? Apache NiFi Architecture Live Demo Community
- 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Let’s Connect A to B Producers A.K.A Things Anything AND Everything Internet! Consumers • User • Storage • System • …More Things
- 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Moving data effectively is hard Standards: http://xkcd.com/927/
- 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Why is moving data effectively hard? Standards Formats “Exactly Once” Delivery Protocols Veracity of Information Validity of Information Ensuring Security Overcoming Security Compliance Schemas Consumers Change Credential Management “That [person|team|group]” Network “Exactly Once” Delivery
- 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Let’s Connect Lots of As to Bs to As to Cs to Bs to Δs to Cs to ϕs Let’s consider the needs of a courier service Physical Store Gateway Server Mobile Devices Registers Server Cluster Distribution Center Core Data Center at HQ Server Cluster On Delivery Routes Trucks Deliverers Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/ Deliverer: Rigo Peter, https://thenounproject.com/rigo/ Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/ Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/
- 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Great! I am collecting all this data! Let’s use it! Finding our needles in the haystack Physical Store Gateway Server Mobile Devices Registers Server Cluster Distribution Center Kafka Core Data Center at HQ Server Cluster Others Storm / Spark / Flink / Apex Kafka Storm / Spark / Flink / Apex On Delivery Routes Trucks Deliverers Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/ Deliverer: Rigo Peter, https://thenounproject.com/rigo/ Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/ Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/
- 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Why is moving data effectively hard when scoped internally? Standards Formats “Exactly Once” Delivery Protocols Veracity of Information Validity of Information Ensuring Security Overcoming Security Compliance Schemas Consumers Change Credential Management “That [person|team|group]” Network “Exactly Once” Delivery
- 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Let’s Connect Lots of As to Bs to As to Cs to Bs to Δs to Cs to ϕs Oh, that courier service is global
- 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Why is moving data effectively hard when scoped globally? Standards Formats “Exactly Once” Delivery Protocols Veracity of Information Validity of Information Ensuring Security Overcoming Security Compliance Schemas Consumers Change Credential Management “That [person|team|group]” Network “Exactly Once” Delivery
- 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved The Unassuming Line: A Case Study We’ve seen a few lines show up in the wild thus far Internet! Inter- & Intra- connections in our global courier enterprise Spotlight: Arthur Lacôte, https://thenounproject.com/turo/
- 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Dataflow Line Anatomy 101 Let’s dissect what this line typically represents Fig 1. Lineus Worldwidewebus. Common Name: Internet! Script or Application Script or Application Data Data Disparate Transport Mechanisms
- 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Dataflow Line Anatomy 201 Sometimes that transport is just more lines Fig 1. Lineus Worldwidewebus. Common Name: Internet! Script or Application Script or Application Line Inception Data Data
- 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Dataflow Line Anatomy 301 But those lines could also have components… Fig 1. Lineus Worldwidewebus. Common Name: Internet! Fig 2. Good Recursion Joke NoSuchJokeException footage not found
- 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda What is dataflow and what are the challenges? Apache NiFi Architecture Live Demo Community
- 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache NiFi Key Features • Guaranteed delivery • Data buffering - Backpressure - Pressure release • Prioritized queuing • Flow specific QoS - Latency vs. throughput - Loss tolerance • Data provenance • Supports push and pull models • Recovery/recording a rolling log of fine- grained history • Visual command and control • Flow templates • Pluggable/multi-role security • Designed for extension • Clustering
- 20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache NiFi Subproject: MiNiFi Let me get the key parts of NiFi close to where data begins and provide bidrectional communication NiFi lives in the data center. Give it an enterprise server or a cluster of them. MiNiFi lives as close to where data is born and is a guest on that device or system
- 21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Let’s revisit our courier service from the perspective of NiFi Physical Store Gateway Server Mobile Devices Registers Server Cluster Distribution Center Kafka Core Data Center at HQ Server Cluster Others Storm / Spark / Flink / Apex Kafka Storm / Spark / Flink / Apex On Delivery Routes Trucks Deliverers Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/ Deliverer: Rigo Peter, https://thenounproject.com/rigo/ Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/ Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/ Client Libraries Client Libraries MiNiFi MiNiFi NiFi NiFi NiFi NiFi NiFi NiFi Client Libraries
- 22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache NiFi Managed Dataflow SOURCES REGIONAL INFRASTRUCTURE CORE INFRASTRUCTURE
- 23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved NiFi is based on Flow Based Programming (FBP) FBP Term NiFi Term Description Information Packet FlowFile Each object moving through the system. Black Box FlowFile Processor Performs the work, doing some combination of data routing, transformation, or mediation between systems. Bounded Buffer Connection The linkage between processors, acting as queues and allowing various processes to interact at differing rates. Scheduler Flow Controller Maintains the knowledge of how processes are connected, and manages the threads and allocations thereof which all processes use. Subnet Process Group A set of processes and their connections, which can receive and send data via ports. A process group allows creation of entirely new component simply by composition of its components.
- 24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved FlowFiles & Data Agnosticism NiFi is data agnostic! But, NiFi was designed understanding that users can care about specifics and provides tooling to interact with specific formats, protocols, etc. ISO 8601 - http://xkcd.com/1179/ Robustness principle Be conservative in what you do, be liberal in what you accept from others“
- 25. 25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved FlowFiles are like HTTP data HTTP Data FlowFile HTTP/1.1 200 OK Date: Sun, 10 Oct 2010 23:26:07 GMT Server: Apache/2.2.8 (CentOS) OpenSSL/0.9.8g Last-Modified: Sun, 26 Sep 2010 22:04:35 GMT ETag: "45b6-834-49130cc1182c0" Accept-Ranges: bytes Content-Length: 13 Connection: close Content-Type: text/html Hello world! Standard FlowFile Attributes Key: 'entryDate’ Value: 'Fri Jun 17 17:15:04 EDT 2016' Key: 'lineageStartDate’ Value: 'Fri Jun 17 17:15:04 EDT 2016' Key: 'fileSize’ Value: '23609' FlowFile Attribute Map Content Key: 'filename’ Value: '15650246997242' Key: 'path’ Value: './’ Binary Content * Header Content
- 26. 26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda What is dataflow and what are the challenges? Apache NiFi Architecture Live Demo Community
- 27. 27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved OS/Host JVM Flow Controller Web Server Processor 1 Extension N FlowFile Repository Content Repository Provenance Repository Local Storage OS/Host JVM Flow Controller Web Server Processor 1 Extension N FlowFile Repository Content Repository Provenance Repository Local Storage Architecture OS/Host JVM NiFi Cluster Manger – Request Replicator Web Server Master NiFi Cluster Manager (NCM) OS/Host JVM Flow Controller Web Server Processor 1 Extension N FlowFile Repository Content Repository Provenance Repository Local Storage Slaves NiFi Nodes
- 28. 28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved NiFi Architecture – Repositories - Pass by reference FlowFile Content Provenance F1 C1 C1 P1 F1 Excerpt of demo flow… What’s happening inside the repositories… BEFORE AFTER F2 C1 C1 P3 F2 – Clone (F1) F1 C1 P2 F1 – Route P1 F1 – Create
- 29. 29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved NiFi Architecture – Repositories – Copy on Write FlowFile Content Provenance F1 C1 C1 P1 F1 - CREATE Excerpt of demo flow… What’s happening inside the repositories… BEFORE AFTER F1 C1 F1.1 C2 C2 (encrypted) C1 (plaintext) P2 F1.1 - MODIFY P1 F1 - CREATE
- 30. 30 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda What is dataflow and what are the challenges? Apache NiFi Architecture Demo Community
- 31. 31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Learn, Share at Birds of a Feather Streaming, DataFlow & Cybersecurity Thursday June 30 6:30 pm, Ballroom C
- 32. 32 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Thank You
- 33. 33 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda What is dataflow and what are the challenges? Apache NiFi Architecture Demo Community
- 34. 34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Matured at NSA 2006-2014 Brief history of the Apache NiFi Community • Contributors from Government and several commercial industries • Releases on a 6-8 week schedule • Apache NiFi 1.0.0. release on the horizon • Zero-Master Clustering Code developed at NSA 2006 Today Achieved TLP status in just 7 months July 2015 Code available open source ASL v2 November 2014
- 35. 35 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Extension / Integration Points NiFi Term Description Flow File Processor Push/Pull behavior. Custom UI Reporting Task Used to push data from NiFi to some external service (metrics, provenance, etc..) Controller Service Used to enable reusable components / shared services throughout the flow REST API Allows clients to connect to pull information, change behavior, etc..
- 36. 36 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
- 37. 37 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
- 38. 38 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Learn more and join us! Apache NiFi site http://nifi.apache.org Subproject MiNiFi site http://nifi.apache.org/minifi/ Subscribe to and collaborate at [email protected] [email protected] Submit Ideas or Issues https://issues.apache.org/jira/browse/NIFI Follow us on Twitter @apachenifi
Editor's Notes
- #28: Introduce the architecture of NiFi, describe major system components, and describe the single node and clustering models. For each component describe its available (and potential)deployment models (relate it to Hadoop).