![6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Why is moving data effectively hard?
Standards
Formats
“Exactly Once” Delivery
Protocols
Veracity of Information
Validity of Information
Ensuring Security
Overcoming Security
Compliance
Schemas
Consumers Change
Credential Management
“That [person|team|group]”
Network
“Exactly Once” Delivery](https://image.slidesharecdn.com/avant-gardeofapachenififinal-160909073936/85/The-Avant-garde-of-Apache-NiFi-6-320.jpg)
![7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
• Web-based User Interface for creating, monitoring,
& controlling data flows
• Directed graphs of data routing and transformation
• Highly configurable - modify data flow at runtime,
dynamically prioritize data
• Easily extensible through development of custom
components
• Data Provenance tracks data through entire system
[1] https://nifi.apache.org/
Dataflow
Apache NiFi](https://image.slidesharecdn.com/avant-gardeofapachenififinal-160909073936/85/The-Avant-garde-of-Apache-NiFi-7-320.jpg)
The Avant-garde of Apache NiFi
- 1. The Avant-garde of Apache NiFi Joe Percivall - @JPercivall Hadoop Summit – Melbourne 31 August 2016
- 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved About Me • Software Engineer at Hortonworks • Apache NiFi committer and PMC member • Github: github.com/JPercivall
- 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda • Intro to NiFi • What’s new in NiFi 1.0.0 • Intro to MiNiFi • MiNiFi Architecture • NiFi & MiNiFi Demo
- 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda • Intro to Apache NiFi • What’s new in NiFi 1.0.0 • Intro to MiNiFi • MiNiFi Architecture • NiFi & MiNiFi Demo
- 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Let’s Connect A to B Producers A.K.A Things Anything AND Everything Internet! Consumers • User • Storage • System • …More Things
- 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Why is moving data effectively hard? Standards Formats “Exactly Once” Delivery Protocols Veracity of Information Validity of Information Ensuring Security Overcoming Security Compliance Schemas Consumers Change Credential Management “That [person|team|group]” Network “Exactly Once” Delivery
- 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved • Web-based User Interface for creating, monitoring, & controlling data flows • Directed graphs of data routing and transformation • Highly configurable - modify data flow at runtime, dynamically prioritize data • Easily extensible through development of custom components • Data Provenance tracks data through entire system [1] https://nifi.apache.org/ Dataflow Apache NiFi
- 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache NiFi Key Features • Guaranteed delivery • Data buffering - Backpressure - Pressure release • Prioritized queuing • Flow specific QoS - Latency vs. throughput - Loss tolerance • Data provenance • Supports push and pull models • Recovery/recording a rolling log of fine- grained history • Visual command and control • Flow templates • Pluggable/multi-role security • Designed for extension • Clustering
- 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Simplified Example Let’s consider the needs of a courier service Physical Store Gateway Server Mobile Devices Registers Server Cluster Distribution Center Core Data Center at HQ Server Cluster On Delivery Routes Trucks Deliverers Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/ Deliverer: Rigo Peter, https://thenounproject.com/rigo/ Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/ Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/
- 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Great! I am collecting all this data! Let’s use it! Finding our needles in the haystack Physical Store Gateway Server Mobile Devices Registers Server Cluster Distribution Center Kafka Core Data Center at HQ Server Cluster Others Storm / Spark / Flink / Apex Kafka Storm / Spark / Flink / Apex Trucks Deliverers Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/ Deliverer: Rigo Peter, https://thenounproject.com/rigo/ Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/ Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/ On Delivery Routes
- 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Let’s revisit our courier service from the perspective of NiFi Physical Store Gateway Server Mobile Devices Registers Server Cluster Distribution Center Kafka Core Data Center at HQ Server Cluster Others Storm / Spark / Flink / Apex Kafka Storm / Spark / Flink / Apex Trucks Deliverers Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/ Deliverer: Rigo Peter, https://thenounproject.com/rigo/ Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/ Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/ NiFi NiFi NiFi NiFi NiFi NiFi On Delivery Routes
- 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Fundamental Terminology FlowFile • Unit of data moving through the system • Content + Attributes (key/value pairs) Processor • Performs the work, can access FlowFiles Connection • Links between processors • Queues that can be dynamically prioritized git clone https://github.com/JPercivall/nifi-developer-tutorial.git
- 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda • Intro to NiFi • What’s new in NiFi 1.0.0 • Intro to MiNiFi • MiNiFi Architecture • NiFi & MiNiFi Demo
- 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache NiFi-1.0.0 Zero Master Clustering UI Refresh Multi-tenant authorization and internal authorization/policy management 15+ new components Over 450 tickets closed!
- 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Zero Master Clustering
- 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Zero Master Clustering
- 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved UI Refresh & Multi-tenant Authorization
- 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda • Intro to NiFi • What’s new in NiFi 1.0.0 • Intro to MiNiFi • MiNiFi Architecture • NiFi & MiNiFi Demo
- 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Revisit: Courier service from the perspective of NiFi Physical Store Gateway Server Mobile Devices Registers Server Cluster Distribution Center Core Data Center at HQ Server Cluster Trucks Deliverers Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/ Deliverer: Rigo Peter, https://thenounproject.com/rigo/ Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/ Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/ NiFi NiFi NiFi NiFi NiFi NiFi On Delivery Routes
- 20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Courier service from the perspective of NiFi & MiNiFi Physical Store Gateway Server Mobile Devices Registers Server Cluster Distribution Center Core Data Center at HQ Server Cluster Trucks Deliverers Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/ Deliverer: Rigo Peter, https://thenounproject.com/rigo/ Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/ Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/ Client Libraries Client Libraries MiNiFi MiNiFi NiFi NiFi NiFi NiFi NiFi NiFi Client Libraries On Delivery Routes
- 21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache NiFi MiNiFi Key Features • Guaranteed delivery • Data buffering - Backpressure - Pressure release • Prioritized queuing • Flow specific QoS - Latency vs. throughput - Loss tolerance • Data provenance • Recovery/recording a rolling log of fine- grained history • Designed for extension • Design and Deploy • Warm re-deploys
- 22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache NiFi MiNiFi Key Features • Guaranteed delivery • Data buffering - Backpressure - Pressure release • Prioritized queuing • Flow specific QoS - Latency vs. throughput - Loss tolerance • Data provenance • Recovery/recording a rolling log of fine- grained history • Designed for extension • Design and Deploy • Warm re-deploys
- 23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Visual Command and Control vs. Design and Deploy
- 24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Created to more effectively collect data at the edge
- 25. 25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda • Intro to NiFi • What’s new in NiFi 1.0.0 • Intro to MiNiFi • MiNiFi Architecture • NiFi & MiNiFi Demo
- 26. 26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved NiFi vs MiNiFi Java Processes NiFi Framework Components MiNiFi NiFi Framework User Interface Components NiFi
- 27. 27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved NiFi Java Processes Bootstrap NiFi UI bootstrap.conf nifi.properties flow.xml.gzreads & modifies reads reads starts NiFi MiNiFi
- 28. 28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved MiNiFi Java Processes MiNiFi Bootstrap Configuration Change Notifier(s) bootstrap.conf nifi.properties flow.xml.gz reads reads starts config.ymltransforms reads into NiFi MiNiFi
- 29. 29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Same Extensible Framework (nars) In minifi-0.0.1, the nifi-0.6.1 standard processors are bundled (~20mb) – Tailing a Log – UpdateAttribute – Routing by content or attributes – PutEmail Allows MiNiFi to use NiFi processors
- 30. 30 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Simple Config.yml Tail a rolling file -> Site to Site
- 31. 31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda • Intro to NiFi • What’s new in NiFi 1.0.0 • Intro to MiNiFi • MiNiFi Architecture • NiFi & MiNiFi Demo
- 32. 32 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Courier service from the perspective of NiFi & MiNiFi Physical Store Gateway Server Mobile Devices Registers Server Cluster Distribution Center Core Data Center at HQ Server Cluster Trucks Deliverers Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/ Deliverer: Rigo Peter, https://thenounproject.com/rigo/ Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/ Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/ Client Libraries Client Libraries MiNiFi MiNiFi NiFi NiFi NiFi NiFi NiFi NiFi Client Libraries On Delivery Routes
- 33. 33 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Questions?
- 34. 34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Thank you!
- 35. 35 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Learn more and join us! Apache NiFi site http://nifi.apache.org Subproject MiNiFi site http://nifi.apache.org/minifi/ Subscribe to and collaborate at [email protected] [email protected] Submit Ideas or Issues https://issues.apache.org/jira/browse/NIFI Follow us on Twitter @apachenifi
- 36. 36 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Back-up
- 37. 37 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Matured at NSA 2006-2014 Brief history of the Apache NiFi Community • Contributors from Government and several commercial industries • Releases on a 6-8 week schedule • Apache NiFi 1.0.0. release on the horizon • Zero-Master Clustering Code developed at NSA 2006 Today Achieved TLP status in just 7 months July 2015 Code available open source ASL v2 November 2014
- 38. 38 © Hortonworks Inc. 2011 – 2016. All Rights Reserved A bit more complex Config.yml Tail a rolling File -> Secure Site to Site with Provenance
- 39. 39 © Hortonworks Inc. 2011 – 2016. All Rights Reserved MiNiFi 0.0.1-Java Declarative configuration of processing flows through a YAML configuration file Exporting of provenance events to another NiFi instance via a Reporting Task over Site to Site Flow change configuration watcher implementations that provide reloading a NiFi instance when receiving an updated flow over REST or changes on a file system Providing a mechanism to query an instance's status <40mb binary distribution Release Notes
- 40. 40 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Change notifier update MiNiFi Bootstrap Configuration Change Notifiers 1. Initial state –Both running
- 41. 41 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Change notifier update MiNiFi Bootstrap Configuration Change Notifiers user creates new configuration 2. User sends update through notifier –HTTP(S) post request –Change watched file
- 42. 42 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Change notifier update MiNiFi Bootstrap Configuration Change Notifiers 3. Bootstrap validation –Basic validation –Rest notifier will respond accordingly –Results logged validate new configuration user creates new configuration
- 43. 43 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Change notifier update MiNiFi Bootstrap Configuration Change Notifiers config.yml saves new 4. Bootstrap saves and transforms –Copy old config.yml to a swap file validate new configuration user creates new configuration nifi.properties flow.xml.gz transforms into
- 44. 44 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Change notifier update MiNiFi Bootstrap Configuration Change Notifiers nifi.properties flow.xml.gz attempt restart config.yml saves new reads transforms into 5. Bootstrap attempts restart –MiNiFi reads in the new nifi.properties and flow.xml.gz validate new configuration user creates new configuration
- 45. 45 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Change notifier update 6. Success or Fail –Successful restart continue processing –Failure, rollback to old config –Existing Data is mapped or orphaned MiNiFi Bootstrap Configuration Change Notifiers nifi.properties flow.xml.gz attempt restart config.yml saves new reads transforms into validate new configuration user creates new configuration
Editor's Notes
- #5: The Apache NiFi project as a whole (including MiNiFi) is all about routing getting the right data to the right place.
- #8: 170 different actions bundled by default
- #12: Can put NiFi on a Gateway server but probably don’t want to mess with a UI on ever single one Maybe not best fit
- #13: Same page with terminology
- #14: The Apache NiFi project as a whole (including MiNiFi) is all about routing getting the right data to the right place.
- #19: The Apache NiFi project as a whole (including MiNiFi) is all about routing getting the right data to the right place.
- #20: Can put NiFi on a Gateway server but probably don’t want to mess with a UI on ever single one Maybe not best fit
- #21: Let me get the key parts of NiFi close to where data begins and provide bidrectional communication NiFi lives in the data center. Give it an enterprise server or a cluster of them. MiNiFi lives close to where data is born and may be a guest on that device or system
- #25: Aggregator vs. Agent
- #26: Pi -> AWS NiFi
- #27: NiFi 0.7.0 is ~600mb, but most of that is UI and components Framework – put a new wrapper on the framework, or in maven terms, we kept the underlying modules and wrote minifi-framework-core replacing nifi-framework-core MiNifI packaged components ~20mb
- #28: Initiates with ./bin/nifi.sh start
- #29: user, only need bootstrap and config.yml nifi.properties and flow.xml are implementation details
- #30: Since it uses the same underlying framework, MiNiFi is extensible exactly like NiFi NiFi 0.7.0 has 155 different processors to chose from Pub/sub communication (ie. Kafka, MQTT) Endpoint delivery (ie. HDFS, HBase) Format validation/transformation (ie. JSON, XML) Yandex Language Translation
- #31: Nifi.properties -> omitted for defaults
- #33: Truck: need to be notified of high temp or humidity
- #40: Now that we’ve covered the basic architecture of MiNiFi we can talk about 0.0.1