State of the Apache NiFi Ecosystem & Community
Download as PPTX, PDF0 likes140 views
The document provides an overview of the Apache NiFi ecosystem, detailing its key features, principles, and the importance of data pipeline management. It introduces the Apache NiFi Registry and MiNiFi, emphasizing their roles in enhancing data flow management and integration across various systems. Additionally, it highlights community engagement and resources for further collaboration and learning within the NiFi ecosystem.
State of the Apache NiFi Ecosystem & Community
- 1. 1 © Hortonworks Inc. 2011–2018. All rights reserved. State of the Apache NiFi Ecosystem & Community Aldrin Piri @aldrinpiri
- 2. 2 © Hortonworks Inc. 2011–2018. All rights reserved. About Me • Sadly, not an Accumulo expert • I was using cbshell before it went mainstream, though • Apache NiFi PMC Member and Committer • Hortonworks
- 3. 3 © Hortonworks Inc. 2011–2018. All rights reserved. Agenda • Apache NiFi • Apache NiFi Registry • Apache NiFi - MiNiFi • Community
- 4. 4 © Hortonworks Inc. 2011–2018. All rights reserved. Apache NiFi – A Primer
- 5. 5 © Hortonworks Inc. 2011–2018. All rights reserved. The Problem at Hand Producers A.K.A Things Anything AND Everything Internet! Consumers • User • Storage • System • …More Things
- 6. 6 © Hortonworks Inc. 2011–2018. All rights reserved. Moving data effectively is hard Standards: http://xkcd.com/927/
- 7. 7 © Hortonworks Inc. 2011–2018. All rights reserved. Data Pipeline: http://xkcd.com/927/ Empower Users to Reason About Their Data Flow
- 8. 8 © Hortonworks Inc. 2011–2018. All rights reserved. 8 Apache NiFi Key Features and Principles • Guaranteed delivery • Data buffering - Backpressure - Pressure release • Prioritized queuing • Flow specific QoS - Latency vs. throughput - Loss tolerance • Data provenance • Recovery/recording a rolling log of fine-grained history • Visual command and control • Pluggable/multi-role security • Designed for extension • Clustering
- 9. 9 © Hortonworks Inc. 2011–2018. All rights reserved. NiFi & Data Agnosticism • NiFi is data agnostic! • But, NiFi was designed understanding that users can care about specifics and provides tooling to interact with specific formats, protocols, etc. ISO 8601 - http://xkcd.com/1179/ Robustness principle Be conservative in what you do, be liberal in what you accept from others“
- 10. 10 © Hortonworks Inc. 2011–2018. All rights reserved. 1 0
- 11. 11 © Hortonworks Inc. 2011–2018. All rights reserved.
- 12. 12 © Hortonworks Inc. 2011–2018. All rights reserved. Apache NiFi – 0.x to 1.x and Beyond
- 13. 13 © Hortonworks Inc. 2011–2018. All rights reserved. Integrate All the Systems: 274+ Processors, 57 Controller Services Hash Extract Merge Duplicate Scan GeoEnrich Replace ConvertSplit Translate Route Content Route Context Route Text Control Rate Distribute Load Generate Table Fetch Jolt Transform JSON Prioritized Delivery Encrypt Tail Evaluate Execute All Apache project logos are trademarks of the ASF and the respective projects. Fetch HTTP Syslog Email HTML Image HL7 FTP UDP XML SFTP AMQP WebSocket Parse Records Convert Records
- 14. 14 © Hortonworks Inc. 2011–2018. All rights reserved. 1.X Clustering Architecture OS/Host JVM Flow Controller Web Server Processor 1 Extension N FlowFile Repository Content Repository Provenance Repository Local Storage Standalone Cluster
- 15. 15 © Hortonworks Inc. 2011–2018. All rights reserved. • Now flowfile content can contain many “record” elements • Read and write with *Reader and *Writer Controller Services • Perform lookups, routing, conversion, SQL queries, validation, and more… • 1 + 1 flowfiles = 2 flowfiles Record Parsing
- 16. 16 © Hortonworks Inc. 2011–2018. All rights reserved. New and Upcoming Features • Enhanced support for containerization • Auto load-balancing in clustered environments (NIFI-5516) • Node decommissioning • Enhanced compatibility with Java 9+ • Introduction of the Apache NiFi Flow Design System • Common UI/UX across the ecosystem
- 17. 17 © Hortonworks Inc. 2011–2018. All rights reserved. Evolving the NiFi Platform
- 18. 18 © Hortonworks Inc. 2011–2018. All rights reserved. • Version my flows? • Drive CI/CD processes? • Migrate flows between environments? • Provision distributions of NiFi with a set of components? • Make reference datasets/extensions available to the entirety of my data flow? • Certify / Audit / Sign-off on flows as compliant per regulations? How can I … How do I ... What about ... Listening to our community
- 19. 19 © Hortonworks Inc. 2011–2018. All rights reserved. Capturing the essence of a flow in your organization • The n-dimensions of data flow • Consider a flowfile to be a singular event at a given juncture in its processing A flow is the directed graph of processing at a given point in time With each component’s: Configuration Version Referenced Assets
- 20. 20 © Hortonworks Inc. 2011–2018. All rights reserved. Apache NiFi Registry
- 21. 21 © Hortonworks Inc. 2011–2018. All rights reserved. Operator Experience
- 22. 22 © Hortonworks Inc. 2011–2018. All rights reserved. Challenges • Templates • Updates/replacement • Sensitive property replacement • flow.xml.gz migration • Key synchronization • Environment promotion • Approval processes • Verifiability
- 23. 23 © Hortonworks Inc. 2011–2018. All rights reserved. Template Replacement • Export a new version of template • Transfer (somehow) • Verify? • Import onto canvas side-by-side existing flow • Stop processors • Empty queues • Reconnect queues • Start • Pray?
- 24. 24 © Hortonworks Inc. 2011–2018. All rights reserved. Template Replacement
- 25. 25 © Hortonworks Inc. 2011–2018. All rights reserved. • Previously, flows were exported via XML templates • Didn’t contain sensitive values • Couldn’t be updated in-place • No tracking system • NiFi Registry brings asset management as first-class citizen to NiFi • Flows can be versioned Introducing Apache NiFi Registry 0.3.0 NiFi Registry for Dataflows
- 26. 26 © Hortonworks Inc. 2011–2018. All rights reserved. Flows can be promoted between environments • Connect multiple NiFi instances to a NiFi Registry instance • Communicate between multiple NiFi Registry instances • via multiple Registry Clients • via NiFi CLI
- 27. 27 © Hortonworks Inc. 2011–2018. All rights reserved. Extensibility • Git-backed persistence • Share flows via GitHub, etc. • Commit hooks • Register a hook & action • “When a new version of the flow is committed to QA Registry, email the QA team and post in the QA Deploy Slack channel” • Pluggable DB implementations
- 28. 28 © Hortonworks Inc. 2011–2018. All rights reserved.
- 29. 29 © Hortonworks Inc. 2011–2018. All rights reserved.
- 30. 30 © Hortonworks Inc. 2011–2018. All rights reserved.
- 31. 31 © Hortonworks Inc. 2011–2018. All rights reserved.
- 32. 32 © Hortonworks Inc. 2011–2018. All rights reserved. Registry acts as an enabler • SDLC • Manage variables, sensitive properties for environments • Extension Registry • Association/tagging of data with the flow that created it
- 33. 33 © Hortonworks Inc. 2011–2018. All rights reserved. Apache NiFi – MiNiFi
- 34. 34 © Hortonworks Inc. 2011–2018. All rights reserved. Apache NiFi - MiNiFi • Let me get the key parts of NiFi close to where data begins • Bidirectional data transfer • Greater illuminate journey with provenance • NiFi lives in the data center. Give it an enterprise server or a cluster of them. • MiNiFi lives as close to where data is born and is a guest on that device or system
- 35. 35 © Hortonworks Inc. 2011–2018. All rights reserved. Apache NiFi - MiNiFi: Scoping • Go small: Java – Write once, run anywhere* • Feature parity and reuse of core NiFi libraries • Go smaller: C/C++ – Write once**, run anywhere • Go smallest: Write n-many times, embed, run anywhere Language libraries to support tagging, FlowFile format, Site to Site protocol, and provenance generation without a full processing framework – Language SDKs, Mobile Platforms Provide all the key principles of NiFi in varying, smaller footprints
- 36. 36 © Hortonworks Inc. 2011–2018. All rights reserved. Apache NiFi - MiNiFi: Centralized Command & Control (C2) • Provide flow updates, information and assets to instances where they live • Act as a gateway to/from network enclaves • https://s.apache.org/minifi-c2-design-proposal Extend the reach of user experience and operations
- 37. 37 © Hortonworks Inc. 2011–2018. All rights reserved. Community & Ecosystem
- 38. 38 © Hortonworks Inc. 2011–2018. All rights reserved. The Evolution of Apache NiFi • Our core substrate for data flow is NiFi & MiNiFi • Command and Control facilitates operations and management of components • Registry for common tasks with disparate resources across the NiFi ecosystem
- 39. 39 © Hortonworks Inc. 2011–2018. All rights reserved. Managing data flow for a courier service Physical Store Gateway Server Mobile Devices Registers Server Cluster Distribution Center Kafka Core Data Center at HQ Server Cluster Others Storm / Spark / Flink / Apex Kafka Storm / Spark / Flink / Apex On Delivery Routes Trucks Deliverers Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/ Deliverer: Rigo Peter, https://thenounproject.com/rigo/ Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/ Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/ Client Libraries Client Libraries MiNiFi MiNiFi NiFi NiFi NiFi NiFi NiFi NiFi Client Libraries
- 40. 40 © Hortonworks Inc. 2011–2018. All rights reserved. Why the Apache NiFi Ecosystem? • Moving data is multifaceted in its challenges and these are present in different contexts at varying scopes • Provide components and a platform with common tooling and extensions that are commonly needed but be flexible for extension in all aspects • Allow organizations to integrate with their existing infrastructure • Empower folks managing your infrastructure to make changes and reason about issues that are occurring • Data Provenance to show context and data’s journey • User Interface/Experience a key component
- 41. 41 © Hortonworks Inc. 2011–2018. All rights reserved. Community Health
- 42. 42 © Hortonworks Inc. 2011–2018. All rights reserved. Project Sites • NiFi: https://nifi.apache.org • Subproject MiNiFi: https://nifi.apache.org/minifi/ • Subproject Registry: http://nifi.apache.org/registry.html Subscribe to and collaborate at • http://nifi.apache.org/mailing_lists.html Follow us on Twitter @apachenifi Apache NiFi Community and Ecosystem Resources Learn More and Join Us!
- 43. 43 © Hortonworks Inc. 2011–2018. All rights reserved. Thanks! Questions?
Editor's Notes
- #15: Introduce the architecture of NiFi, describe major system components, and describe the single node and clustering models. For each component describe its available (and potential)deployment models (relate it to Hadoop).