Data at Scales and the Values of Starting Small with Apache NiFi & MiNiFi

Data at Scales &
the Values of
Starting Small
Aldrin Piri - @aldrinpiri
DataWorks Summit 2017 – Munich

2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Key: 'Apache NiFi’
Value: 'PMC Member'
Key: 'Work’
Value: ’Sr. Member of Technical Staff @ Hortonworks'
Key: 'Working with NiFi Since’
Value: '2010’

Agenda
Apache NiFi: A Primer
Apache MiNiFi
Architecture
Apache NiFi: The Ecosystem
Community

Byte Scales for Data
SI Prefix
- 10
0
kilo 10
3
mega 10
6
giga 10
9
tera 10
12
peta 10
15
exa 10
18
zetta 10
21
yotta 10
24
“Big Data”
”everything else”
Greek
for

The Problem at Hand
Producers A.K.A Things
Anything
AND
Everything
Internet!
Consumers
• User
• Storage
• System
• …More Things

Use Case: Courier Service
Physical Store
Gateway
Server
Mobile Devices
Registers
Server Cluster
Distribution Center
Kafka
Core Data Center at HQ
Server Cluster
Others
Storm / Spark /
Flink / Apex
Kafka
Storm / Spark / Flink / Apex
On Delivery Routes
Trucks Deliverers
Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/
Deliverer: Rigo Peter, https://thenounproject.com/rigo/
Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/
Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/
NiFi NiFi NiFi NiFi NiFi NiFi
Gathering data from disparate sources
NiFi

Apache NiFi: A Primer
Key Features and Principles
• Guaranteed delivery
• Data buffering
- Backpressure
- Pressure release
• Prioritized queuing
• Flow specific QoS
- Latency vs. throughput
- Loss tolerance
• Data provenance
• Recovery/recording
a rolling log of fine-
grained history
• Visual command and
control
• Flow templates
• Pluggable/multi-role
security
• Designed for extension
• Clustering

NiFi is based on Flow Based Programming (FBP)
FBP Term NiFi Term Description
Information
Packet
FlowFile Each object moving through the system.
Black Box FlowFile
Processor
Performs the work, doing some combination of data routing, transformation,
or mediation between systems.
Bounded
Buffer
Connection The linkage between processors, acting as queues and allowing various
processes to interact at differing rates.
Scheduler Flow
Controller
Maintains the knowledge of how processes are connected, and manages the
threads and allocations thereof which all processes use.
Subnet Process
Group
A set of processes and their connections, which can receive and send data via
ports. A process group allows creation of entirely new component simply by
composition of its components.

FlowFiles are like HTTP data
HTTP Data FlowFile
HTTP/1.1 200 OK
Date: Sun, 10 Oct 2010 23:26:07 GMT
Server: Apache/2.2.8 (CentOS) OpenSSL/0.9.8g
Last-Modified: Sun, 26 Sep 2010 22:04:35 GMT
ETag: "45b6-834-49130cc1182c0"
Accept-Ranges: bytes
Content-Length: 13
Connection: close
Content-Type: text/html
Hello world!
Standard FlowFile Attributes
Key: 'entryDate’ Value: 'Fri Jun 17 17:15:04 EDT 2016'
Key: 'lineageStartDate’ Value: 'Fri Jun 17 17:15:04 EDT 2016'
Key: 'fileSize’ Value: '23609'
FlowFile Attribute Map Content
Key: 'filename’ Value: '15650246997242'
Key: 'path’ Value: './’
Binary Content *
Header
Content

FlowFiles & Data Agnosticism
 NiFi is data agnostic!
 But, NiFi was designed understanding that users
can care about specifics and provides tooling
to interact with specific formats, protocols, etc.
ISO 8601 - http://xkcd.com/1179/
Robustness principle
Be conservative in what you do,
be liberal in what you accept from others“

Apache MiNiFi
 NiFi lives in the data center. Give it an
enterprise server or a cluster of them.
 MiNiFi lives as close to where data is born
and is a guest on that device or system
“Let me get the key parts of NiFi close to where data begins and provide bidirectional
data transfer"

Apache MiNiFi
 Limited computing capability
 Limited power/network
 Restricted software library/platform availability
 No UI
 Physically inaccessible
 Not frequently updated
 Competing standards/protocols
 Scalability
 Privacy & Security
Realities of computing outside the comforts of the data center

MiNiFi: Precedent from NiFi
 Provides the semantics between two NiFi components across network boundaries
– A custom protocol for inter-NiFi communication
– Secure, Extensible, Load Balanced & Scalable Delivery to Cluster
 Extracted out to a client library which powers integration into popular frameworks like
Apache Spark, Apache Storm, Apache Flink, and Apache Apex
 Attributes and the FlowFile format maintained
A quick look at NiFi Site to Site
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#site-to-site

 Fine-grained, event level access of interactions with FlowFiles
– CREATE, RECEIVE, FETCH, SEND, DOWNLOAD, DROP, EXPIRE, FORK, JOIN …
 Captures the associated attributes/metadata at the time of the event
 A map of a FlowFile’s journey and how they relate to other FlowFiles in a system
– MiNiFi enables us to get more and further illuminate the map of data processing
A deeper dive into provenance
http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance

RECEIVE event

Apache MiNiFi
 The feedback loop is longer and not
guaranteed
– Removal of Web Server and UI
 Declarative configuration
– Lends itself well to CM processes
– Extensible interface to support varying formats
• Currently provided in YAML
 Reduced set of bundled components
Departures from NiFi in getting the right fit

Apache MiNiFi: Scoping
 Go small: Java – Write once, run anywhere*
– Feature parity and reuse of core NiFi libraries
 Go smaller: C++ – Write once**, run anywhere
 Go smallest: Write n-many times, run anywhere
Language libraries to support tagging, FlowFile format, Site to Site protocol, and
provenance generation without a processing framework
– Mobile: Android & iOS
– Language SDKs
Provide all the key principles of NiFi in varying, smaller footprints

WHAT IS THIS!?
A NiFi FOR ANTS!?!

MiNiFi: Use Case - Connected Car
 Outside vehicle’s
network firewall
 On telematics layer
VEHICLE NETWORK FIREWALL
TRANSMIT
EXECUTE FILTER
PRIORITIZE
PARSE
LISTEN
ROUTE

Connecting the Drops
SOURCES
REGIONAL
INFRASTRUCTURE
CORE
INFRASTRUCTURE

Use Case: Courier Service with Apache NiFi & MiNiFi
Physical Store
Gateway
Server
Mobile Devices
Registers
Server Cluster
Distribution Center
Kafka
Core Data Center at HQ
Server Cluster
Others
Storm / Spark /
Flink / Apex
Kafka
Storm / Spark / Flink / Apex
On Delivery Routes
Trucks Deliverers
Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/
Deliverer: Rigo Peter, https://thenounproject.com/rigo/
Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/
Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/
Client
Libraries
Client
Libraries
MiNiFi
MiNiFi
NiFi NiFi NiFi NiFi NiFi NiFi
Client
Libraries
Gathering data from disparate sources

23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved© Hortonworks Inc. 2011 – 2017. All Rights ReservedX
Data Provenance
▪ Constrained
▪ High-latency
▪ Localized context
▪ Hybrid – cloud/on-premises
▪ Low-latency
▪ Global context
Origin – attribution
Replay – recovery
Evolution of topologies
Long retention
Types of Lineage
• Event
• Configuration

 Site-to-Site in MiNiFi instances provides machine-to-machine (M2M) communication
– Data arrives to NiFi in a transparent manner allowing integration to existing flows
 Similar attention to extensibility in both Java and C++ clients allows agents to fit the
needs of your organization
 Reduced footprint allows NiFi functionality to aid in production of high fidelity data,
more closely attributable and tracked from where it is generated
We’ve provided a framework to extend the reach of data ingest

 Enter the MiNiFi Command & Control
– Provide tooling to map the UX of
interactive command and control in NiFi
to the design and deploy approach of
MiNiFi
But more instances complicate my operational management!
https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control

 Configuration Management of Flows & Versioning
– The evolution of templates to better support SDLC functions
– https://cwiki.apache.org/confluence/display/NIFI/Configuration+Management+of+Flows
 Extension Repositories
– Publish & Share extension bundles (NARs)
– https://cwiki.apache.org/confluence/display/NIFI/Extension+Repositories+%28aka+Extension+Regis
try%29+for+Dynamically-loaded+Extensions
 Variable Registry
– Initial framework support & file-based implementation
– https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry
Building on efforts for reusable components in the community

Why Apache NiFi & MiNiFi?
 Moving data is multifaceted in its challenges and these are present in different contexts
at varying scopes
– Think of our courier example and organizations like it: inter vs intra, domestically, internationally
 Provide common tooling and extensions that are commonly needed but be flexible for
extension
– Leverage existing libraries and expansive Java ecosystem for functionality
– Allow organizations to integrate with their existing infrastructure
 Empower folks managing your infrastructure to make changes and reason about issues
that are occurring
– Data Provenance to show context and data’s journey
– User Interface/Experience a key component

Learn more and join us!
Apache NiFi site
https://nifi.apache.org
Subproject MiNiFi site
https://nifi.apache.org/minifi/
Subscribe to and collaborate at
dev@nifi.apache.org
users@nifi.apache.org
Submit Ideas or Issues
https://issues.apache.org/jira/browse/NIFI
https://issues.apache.org/jira/browse/MINIFI
Follow us on Twitter
@apachenifi

Apache NiFi Crash Course
Thursday, 6 April
11:15 AM – 1:45PM, Room 12
• Learn more about NiFi, the community, and work through a hands-on lab
• Seats available on a first come, first served basis
• Make sure you are in possession of the latest version of VirtualBox
• More details: https://tinyurl.com/nifi-cc-munich17

Learn, Share at Birds of a Feather
IOT, STREAMING & DATA FLOW
Thursday, April 6
5:50 pm, Room 5

Thank You

Data at Scales and the Values of Starting Small with Apache NiFi & MiNiFi

More Related Content

What's hot (20)

Similar to Data at Scales and the Values of Starting Small with Apache NiFi & MiNiFi (20)

Recently uploaded (20)

Data at Scales and the Values of Starting Small with Apache NiFi & MiNiFi