Make Streaming IoT Analytics Work for You
Download as PPTX, PDF11 likes2,272 views
1) Streaming analytics platforms for IoT need to focus on ingesting data from various sources, processing data in real-time, analyzing data, responding to events, and visualizing data. 2) Key areas for building such a platform include using a common abstraction layer, minimizing latency, integrating static and real-time data using lambda architecture, scaling out linearly, enabling rapid application development, and providing data visualization. 3) An example use case of a connected car generates large amounts of data that can be used for various purposes through a streaming analytics platform like predictive maintenance and customized experiences.
Make Streaming IoT Analytics Work for You
- 1. Make Streaming IoT Analytics Work for You Kanishk Mahajan & Dhruv Kumar Hortonworks May 2016
- 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved IoT - Relevance 7.2 Billion active SIM Cards 25 Billion connected things 500 Million tweets per day – Average life of a tweet is 18 minutes 2 Zettabytes per year of Global IP Traffic – 80% is unstructured
- 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved The Four Vs of Big Data http://www.ibmbigdatahub.com/infographic/four-vs-big-data
- 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved 5 Attributes of a Streaming Platform Ingest Process Analyze Respond Visualize
- 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Data Flow versus Stream Analytics for IoT Data Flow – Ingest and route terabytes data into a ”unified firehose” – Actively performance manage the latency and quality of these data flows – • Across high variability of data formats, size of data and speed of data Real Time Stream Analytics – Sub second event processing with linear scalability to billions of events – Predictive Analytics at Scale • Real time data aggregation across edge nodes while processing 10s of millions of events and 100s of gigabytes per second – Guaranteed no data loss and events processed in order
- 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved 6 Key Focus Areas for Building a Streaming Platform for IoT 1. Common Abstraction Layer 2. Latency 3. Lambda Architecture 1. “Orchestrate” over static and real time data 4. Scale-out 5. Rapid Application Development 6. Data Visualization
- 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Focus Areas for Streaming Platform for IoT 1. Common Abstraction Layer – Select one or more streaming engines – Select one or more cloud providers – Select one or more resource managers – Select one or more event sources – …Future Proof
- 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Focus Areas for Streaming Platform for IoT 2. Latency – 500 millisecond or less- Dashboards, Security Incidents, Asset Performance – 20 milliseconds or less - Ad Networks, Preventive Maintenance – …
- 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Focus Areas for Streaming Platform for IoT 3. Lambda Architecture – Integrate static and real time data – Enrichment – Orchestration of Batch workflows – Predictive Classification and Scoring
- 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Focus Areas for Streaming Platform for IoT 4. Scale Out – Linear Scale out or scale down – Resource Management – Handle Transient workloads
- 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Focus Areas for Streaming Platform for IoT 5. Rapid Application Development – Industry vertical specific applications – Aggregations – Filters – Multi Stream Correlations – Splits – Joins – Normalizations – Business Rules Editor
- 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Focus Areas for Streaming Platform for IoT 6. Data Visualization – Time Series Visualizations – Metrics Dashboarding – Trends – Comparisons – Thresholds – Custom UI extensions
- 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved IoT - Real Time Stream Analytics versus Streaming Engine Abstracts underlying Streaming Engine- Storm, Spark Streaming, Flink.. OOTB Support for multiple (cloud) event sources - Kafka, AWS Kinesis, Azure Event Hub Built-in Operators for Complex Event Processing Built-in Real Time Dashboarding- Metrics and Events PMML Support Pluggable Workflow Management Business Rules Editor and Rapid Application Development Framework Cloud Deployment Scalable Architecture Handles different latency requirements
- 14. IoT: The Big Picture
- 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved © Hortonworks Inc. 2011 Hortonworks IoT Platform – The Big Picture Page 15 Data Acquisition Edge Processing Real Time Stream Analytics IoT Services Rapid Application Development IoT ANALYTICS CLOUD
- 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved IoT Edge Data Acquisition & Processing Device Specific Protocol Device Data Acquisition Edge Processing IoT Analytics CloudHTTP(s) Edge Processing • Reliable Delivery • Buffering and Flow Control • Simple Event Processing • Edge Analytics High Availability • Scale out and Clustering • Multi Data Center Support Security • 2 way SSL • Data Element Masking • Flexible Routing based on Data Sensitivity Kafka Enablement • Data Ingest and Drain based on Kafka Consumer and Producer Support
- 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Device Management Agent • Send Diagnostics • Receive F/W updates • Connectivity Logs Enrollment & Authentication • 3-legged OAuth based flow • Encryption • Confidentiality Registry • Device Metadata Catalog User Linkage • Device - User Linkage • May be many - many between Device and User • Temporary or Persistent De- Authorization • Deregister • Unlink User • Disable
- 18. Use Cases - Connected Car
- 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Connected Car - Generator of Big Data: http://www.slideshare.net/AditiTechnologies/how-internet-of-things-iot-is-reshaping-the-automotive-sector-infographic
- 20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Connected Car - Use Cases http://gelookahead.economist.com/infograph/car-os/
- 21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Reference Architeucture: Connected Car Platform and Hortonworks For more details contact [email protected] Hortonworks Data Platform Core Data Engineers (CAD) Systems of Record (ERP) Customers (CRM)
- 22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache NiFi Demo: https://www.youtube.com/watch?v=r4NsWbE4_-I (Download here: http://hortonworks.com/products/hdf/
- 23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Questions? http://community.hortonworks.com