Streaming Analytics for IoT with Apache Spark
4 likes980 views
The webinar discussed streaming analytics for IoT using Apache Spark, focusing on its market potential and application architecture. Key topics included real-time data ingestion, device management, and use cases in connected cars and industrial IoT. It emphasized the need for a vendor-neutral, interoperable architecture to efficiently operationalize IoT use cases.
Streaming Analytics for IoT with Apache Spark
- 1. Streaming Analytics for IoT with Apache Spark WEBINAR | FEBRUARY 16, 2018 Sameer BhideAnand Venugopal AVP & Business Head, STREAMANALYTIX Senior Solution Architect, STREAMANALYTIX
- 2. Agenda | IoT Market Perspective | The IoT Application Architecture | Use Cases: Connected Car | Industrial IoT | Demo ā Building IoT Spark Apps Using | Key Takeaways | Q & A
- 3. Mission critical technology solutions since 1996 Fortune 500: Big data clients 1700 people; US, India, global reach Unique mix of big data products and services ABOUT
- 4. IOT Market: Gartner Prediction 7.6 Billion Things that will ship in 2021 32% CAGR End point growth rate 2016-2021 25.1 Billion Units install base in 2021 $3.9 Trillion Total spending: 2021 end points and services
- 5. Smart Home Smart Car Smart Building Smart City Smart Agriculture Smart Factory Smart Healthcare Smart Data Center Smart Energy Smart Retail IoT ā Market Sub-Domains
- 6. IoT Solution Architecture and the Role of Spark Field IoT Gateway Cloud IoT Integration HUB Field IoT Gateway Connected Things Enterprise Applications Centralized IoT Data Mgmt. & Analytics Platform
- 7. POLL
- 8. ā¢Variation in Device Capabilities ā¢ MEP: State-notifications, Commands, Telemetry ā¢ State Management & Security ā¢Agility ā¢Extensibility ā¢Time-to-market ā¢Automation ā¢ RT, NRT Analytics ā¢ Offline Analytics & Reporting ā¢ Schematization ā¢ Data Blending & Enrichment ā¢ Multi-party Interaction ā¢Network & Connectivity Dynamics ā¢Real Time Data Ingestion Telemetry Computation & Analytics Device Management Operationalize IOT Application Architecture: Key Aspects
- 9. TELEMETRY ā¢ Network & Connectivity Dynamics ā¢ Real Time Data Ingestion Data Ingestion Abstractions COMPUTATION & ANALYTICS ā¢ RT, NRT Analytics ā¢ Offline Analytics & Reporting ā¢ Schematization ā¢ Data Blending & Enrichment ā¢ Multi-party Interaction Event & Micro-Batch Computation Engines (Spark, Storm etc.) DEVICE MANAGEMENT ā¢ Variation in Device Capabilities ā¢ MEP - State-notifications, Commands, Telemetry ā¢ State Management & Security Metadata Registry & Management Gateway OPERATIONALIZATION ā¢ Agility ā¢ Extensibility ā¢ Time-to-market ā¢ Automation Dataflow (Pipeline) Management & DSL Capabilities 1 2 3 4 IOT Application Architecture: Foundational Elements
- 10. Managed Devices EDA / SEDA Sources IOT Application Architecture: Conceptual Layers Management LayerIoT Gateway Data Ingestion Data Processing & Storage Layer Insights Layer Action Layer Security Data Sources Compute Engine (Spark) ML - Model Updates Patternsā (A-B, Champion Challenger etcā¦) Notification Services Fault Tolerance Protocol Support Data Filtering, Blending & Enrichment Rule Engine Alerts State Management Ontology & Metadata Management Structured Query Feedback Loop External Services Integration Device Proxy Data Persistence Custom Business Flows (IFTTT, Lambda etc) INGEST ENRICH ANALYZE ACT Configuration & Connection Management Performance Management Application Life Cycle Management Version Updates PaaS Integration Computer Infrastructure SPARK Infrastructure Layer
- 11. Spark as the IoT Compute Engine | Massively scalable | Rich set of transformations | Industry adoption | Unified & simplified programming model | Support for machine learning | Micro-batch capable ā tending to NRT
- 12. Recommendations | Adopt an integrated approach to IoT development | Design a platform layer that can adopt to businessā dynamic needs | Create a vendor neutral & interoperable architecture | Adopt software products to quickly operationalize IoT use cases
- 13. Ā© 2017 Impetus Technologies Real-time Stream Processing & Machine Learning Platform + Visual Spark Studio
- 14. Ingest Analyze Define Rules & Actions Design Once ā Execute Anywhere Manage Key Features
- 15. Connected Cars Industrial IoT Use Cases
- 16. Connected Car ā Driver Risk Profiling Brief Background Leading insurance provider in the US ā¢ Classify drivers based on current driving pattern and historical data ā¢ Raise alerts on behavior change ā¢ Blend data from syndicated and open / public data marts & services ā¢ Derive additional analytics through supplemental data flows Business Need To create an end-to-end analytics application for driver profiling & RT risk assessment
- 17. Central Aggregation Server / Data Flow Manager On-premise: Bare-Metal and/or VMs | Public / Hybrid Cloud Data Center / Cloud Storage and Offline Analytics Device Provisioning and Management (identity / registration etc.) Open Interfaces ā extensibility and customizability in all directions Real Time Dash-boarding Condition Monitoring Predictive Maintenance Smart Alerting Root Cause Analytics Closed-loop Feedback Edge Custom solution OR 3rd party IoT interface vendor Data flow Control flow End Device 1 End Device 2 End Device 3 End Device 1 End Device 2 End Device 3 Smart Car 1 Smart Car 2 IoT / Connected Car Solution with StreamAnalytix IOT data interface (MQTT / HTTP / WebSockets) IOT data interface (MQTT / HTTP / WebSockets) Gateway
- 18. AWS IoT (Spark) Ingestion Enrichment Analytics Public Cloud Services / Third-party Services Dashboards Handheld Devices Persistence Automated Device Installed in OBD-II Port High Level Solution Overview Alerts
- 19. Connected Car ā Driver Risk Profiling
- 20. Connected Car ā Driver Risk Profiling Ingest events using AWS IoT gateway Mask PII & enrich data with external & historical sources Score a āRisk Assessmentā model that uses ā¢ Weather conditions ā¢ Time of trip ā¢ Hard brakes and acceleration ā¢ Duration over 70mph ā¢ Previous number of risk instances Raise alerts based on risk scores Create RT and historical dashboards
- 21. Industrial IoT Use Case ā Device Health Monitoring Brief Background Leader in industrial automation, information, and engineering services ā¢ Various machine health parameters collected in different timelines from an array of sensors ā¢ Compute and store correlation between sensor data when a process parameter is altered ā¢ Leverage existing investments in Azure cloud infrastructure Business Need Measure impact to process dynamics by calculating correlation between various sensor data
- 22. End-to-end solution deployed on StreamAnalytix Data pipelines used pre-built components: ā¢ Data ingestion ā¢ Statistical functions ā¢ Data enrichment ā¢ Visualization Cloud: Microsoft Azure Source: Event Hub Compute: Spark jobs on HDInsight Orchestration: StreamAnalytix High Level Solution Approach Reporting Dashboard Manufacturing Units HD-Insights
- 23. ā¢ Ingest data from different MS Azure Event Hub sources ā¢ Enrich incoming data ā¢ Outer-join on incoming datasets ā¢ Aggregate result data and group by plantIDs ā¢ Post streaming results on WebSockets Data Pipeline View Industrial Automation - Turbine Data Analytics
- 24. Key Takeaways | IoT capabilities in StreamAnalytix ā¢ Data Sources : Azure Event Hub, AWS IoT, MQTT, Kinesis, S3 ā¢ Data Sink : Redshift, Hadoop, MQTT, S3, Kinesis, WebSockets ā¢ PaaS Service Integration : SQS, Lambda, SNS | Integrated approach to IoT development | IoT applications are dynamic | Vendor neutral & interoperable architecture | COTS & open source offerings to quickly operationalize IoT use cases
- 26. Thank you. Questions? Ā© 2018 Impetus Technologies Email: [email protected] | @ImpetusTech / @StreamAnalytix Download Visual Spark Studioā¢ - Zero cost, light weight development tool. Full range of data processing and analytics functionality to build Spark applications.