Spark Streaming and IoT by Mike Freedman
13 likes•4,839 views
This document discusses using Spark Streaming for IoT applications and the challenges involved. It notes that while Spark simplifies programming across different processing intervals from batch to stream, programming models alone are not sufficient as IoT data streams can have varying rates and delays. It proposes a unified data infrastructure with abstractions like data series that support joining real-time and historical data while handling delays transparently. It also suggests approaches for Spark Streaming to better support processing many independent low-volume IoT streams concurrently and improving resource utilization for such applications. Finally, it introduces the Device-Model-Infra framework for addressing these IoT analytics challenges through combined programming models and data abstractions.
Spark Streaming and IoT by Mike Freedman
- 1. Spark Streaming and IoT Michael J. Freedman iobeam
- 2. Technology confluence in IoT UBIQUITOUS SENSORS REAL-TIME SYSTEMS MACHINE LEARNING DATA ANALYSIS INTERSECTION OF 3 MAJOR TRENDS
- 3. Data analysis is the killer app CASE STUDY: PREDICTIVE MAINTENANCE Predicting motor failure through analysis of vibration data CASE STUDY: HEALTH & FITNESS Exercise identification based on 3D motion data analysis CASE STUDY: SMART CITIES Traffic and air quality monitoring via GPS and environmental sensor CASE STUDY: SMART GRID Demand-response optimizations on supply-side capacity, spot prices
- 4. Challenges in applying Spark to IoT REQUIREMENTS 2 Devices send data at varying delays and rates 2 Handling delayed data transparently 3 Processing many low-volume, independent streams 1 One IoT app performs tasks at different time intervals 1 Supporting full spectrum of batch to real-time analysis 3 Within org, multiple IoT apps run concurrently 4 Multi-tenancy with low-volume apps and high utilization CHALLENGES Potential economic impact of IoT is >$11 trillion per year, even while 99% of IoT data goes unused today. — 2015 McKinsey study
- 5. Required: Programming + data infra abstractions
- 6. Supporting full spectrum of batch to real-time analysis 1
- 7. IoT analysis spans many intervals BATCH PROCESSING (HOURS, NIGHTLY) STREAM PROCESSING (REAL-TIME) Fire / Hazard Detection Immediately Bus Location Updates 15 sec Traffi c Conditions 1 min Environmental Conditions 15 min Traffi c Optimizations Daily
- 8. Spark simplifies programming across intervals val readings = iobeamInterface.getInputStreamRecords() // Trigger temperatures that fall outside acceptable conditions val bad_temps = readings.filter(t => t > highTempThreshold || t < lowTempThreshold) val triggers = new TriggerEventStream(bad_temps.map(t => new TriggerEvent("bad_temperature", t))) // Compute mean temperatures over 5 min windows val windows = readings.groupByKeyAndWindow(Seconds( 300 ), Seconds(60)) val mean_temps = new TimeSeriesStream("mean_temperature", windows.map(t => t.sum / t.length)) new OutputStreams(mean_temps, triggers) 30 1800
- 9. But programming != data abstractions DATA STREAMS (KAFKA, FLUME, SOCKETS, ETC.) DATA FILES (HDFS, ETC.) BATCH PROCESSING (HOURS, NIGHTLY) STREAM PROCESSING (REAL-TIME)
- 10. Programming != data abstractions Traffi c Conditions 30 sec Traffi c Conditions 1 hour Frequencies change as products evolve1
- 11. Programming != data abstractions Joining real-time with historical data2 Frequencies change as products evolve1 5 min mean vs. trailing - hourly mean - hourly mean from yesterday - hourly mean from last week
- 12. Programming != data abstractions Joining real-time with historical data2 Supporting backfill for delayed data3 Frequencies change as products evolve1
- 13. Programming != data abstractions Joining real-time with historical data2 Supporting backfill for delayed data3 Frequencies change as products evolve1 Data Series Abstraction
- 14. Handling delayed data transparently2
- 15. Windows in streaming DBs Tumbling windows titjtk
- 16. Windows in streaming DBs Sliding windows • Defined over # of tuples • Defined over time period …using arrival_time of tuples titjtk
- 17. But IoT data is often delayed Seconds due to network congestion Minutes due to duty cycling for energy savings Minutes to hours due to intermittent connectivity Windowing data by arrival time has no semantic meaning titjtk
- 18. Wanted: Data generation time, not arrival time titjtk filter by timestamp Data semantics defined over timestamp e.g., aggregation JOIN ( , historical data)
- 19. Wanted: Backfill does not change semantics titjtk filter by timestamp Data semantics defined over timestamp e.g., aggregation JOIN ( , historical data) …from recent streaming data… …from historical archive…
- 20. Wanted: Better data infra abstractions titjtk filter by timestamp e.g., aggregation JOIN ( , historical data) Data Series Abstraction …from recent streaming data… …from historical archive…
- 21. Processing many low-volume, independent streams 3 IoT Device Streams
- 22. Wanted: Maintain state across batches Map to good/bad conditions titjtk Alert on condition transition
- 23. Spark: Share state through RDDs Click streams Ad impressions Market feeds Shared state b/w stream partitions ‣ Transforms RDD, makes state available across cluster ‣ Many great uses, e.g., learning parameters in iterative ML ‣ But increases data lineage increases checkpointing cost Maintain shared state via updateKeyByState()
- 24. IoT: Many independent streams ‣ Each worker handles 1+ streams, not multiple workers per stream ‣ Use language data structures (e.g., Java Map) to maintain state within worker ‣ No RDD transform no lineage increase no increased checkpointing cost Independent state per stream IoT Device Streams Often only need to maintain state within individual streams
- 25. Multi-tenancy with low-volume apps and high utilization 4
- 26. Multi-tenancy for batch processing Job Queue Server Cores Spark: 1 worker = 1 server core Goal: Minimize time-to-completion
- 27. Multi-tenancy for batch processing Job Queue Server Cores Spark: 1 worker = 1 server core Goal: Minimize time-to-completion
- 28. Multi-tenancy for stream processing Job Queue Server Cores Spark: 1 worker = 1 server core Problem: Low utilization with low-rate apps
- 29. Multi-tenancy for stream processing Job Queue Server Cores Virtual Cores (e.g., resource-limited containers) 1 worker = 1 virtual core N workers = 1 server core Goal: Improve utilization with low-rate apps
- 30. Multi-tenancy for stream processing Job Queue Server Cores Virtual Cores (e.g., resource-limited containers) 1 worker = 1 virtual core N workers = 1 server core Goal: Improve utilization with low-rate apps
- 31. Spark + Unified Data Infrastructure
- 32. Required: Programming + data infra abstractions
- 33. Required: Programming + data infra abstractions
- 34. Device-Model-Infra (DMI) framework for IoT