Devising your Data Movement Strategy for IoT
Download as PPTX, PDF0 likes292 views
The document outlines strategies for data movement in the Internet of Things (IoT), detailing architectures, messaging patterns, protocols, and tackling scalability challenges. Key topics include enterprise bus connection tiers, message exchange patterns, security, and high throughput requirements for applications, particularly in smart city implementations like Singapore's ERP system. It emphasizes the importance of a flexible, robust architecture that can adapt to the evolving demands of IoT infrastructures.
Devising your Data Movement Strategy for IoT
- 1. 1 Copyright Solace Confidential Devising your Data Movement Strategy for IoT Dave McAllister September, 2016
- 2. 2 Copyright Solace Confidential Data Movement Between Elements of IoT Top level system considerations Architectures for different types of data movement Which standards to use; where and why
- 3. 3 Copyright Solace Confidential Relevant System Requirements Scale of Devices Capabilities of Devices Message Exchange Patterns Messaging QoS Traffic Characteristics Security IT Application Architecture & Technologies
- 9. 9 Copyright Solace Confidential Firewall Connection Tier Load Balancer EDGE FOG Typical IoT Cloud Deployment Architecture Enterprise Bus Firewall CORE
- 10. 10 Copyright Solace Confidential Firewall Connection Tier Load Balancer EDGE FOG System Considerations • Connections, queues, subscription • MEPs • Device initiated pub/sub • Device initiated request/reply • Cloud app initiated pub/sub • Cloud app initiated request/reply • QoS – MQTT QoS0, QoS1 • Single point of contact for apps Enterprise Bus Firewall CORE
- 11. 11 Copyright Solace Confidential Enterprise Bus Connection Tier System Considerations Load Balancer o Connections, queues, subscrptns o MEPs ‐ Device-initiated pub/sub ‐ Device-initiated request/reply ‐ Cloud app-initiated pub/sub ‐ Cloud app-initiated request/reply o QoS – MQTT QoS0, QoS1 o Single point of contact for apps
- 12. 12 Copyright Solace Confidential Enterprise Bus Connection Tier Load Balancer System Considerations o Connections, queues, topics o MEPs ‐ Device-initiated pub/sub ‐ Device-initiated request/reply ‐ Cloud app-initiated pub/sub ‐ Cloud app-initiated request/reply o QoS – MQTT QoS0, QoS1 o Single point of contact for apps
- 13. 13 Copyright Solace Confidential Enterprise Bus Connection Tier Load Balancer Challenges with Stateless Connect Layer o Device Queues ‐ Where is the queue for each device for QoS1 subscriptions? ‐ How does Connection layer find its queue/state? (among the millions….) o App to device MEP ‐ How does Core layer find Connection layer server for given MQTT device for QoS0? ‐ Where is the Q for a given device for QoS1? ‐ Need massive subscription scalability updated in real time
- 14. 15 Copyright Solace Confidential MQTT Paho Proprietary Proprietary/ JMS/JCA HTTP/REST In Runtime AMQP 1.0 Qpid / JMS CoAP Several OSS P R O T O C O L A P I Relevant Protocols & APIs
- 15. 16 Copyright Solace Confidential MQTT versus CoAP MQTT CoAP Many to many (but Pub-sub) One to one Event based State based No discovery (Pub-sub) Supports content negotiation OASIS / ISO IETF 3 levels QoS 2 levels QoS
- 16. 17 confidential o Device Constraints o Message Exchange Patterns o Messaging QoS o Traffic Characteristics o Security Fog
- 17. 18 confidential Fog MQTT Paho Proprietary Proprietary/ JMS/JCA HTTP/REST In Runtime AMQP 1.0 Qpid / JMS CoAP Several OSS P R O T O C O L A P I
- 18. 19 Copyright Solace Confidential Edge o # Concurrent Connections o “Always on” Connections? o Burstiness o Congestion / Prioritization o Multi-protocol? o QoS o Security
- 19. 20 Copyright Solace Confidential MQTT Paho Proprietary Proprietary/ JMS/JCA HTTP/REST In Runtime AMQP 1.0 Qpid / JMS CoAP Several OSS P R O T O C O L A P I Edge
- 20. 21 Copyright Solace Confidential o Less scale, more concentration o Need enterprise features o Enterprise grade, popular APIs o Support streaming & microservices Core
- 21. 22 Copyright Solace Confidential Core MQTT Paho Proprietary Proprietary/ JMS/JCA HTTP/REST In Runtime AMQP 1.0 Qpid / JMS CoAP Several OSS P R O T O C O L A P I
- 22. 23 Copyright Solace Confidential IoT is all about the data Capacity & Availability Limits Big Data RIVER Shock Absorber Network Processing Storage Outages Upgrades Inconsistent Aggregate Input Stream
- 23. 24 Copyright Solace Confidential Public Clouds Public CloudAp p Ap p Ap p Big Data River Big Data Private Cloud App App App App App App PaaS IaaS App App App Public Cloud On Premise
- 24. 25 Copyright Solace Confidential Enterprise Bus Connection Tier Addressing Scalability o Command & Control sending message to a specific device
- 25. 26 Copyright Solace Confidential Enterprise Bus Connection Tier Addressing Scalability o Command & Control sending message to a specific device o Device-initiated request/reply o Among millions of devices o Across 2 middleware tiers o Need Integrated addressing between Enterprise Bus & Connection Tier at massive scale
- 26. 27 Copyright Solace Confidential Enterprise Bus Connection Tier Addressing Scalability Load Balancer
- 27. 28 Copyright Solace Confidential Enterprise Bus Connection Tier Addressing Scalability o Connection Tier Addressing ‐ MQTT Topics o Enterprise Bus Addressing ‐ Other topics & queues o Need to scale ‐ In an integrated manner ‐ Update in real-time ‐ Handling bursts of changes
- 28. 29 Copyright Solace Confidential Singapore Land Transport Authority – ERP2 http://www.straitstimes.com/singapore/transport/ncs-mhi-to-build-islandwide-satellite-based-erp-for-556m Routing Server Payments Analytics 0 0 0 • 200K Concurrent Connections • MQTT Termination • Request/reply for firmware upgrades • Push Notifications • Persistent Messaging • Shock Absorber • High Throughput • Fanout • Slow Consumer Handling MQTT JMS MHI OBU Next-Gen Electronic Road Pricing o Connect 1.5M vehicles (200K at a time) in real-time for pay per use road tolls o Suggest better routes, traffic management – real time and long term and value-added services/analytics
- 29. 30 Copyright Solace Confidential Bus Monitoring o Collect real-time location, loading information and more thousands of buses o Optimize bus movement, fleet management, avoid bunching, forecast arrival time and more REST/MQTT Regulator’s Dashboard Commuter’s Apps and RIAs Analytics DWH ESB (if needed) CEP JMS Legacy REST Legacy Gateway REST/MQTT
- 30. 31 Copyright Solace Confidential Overall Architecture Requirements o Scalability: connections, queues, subscriptions o “single bus” connectivity for both apps & devices – solve the whole problem! ‐ Unified addressing between cloud apps & devices ‐ Single point of application connectivity o Support necessary QoS o Support all MEPs(?) ‐ Device-initiated pub/sub ‐ Device-initiated request/reply ‐ Cloud app-initiated pub/sub ‐ Cloud app-initiated request/reply IoT Connectivity Concerns
- 31. 32 Copyright Solace Confidential At the Edge o Scalability: connections, queues, subscriptions o Protocols: MQTT, REST, CoAP(?) o Security: ‐ Authentication & Encryption ‐ Per-device authorization ‐ deviceID in topics ‐ /in/<devID>/<system> ACL on /in/<devID>/> In the Core o Single, simple point of contact to talk to any device o APIs & protocols to cloud apps ‐ JMS, AMQP, Node.js, REST ‐ App-to-device and app-to-app o Message filtering o Application horizontal scaling with non-exclusive queues o Integration with Hadoop IoT Connectivity Concerns(cont)
- 32. 33 Copyright Solace Confidential Can’t predict coming evolution of IoT, and changing infrastructure isn’t cheap. Need flexible data movement platform
- 33. 34 Copyright Solace Confidential Strong Architecture • Decoupling of concerns • Scales to meet needs of clients • Right features at each layer Smart Use of Standards • Right standards • Features & exchange patterns • Optimized for needs of users • Security • Supported by key products and open source projects
- 34. 35 Copyright Solace Confidential Thanks! Dwmcallister [email protected] Download the free VMR Community Edition at http://dev.solace.com
Editor's Notes
- #6: - Producer / Consumer
- #11: QoS 0 at most once QoS 1 at least once QoS 2 exactly once
- #15: Talk about Data River – Messaging and Networking Talk about Data Lake.
- #16: JMS is an API specification (part of the Java EE specification) that defines how message producers and consumers are implemented. JMS does not guarantee interoperability between implementations, and the JMS-compliant messaging system in use may need to be deployed on both client and server. On the other hand, AMQP is a wire-level protocol specification. In theory AMQP provides interoperability as different AMQP-compliant software can be deployed on the client and server sides. Note that, like HTTP and XMPP, AMQP does not have a standard API. MQTT (Message Queue Telemetry Transport) Open : created by IBM & Eurotech and donated to Eclipse “Paho” M2M project (OASIS standard in 2014) Lightweight : smallest packet size 2 bytes (header), reduced clients footprint (C# M2Mqtt library 30 KB) Reliable : three QoS and patterns to avoid packet loss on client disconnection Simple : TCP based Asynchronous Publish/Subscribe Few verbs Payload agnostic