In search of the perfect IoT Stack - Scalable IoT Architectures with MQTT
2 likes696 views
The document discusses the complexities of building scalable IoT architectures, emphasizing the role of MQTT as a centralized communication protocol. It highlights various architectural components like load balancers, MQTT brokers, and data stores, as well as the importance of resilience and high availability in system design. Additionally, it provides insights on deployment options, processing methods, and key technologies for managing large volumes of data in IoT environments.
In search of the perfect IoT Stack - Scalable IoT Architectures with MQTT
- 1. In search for the perfect IoT Stack Scalable IoT Architectures with MQTT Dominik Obermaier | dc-square GmbH
- 2. Dominik Obermaier CTO @ HiveMQ Welcome! @dobermai
- 4. What makes an IoT Stack special?
- 5. What is different Small data “packets“,but a lot of them Large amount of concurrently connected devices Machine communication without human interaction High grade of automation Continuous flow of data
- 6. Dimensions of data GB/hour GB/day GB/month GB/year 1001 msg / minute each 1kB in size 0.006 Mbps 0.14 4 53 0.01 number of devices
- 7. Dimensions of data GB/hour GB/day GB/month GB/year 1001 msg / minute each 1kB in size 0.006 Mbps 0.14 4 53 0.01 number of devices 1,000 10,000 100,000 0.06 0.6 6 1.4 43 526 0.1 14 432 5,256 1.3 144 4,320 52,560 13 1,000,000 60 1,440 43,200 525,600 133
- 8. Dimensions of data GB/hour GB/day GB/month GB/year 100 1,000 10,000 100,0001 msg / second each 1kB in size 0.36 3.6 36 360 Mbps 8.64 259 3,154 0.8 86.4 2,592 31,536 8 864 25,920 315,360 80 8,640 259,200 3,153,600 800 1,000,000 3,600 86,400 2,592,000 31,536,000 8,000 number of devices
- 9. 1,000,000 msg/s: 2.5 PB/month whole Internet Traffic in 1996: 1.9 PB/month Source: https://en.wikipedia.org/wiki/Internet_traffic Putting this into perspective 100,000 msg/s: 8.6 TB/day 1 server hard drive per day
- 10. Architecture - Lots of choices
- 11. Microsoft IoT Reference Architecture Source: https://azure.microsoft.com/de-de/updates/microsoft-azure-iot-reference-architecture-available/
- 12. Intel IoT Reference Architecture Source: http://www.intel.de/content/www/de/de/internet-of-things/white-papers/iot-platform-reference-architecture-
- 13. WSO2 IoT Reference Architecture Source: http://wso2.com/wso2_resources/wso2_whitepaper_a-reference-architecture-for-the-internet-of-things.pdf
- 14. Industrial Internet Consortium IIoT Architecture Source: https://www.iiconsortium.org/IIC_PUB_G1_V1.80_2017-01-31.pdf
- 16. Load Balancer MQTT Broker Data Store Stream Processing Batch Processing Queues / Logs Typical Components
- 17. Load Balancing Data Ingestion Stream Analytics Data Storage Batch Processing High Level IoT Data Processing Architecture
- 18. Load Balancer Enables distribution of workloads across multiple computing resources First line of defense Central component for horizontal scaling,high availability and resiliency
- 19. MQTT Broker Central Communication software for Internet connected devices Distributes raw messages to the backend Allows bi-directional communication for device-to-cloud and cloud-to-device communication
- 20. (Big) Data Store Multiple data stores may be needed,persistent and ephemeral: Data lake, metadata stores,data marts,caches Raw data as well as refined data may be stored IoT architectures almost always deal with Big Data problems
- 21. Queues / Logs Decoupling of data ingestion and processing Add fault-tolerance Persistent and in-memory options
- 22. Stream Processing Processes data continuously when it arrives Short Processing time per value Result is available in near-time
- 23. Batch Processing Processes high volume of data Result is available after whole batch is processed Processing time can be hours or days
- 24. Difference between Stream & Batch Processing?
- 25. Stream vs Batch Processing Running Sushi Candy Store Stream Batch Supply Access Selection Running Sushi virtually endless sequential, as items pass by continuous Candy Store limited by available containers random access one-off selection Source: https://www.mapr.com/blog/data-processing-vocabulary-101-key-terms-you-need-know
- 27. Bare Metal Virtual Machines Containers Deployment Options
- 28. Bare Metal Perfect for On-Premise Installations Best performance Unflexible for dynamic workloads
- 29. Virtual Machines Ideal for cloud environments Full OS isolation Easy to provision
- 30. Virtual Machines
- 31. Containers Ideal for cloud environments with multiple services that need to be orchestrated All containers per Runtime share the same kernel Very easy to implement Infrastructure as Code
- 33. Containers - Docker vs VMs Docker Virtual Machines Source: https://www.docker.com/what-docker
- 35. Resiliency DEFINITION “The capacity to recover quickly from difficulties; toughness.” Oxford Dictionary
- 36. Resiliency The ability for a software system as a whole to recover from failures Failure of individual components is expected and the system as a whole can continue to work (with service degradation) and is able to recover All software components need to be designed for resiliency. Resilient systems often prefer Availability and Partition Tolerance if possible.
- 37. 1. The network is reliable. 2. Latency is zero. 3. Bandwidth is infinite. 4. The network is secure. 5. Topology doesn't change. 6. There is one administrator. 7. Transport cost is zero. 8. The network is homogeneous. Fallacies of Distributed Computing
- 38. High Availability DEFINITION “High Availability refers to the availability of resources in a computer system, in the wake of component failures in the system.” IEEE
- 39. High Availability Typically refers to the Uptime of a whole system or service Achieved with redundancy for hardware and software components All components in a single system must be designed for high availability
- 40. Load Balancing Data Ingestion Stream Analytics Data Storage Batch Processing High Level IoT Data Processing Architecture
- 41. MQTT
- 42. Standardized at OASIS Designed for reliable communication over unreliable networks (e.g. mobile networks) Publish / Subscribe architecture Data agnostic Internet of Things Messaging Protocol MQTT
- 43. Pub / Sub temperature sensor MQTT-Broker laptop mobile device publish: “21°C“ publish: “21°C“ publish: “21°C“ subscribe subscribe
- 44. MQTT Use Cases Push Communication Reliable Communication over unreliable networks Constrained Devices Low Bandwidth and High Latency Enterprise backends to mobile communication
- 45. MQTT in IoT architectures Massively scalable - hundreds of thousands messages per second Bi-directional,event-driven push communication Very low protocol footprint, suitable for millions of devices
- 47. MQTT Broker Plugin vs Shared Subscribers vs Broker Plugin Shared Subscribers Client Load Balancing No business logic on broker Can be written in any language Client must take care itself of store-and- forward semantics No additional services needed No additional TCP connections needed Automatic Backpressure Handling Deeper Integration to MQTT lifecycle Horizontal scaling
- 48. Backpressure
- 49. Backpressure - Principle ILLUSTRATED 1 3 42 Data Flow Temporary overload Normal Flow System Crash Source: http://ferd.ca/queues-don-t-fix-overload.html
- 50. Little’s Law The long-term average number of customers in a stable system L is equal to the long-term average effective arrival rate, λ, multiplied by the (Palm-)average time a customer spends in the system, W; or expressed algebraically: L = λW. John Little
- 52. Solution 2 - Load Shedding
- 53. Solution 3 - Block Producer
- 54. Solution 4 - Slow down producer
- 57. Load balancers Distribute client connects over multiple MQTT brokers SSL/TLS Termination, special hardware Software and Hardware options
- 58. Relational Database ACID compliant Easy to use Stored data can be queried easily
- 59. Cassandra Distributed database Scales to hundreds of nodes and terabytes of data Fast reads and writes, scalable performance
- 60. Kafka Distributed streaming message queue Data is organized in topics and partitions Scales to hundreds of nodes and terabytes of data
- 61. Apache Spark Framework for batch processing of large datasets Supports streaming via micro-batch processing Can process terabytes of data
- 62. Apache Flink Framework for streaming processing of large datasets also supports batch processing Aims to provide high throughput with low latency
- 63. Akka Framework for highly concurrent data processing Provides an actor model Suited for CPU intensive processing
- 64. Distributed File System Distribute Storage over multiple machines Simple alternative to HDFS for smaller data Built for commodity server hardware
- 65. Hadoop Ecosystem for very large datasets Distributed Filesystem HDFS Scales to hundreds of nodes and terabytes of data
- 66. HiveMQ MQTT broker built for enterprise applications Extensive plugin system Scales to > 10 million of concurrent connections
- 68. Load Balancing Data Ingestion Stream Analytics Data Storage Batch Processing High Level IoT Data Processing Architecture
- 69. Minimal Architecture for Low Volume Data
- 70. Load Balancing Data Ingestion Stream Analytics Data Storage Batch Processing Example Architecture for Medium Volume Data
- 71. Load Balancing Data Ingestion Stream Analytics Data Storage Batch Processing Example Architecture for High Volume Data
- 72. Lessons Learned
- 74. “A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable.“ All components must be resilient Leslie Lamport, Turing Award Winner 2013
- 75. Know your limits
- 76. (Pre-) compute on the edge
- 77. Only persist data you really need
- 79. Don’t choose a technology just because it’s cool
- 80. dc-square GmbH www.dc-square.de [email protected] Dominik Obermaier [email protected] @dobermai Get in touch HiveMQ - Enterprise MQTT Broker www.hivemq.com @hivemq