![978-1-098-14286-5
[LSI]
Architecting IoT Solutions on Azure
by Blaize Stewart
Copyright © 2024 Blaize Stewart. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.
O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are
also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional
sales department: 800-998-9938 or corporate@oreilly.com.
Acquisitions Editor: Aaron Black
Development Editor: Rita Fernando
Production Editor: Clare Laylock
Copyeditor: Charles Roumeliotis
Proofreader: Piper Editorial Consulting, LLC
Indexer: Potomac Indexing, LLC
Interior Designer: David Futato
Cover Designer: Karen Montgomery
Illustrator: Kate Dullea
January 2024: First Edition
Revision History of the First Edition
2024-01-09: First Release
See http://oreilly.com/catalog/errata.csp?isbn=9781098142865 for release details.
The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Architecting IoT Solutions on Azure, the
cover image, and related trade dress are trademarks of O’Reilly Media, Inc.
The views expressed in this work are those of the author and do not represent the publisher’s views. While
the publisher and the author have used good faith efforts to ensure that the information and instructions
contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or
omissions, including without limitation responsibility for damages resulting from the use of or reliance
on this work. Use of the information and instructions contained in this work is at your own risk. If any
code samples or other technology this work contains or describes is subject to open source licenses or the
intellectual property rights of others, it is your responsibility to ensure that your use thereof complies
with such licenses and/or rights.](https://image.slidesharecdn.com/27434345-250509181006-818cc2dc/85/Architecting-Iot-Solutions-On-Azure-Conquering-Complexity-For-Scalable-Device-And-Data-Management-1st-Edition-Blaize-Stewart-8-320.jpg)
![This ebook is for the use of anyone anywhere in the United
States and most other parts of the world at no cost and with
almost no restrictions whatsoever. You may copy it, give it away
or re-use it under the terms of the Project Gutenberg License
included with this ebook or online at www.gutenberg.org. If you
are not located in the United States, you will have to check the
laws of the country where you are located before using this
eBook.
Title: The Waterloo Campaign, 1815
Author: William Siborne
Author of introduction, etc.: Edward Arber
Release date: November 11, 2018 [eBook #58268]
Language: English
Credits: Produced by Brian Coe, Graeme Mackreth and the
Online
Distributed Proofreading Team at http://www.pgdp.net
(This
file was produced from images generously made
available
by The Internet Archive)
*** START OF THE PROJECT GUTENBERG EBOOK THE WATERLOO
CAMPAIGN, 1815 ***](https://image.slidesharecdn.com/27434345-250509181006-818cc2dc/85/Architecting-Iot-Solutions-On-Azure-Conquering-Complexity-For-Scalable-Device-And-Data-Management-1st-Edition-Blaize-Stewart-60-320.jpg)
![Prussian operations; more especially as regards the opening of the
Campaign.
A Dutch work published, apparently under authority, by Major Van
Löben Sels, Aide de Camp to his Royal Highness Prince Frederick of
the Netherlands, and entitled B dragen tot de Kr gsgeschiedenis van
Napoleon Bonaparte, of which I was not previously in possession, has
enabled me to give additional particulars respecting the movements
and dispositions of the most advanced portion of the Dutch-Belgian
troops, on the first advance of the Enemy; and also to explain
particular circumstances and qualify some observations respecting
those troops which appeared in former Editions.
The Editor of an Article in The Quarterly Review, No. CLI., entitled
Marmont, Siborne, and Alison, having, in his comments upon this
Work, denied the accuracy of one or two important facts therein
stated; I have, in notes at pages 57 and 152,[2] entered into more
minute details, which explain the grounds that warrant me in
adhering to the original statements.
The observations made in the Preface of a Volume of "Murray's
Home and Colonial Library," entitled The Story of Waterloo, and the
palpable embodiment of the present Work into the pages of the
latter, have been such as could scarcely fail to attract attention; and
I have accordingly appended to this Edition, in a separate form,
some remarks upon that publication.[3] Public opinion (if I may
judge by the unanimous consent of the press) having so distinctly
pronounced its acknowledgment of the value of my Work, as one of
History; I could not disregard the conduct of a Writer, who, in the
first place endeavours to depreciate that value, and then
unblushingly makes the most ample and unlicensed use of it for his
own purposes.
W. SIBORNE.
18th June 1848.](https://image.slidesharecdn.com/27434345-250509181006-818cc2dc/85/Architecting-Iot-Solutions-On-Azure-Conquering-Complexity-For-Scalable-Device-And-Data-Management-1st-Edition-Blaize-Stewart-65-320.jpg)
![FOOTNOTES:
[2] Omitted in this Fourth Edition.—E.A.
[3] Omitted in this Fourth Edition.—E.A.
The Duke of Wellington.
PREFACE TO THE SECOND EDITION.
THE circumstance of the First Edition having been sold off within a
very few days, combined with the highly favourable notices taken of
the Work by professional as well as other critics, and, I may be
permitted to add, the very flattering encomiums which have been
pronounced upon it by so many who, from their position, are the
most competent to form an opinion on its merits, cannot fail to
afford proofs, the most satisfactory to the Public, and, at the same](https://image.slidesharecdn.com/27434345-250509181006-818cc2dc/85/Architecting-Iot-Solutions-On-Azure-Conquering-Complexity-For-Scalable-Device-And-Data-Management-1st-Edition-Blaize-Stewart-66-320.jpg)
![time, the most gratifying to the Author, that, in the production of
these Volumes,[4] upon a subject of such stirring national interest,
neither the expectations of the former have been altogether
disappointed, nor the labours of the latter bestowed in vain.
The present Edition contains corrections on one or two points of
trivial importance, to which my attention has been directed; and I
shall be happy to receive further information from surviving Eye
Witnesses who may discover any instances in which the facts related
appear either inaccurately or insufficiently explained.
W. SIBORNE.
August 23rd, 1844.
FOOTNOTES:
[4] The First and Second Editions of this Work were each
published in Two Volumes.—E.A.
PREFACE.
SOME years ago, when constructing a Model of the Field of
Waterloo, at a particular period of the Battle; I found it necessary to
make great exertions to procure that detailed information for which I
had sought in vain in the already numerous published accounts of
the military transactions of 1815. Anxious to ensure the rigorous
accuracy of my work, I ventured to apply for information to nearly all
the surviving Eye Witnesses of the incidents which my Model was
intended to represent. In every quarter, and among Officers of all
ranks, from the General to the Subaltern, my applications were](https://image.slidesharecdn.com/27434345-250509181006-818cc2dc/85/Architecting-Iot-Solutions-On-Azure-Conquering-Complexity-For-Scalable-Device-And-Data-Management-1st-Edition-Blaize-Stewart-67-320.jpg)
Architecting Iot Solutions On Azure Conquering Complexity For Scalable Device And Data Management 1st Edition Blaize Stewart
- 1. Architecting Iot Solutions On Azure Conquering Complexity For Scalable Device And Data Management 1st Edition Blaize Stewart download https://ebookbell.com/product/architecting-iot-solutions-on- azure-conquering-complexity-for-scalable-device-and-data- management-1st-edition-blaize-stewart-54868690 Explore and download more ebooks at ebookbell.com
- 2. Here are some recommended products that we believe you will be interested in. You can click the link to download. Architecting Iot Solutions On Azure Blaize Stewart https://ebookbell.com/product/architecting-iot-solutions-on-azure- blaize-stewart-59141488 Internet Of Things For Architects Architecting Iot Solutions By Implementing Sensors Communication Infrastructure Edge Computing Analytics And Security 1st Edition Perry Lea https://ebookbell.com/product/internet-of-things-for-architects- architecting-iot-solutions-by-implementing-sensors-communication- infrastructure-edge-computing-analytics-and-security-1st-edition- perry-lea-7396328 Industrial Iot For Architects And Engineers Architecting Secure Robust And Scalable Industrial Iot Solutions With Aws 1st Edition Joey Bernal https://ebookbell.com/product/industrial-iot-for-architects-and- engineers-architecting-secure-robust-and-scalable-industrial-iot- solutions-with-aws-1st-edition-joey-bernal-47589936 Dependable Iot For Human And Industry Modeling Architecting Implementation Vyacheslav Kharchenko https://ebookbell.com/product/dependable-iot-for-human-and-industry- modeling-architecting-implementation-vyacheslav-kharchenko-11052240
- 3. Architecting A Knowledgebased Platform For Design Engineering 40 Zhenjun Ming https://ebookbell.com/product/architecting-a-knowledgebased-platform- for-design-engineering-40-zhenjun-ming-46703900 Architecting Cloud Computing Solutions Build Cloud Strategies That Align Technology And Economics While Effectively Managing Risk 1st Edition Kevin L Jackson https://ebookbell.com/product/architecting-cloud-computing-solutions- build-cloud-strategies-that-align-technology-and-economics-while- effectively-managing-risk-1st-edition-kevin-l-jackson-47173342 Architecting Vuejs 3 Enterpriseready Web Applications Build And Deliver Scalable And Highperformance Enterpriseready Applications With Vue And Javascript 1st Edition Solomon Eseme https://ebookbell.com/product/architecting-vuejs-3-enterpriseready- web-applications-build-and-deliver-scalable-and-highperformance- enterpriseready-applications-with-vue-and-javascript-1st-edition- solomon-eseme-48489434 Architecting Modern Data Platforms Jan Kunigk Ian Buss Paul Wilkinson https://ebookbell.com/product/architecting-modern-data-platforms-jan- kunigk-ian-buss-paul-wilkinson-49042208 Architecting Distributed Transactional Applications Dataintensive Distributed Transactional Applications 1st Edition Guy Harrison https://ebookbell.com/product/architecting-distributed-transactional- applications-dataintensive-distributed-transactional-applications-1st- edition-guy-harrison-49540988
- 5. Blaize Stewart Architecting IoT Solutions on Azure Conquering Complexity for Scalable Device and Data Management
- 6. DATA “Blaize distills years of hands-on experience designing and implementing IoT solutions at a global scale into a fantastic guide for beginners and a reference manual for practitioners. This is the one book you must read if you’re architecting IoT solutions!” —Ken Muse Four-time Microsoft MVP and Senior DevOps Architect, GitHub Architecting IoT Solutions on Azure Twitter: @oreillymedia linkedin.com/company/oreilly-media youtube.com/oreillymedia How can you make sense of the complex IoT landscape? With dozens of components ranging from devices to metadata about the devices, it’s easy to get lost among the possibilities. But it’s not impossible if you have the right guide to help you navigate all the complexities. This practical book shows developers, architects, and IT managers how to build IoT solutions on Azure. Author Blaize Stewart presents a comprehensive view of the IoT landscape. You’ll learn about devices, device management at scale, and the tools Azure provides for building globally distributed systems. You’ll also explore ways to organize data by choosing the appropriate dataflow and data storage technologies. The final chapters examine data consumption and solutions for delivering data to consumers with Azure. Get the architectural guidance you need to create holistic solutions with devices, data, and everything in between. This book helps you: • Meet the demands of an IoT solution with Azure-provided functionality • Use Azure to create complete scalable and secure IoT systems • Understand how to articulate IoT architecture and solutions • Guide conversations around common problems that IoT applications solve • Select the appropriate technologies in the Azure space to build IoT applications Blaize Stewart is a Microsoft Azure MVP with 20 years of development and infrastructure experience who’s architected and implemented numerous small to enterprise-grade applications—everything from embedded systems, handheld devices, desktop apps, and web apps to scalable systems in the cloud with global scale. 9 7 8 1 0 9 8 1 4 2 8 6 5 5 6 5 9 9 US $65.99 CAN $82.99 ISBN: 978-1-098-14286-5
- 7. Blaize Stewart Architecting IoT Solutions on Azure Conquering Complexity for Scalable Device and Data Management Boston Farnham Sebastopol Tokyo Beijing Boston Farnham Sebastopol Tokyo Beijing
- 8. 978-1-098-14286-5 [LSI] Architecting IoT Solutions on Azure by Blaize Stewart Copyright © 2024 Blaize Stewart. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or [email protected]. Acquisitions Editor: Aaron Black Development Editor: Rita Fernando Production Editor: Clare Laylock Copyeditor: Charles Roumeliotis Proofreader: Piper Editorial Consulting, LLC Indexer: Potomac Indexing, LLC Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea January 2024: First Edition Revision History of the First Edition 2024-01-09: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781098142865 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Architecting IoT Solutions on Azure, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the author and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.
- 9. Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi 1. The IoT Landscape. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Off Azure 2 IoT Devices 2 Edge Computing 3 Azure 4 IoT Messaging and Management 4 Data Processing 5 Data Persistence 8 Data Presentation Layer 10 Data Consumers 10 Monitoring, Logging, and Security 11 Conclusion 12 2. Azure-Centric IoT Devices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 IoT Devices as the Nexus of Three Domains 14 Hardware 15 Software 16 Cloud 16 IoT Devices and Microsoft 17 Microsoft the Software Company 17 Microsoft as a Hardware Company 18 Microsoft as a Cloud Company 19 IoT on Azure: Microsoft’s Combination of Software, Hardware, and Cloud 19 Azure Sphere 20 Azure Sphere Hardware 20 Azure Sphere Software 20 iii
- 10. Azure Sphere Cloud Services 21 What’s It For? 21 What Makes It Unique? 22 Azure MXChip 22 MXChip Hardware 22 MXChip Software 23 MXChip Cloud Services 24 What’s It For? 24 What Makes It Unique? 24 Kinect 25 Kinect Hardware 25 Kinect Software 25 Kinect Cloud Services 26 What’s It For? 26 What Makes It Unique? 26 Windows for IoT 26 Windows IoT Software 27 Windows for IoT Hardware 28 Windows for IoT Cloud Services 29 What’s It For? 29 What Makes It Unique? 29 Azure IoT Device SDKs 29 Supported Languages and Platforms 31 Real-Time Operating System (RTOS) 31 When All Else Fails, Use the APIs 32 Summary 32 3. How to Try Before You Buy, IoT Edition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Thinking Through Your Software 35 The User Experience 36 Collecting Data 36 Data Collected from User Inputs 37 Data Collected from the Environment 37 Data About the Device 37 Data About the Software on the Device 37 Device Simulators 38 Accelerate Development 38 Enable Feature Development Independent of Device Development 38 Enable Automated Testing 39 Device Simulator Best Practices 40 A Word About Device Simulator Services 41 Experiment Using Virtualization 41 iv | Table of Contents
- 11. Hardware Without Dev Boards 45 Creating a Device for the Examples 45 Setting Up the Sample Device or the Device Simulator 46 Dependencies 46 Cloning the Repository 47 Explore the Code 48 Committing to a Dev Board 50 Summary 51 4. The Device Lifecycle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Device Lifecycle Management 54 Research and Design 55 Three Phases of R&D 56 Hardware 57 Software 62 Manufacturing 66 Shipping 67 Claiming and Provisioning 67 Provision Devices with an IoT Hub through the Device Provisioning Service 69 Main Sequence 71 Communication 72 Twinning 72 Device Twinning with IoT Hub 73 Updates 76 Deprovisioning 81 Summary 82 5. Device Messaging. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 Synchronous Versus Asynchronous Messaging 84 Real-Time Versus Store-and-Forward Messaging 85 Bidirectional Versus Unidirectional Communication 85 Message Formatting 86 Properties 86 Body 88 Common Protocols 91 MQ Telemetry Transport (MQTT) 92 Advanced Message Queuing Protocol (AMQP) 92 Hypertext Transfer Protocol (HTTP) 93 Device-to-Cloud (D2C) Messaging 94 Telemetry 94 Events 95 Table of Contents | v
- 12. File Uploads 95 Message Routing 96 Message Enrichments 101 Cloud-to-Device (C2D) Messaging 102 Commands (Direct Methods) 102 General Messages 104 Custom Solutions 105 Security 105 Device Management 106 Message Routing 106 Scaling 107 Integrations with Azure 107 Summary 108 6. Life on the Edge. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 Why Use Edge Computing? 110 Better Response Time to Events on Premises 111 Reliable Connectivity for Critical Services 111 Access to Critical Services Without Bandwidth Constraints 111 Aggregations and Filtering Data to Reduce Network Traffic 111 Compute Offloading to Save Network Bandwidth 112 Data Localization When Dealing with Data Sovereignty and Security Issues 112 A Disconnected Cloud That Performs with No Network Connection 112 Container Basics 113 Azure IoT Edge 114 IoT Edge Modules 115 Message Brokering 115 Data 116 Storage 116 Bringing the Cloud Closer with AI 116 Extensibility (Bring Your Own Code) 117 Creating an IoT Edge Device and Deploying a Module 117 Install Azure IoT Edge 118 Deploy a Module on IoT Edge 118 Azure Arc and Kubernetes 120 IoT Edge or Arc with Kubernetes? 122 Setting Up Arc with MicroK8S 122 Set Up MicroK8S 123 Connect MicroK8S to Azure Arc 124 Install the Device Simulator to MicroK8S Using the Azure Portal 126 Azure Data Box Gateway 128 Azure Stack 128 vi | Table of Contents
- 13. Azure Stack Hub 128 Azure Stack HCI 129 Azure Stack Edge 129 Summary 130 7. Scalable Data Architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 General Principles for Data Storage 132 Partition Your Data Appropriately 132 Storage Is Cheap, Compute Is Expensive 133 Prefer Eventual Consistency over Strong Consistency 135 Separate Reads and Writes 137 Treat Writes as Idempotent 137 Store Data in an Optimized Form for Its Intended Use 138 Create Retention Policies 139 Landing Your Data 139 Azure Blob Storage 140 Azure Data Lake 141 General-Purpose Blob Storage Versus Data Lake 141 Set Up Azure Blob Storage to Land Data 142 What About File Shares or File Sharing Services? 143 Online Transaction Processor (OLTP) Versus Online Analytics Processor (OLAP) 144 OLTP Solutions on Azure 145 Cosmos DB 145 Azure Data Explorer (ADX) 147 Azure SQL 147 Open Source Databases 149 OLAP Solutions on Azure 149 Azure Synapse Analytics 150 HDInsight 151 Data Lakes Versus Data Warehouses 152 Lakehouse 153 Hybrid Transaction Analytics Processor (HTAP) 154 OLTP as OLAP 154 Creating an HTAP with Cosmos DB and Azure Synapse Analytics 155 Summary 160 8. Data Processing Architectures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 Data Storage (Again) 164 Data Processing Cycle 164 Data Collection 165 Data Preparation 165 Table of Contents | vii
- 14. Data Input 165 Data Processing 165 Output 167 Storage 167 Data Movement 167 Hot Path 168 Cold Path 171 Warm Path 174 Which Type of Data Path Should You Use? 175 Data Architectures 176 Lambda Architecture 177 Kappa Architecture 178 Delta Architecture 181 Which Style of Architecture Should You Use? 182 Summary 183 9. Hot Path Data Processing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 Messaging Platforms 186 Hot Paths 186 Azure Stream Analytics 187 Azure Functions 191 Azure Logic Apps 195 Azure Service Bus 198 Queues and Topics 198 Auto-Forwarders and Subscription Filtering 198 Topic Actions 199 Performance Considerations 199 Create a Service Bus with a Topic and Subscriptions 200 Create a New Route on IoT Hub 201 Create Some Functions Save Data 202 Summary 204 10. Cold Path Data Processing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 Azure Data Explorer 208 Batch Processing on Azure 210 Azure Batch 211 Create a Batch Account 213 Set Up the Batch Job 213 Azure Data Factory 215 Create a Data Factory to Move Data 216 Create a Source Dataset from Your Storage Account 216 Create a Data Sink for Cosmos DB 217 viii | Table of Contents
- 15. Create a Data Flow to Move Data 218 Create a Pipeline to Move Data 218 Start the Data Flow 219 Summary 220 11. The Servicing Layer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221 Datasets 222 Data Formats for Datasets 223 Push-Style Delivery 225 Cold Path Push 225 Hot Path Push 226 Pull-Style Delivery 232 Azure Data Share 233 HTTP APIs 234 Hybrid approaches 238 Summary 239 12. Data Consumers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 Reporting Tools 242 Business Intelligence Tools 242 Power BI 244 Connecting Power BI to Data 245 Applications 247 External Systems Integrations 248 Raw Data Consumers 250 Security and Privacy 250 Security 251 Data Privacy 251 Summary 253 13. Monitoring and Logging. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255 Setting Goals with Service Level Agreements 255 Monitoring Your Solution 257 Azure Monitor 260 Data Collection 260 Azure Data Explorer (Log Analytics) 261 Kusto Query Language (KQL) 261 Monitoring and Alerting 262 Azure Application Insights 263 Instrumentation 264 Application Logging 264 User Analytics 265 Table of Contents | ix
- 16. Azure Security Center 265 Security Assessments 265 Security Monitoring 266 Incident Response 266 Azure Sentinel 267 Summary 268 14. IoT Security. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269 Software Vulnerabilities 270 Malware 271 Botnets 271 Ransomware 272 Data Leaks 273 DoS 274 Insecure Communications 274 Device Spoofing 275 Insecure Data 275 Lax Access Controls 276 Physical Threats 277 Lax Network Security 277 DNS Threats 278 Man-in-the-Middle 279 Social Engineering 279 Advanced Persistent Threats 280 Managing Threats with Microsoft Defender for IoT 280 Summary 281 15. Further Reading. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 Devices 283 Edge and Containers 286 Containers and GitOps 286 Kubernetes on the Edge 287 Azure IoT Edge 288 IoT Management 288 Data Architecture 288 Conclusion 290 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291 x | Table of Contents
- 17. Preface At a Gartner conference in sunny San Diego around 2012, I first encountered the term “IoT Architecture.” At first glance, it seemed a simple idea: the nexus of internet-connected devices. Just as the mobile wave was cresting and cloud comput‐ ing was gaining traction, the challenge of efficiently managing IoT workloads emerged as a resonant topic among technologists and device makers everywhere. As I delved deeper, this seemingly straightforward realm revealed myriad nuances. As it turns out, the Internet of Things (IoT) isn’t merely a web of gadgets that flicker to life at the touch of a button. It’s a sophisticated network of interconnected devices, auton‐ omously collecting, transmitting, and receiving vast amounts of data. As I ventured further into this domain, even more pressing questions surfaced: How does one seamlessly update all those devices? What measures ensure their tamper-resistance? And perhaps most dauntingly, how can one effectively process and analyze the deluge of data they produce? In response to these questions, Microsoft has unveiled a vast IoT ecosystem on Azure, encompassing a range of Microsoft-managed cloud services, state-of-the-art edge components, and SDKs. Each component plays a pivotal role in the lifecycle of an IoT solution. Navigating this expansive realm requires methodical organization. I’ve found mapping out domains, creating a taxonomy, to be an invaluable strategy in clarifying the otherwise overwhelming world of IoT. Each domain, while self- contained, intertwines with others, forming a comprehensive picture I’ve come to describe as the IoT Landscape. In this book, I segment this intricate topic into smaller domains. My aim is to provide readers with both a panoramic view and deep dives into its many aspects. My goal extends beyond mere knowledge dissemination; I strive to empower you, the reader, to navigate the Azure IoT ecosystem with ease and design IoT solutions tailored to your distinct needs and budget. xi
- 18. Who Should Read This Book If you’re looking for a book that speaks to holistic IoT solutions on Azure, you’re in the right place. This book has a little something for everyone in IT, including man‐ agement, architects, engineers, and administrators. For folks in the C-suite and IT managers, this book will help inform your business decisions when implementing an IoT solution on Azure. You’ll learn why something is done a particular way, and why Azure is a versatile platform for solutions. Defi‐ nitely start with Chapter 1 to build your foundational knowledge of what goes into an IoT solution. For those in architect roles, there will be plenty for you to feast on in this book. I’ll serve up the nitty-gritty details of each domain so you can enrich your understanding of IoT solutions on Azure. Whether you’re a new architect or a battle-tested one, you’ll find something of interest in this book. For engineers who may be responsible for implementing many of these solutions, the domain-specific chapters in particular, such as those on data engineering or cloud mes‐ saging, will give you what you need to know to implement an Azure-based solution. Finally, for administrators and those managing cloud infrastructure, you’ll be particu‐ larly interested in the chapters dedicated to monitoring, security, and governance. In essence, no matter your role in IT and IoT, there’s something here for you! Navigating This Book Chapter 1 introduces you to the Azure IoT Landscape and gives an overview of how all the domains I discuss in this book fit together into an IoT solution. Chapter 2 marks the beginning of your journey through the Azure IoT Landscape. Chapters 2 through 6 focus on dealing with devices at scale. These discussions pro‐ vide theory and practice around building devices interacting with Azure IoT offerings and managing those devices through a device’s lifecycle. Although the book discusses a few devices from Microsoft oriented toward Azure, the primary takeaway is how to think about those devices in the context of Azure under the guiding principles out‐ lined in the architecture. Chapters 7 through 12 shift the focus to what one does once the devices have done their job and have sent data to the cloud. These chapters are driven by architecture and show how to understand data architectures and implement them practically using Azure solutions. As before, it would be impossible to cover every permutation, and there are many different practical solutions for problems. Still, they can generally fit under a typical reference architecture that guides the decision making about and integration of those services. xii | Preface
- 19. Chapter 13 highlights the need to watch over not just the information from IoT devi‐ ces but also the overall well-being of the whole IoT system. It talks about setting clear goals, measuring them with KPIs, and using tools like Azure Monitor and Azure Security Center. These tools help gather records, solve problems, and boost security, which Chapter 14 digs deeper into. Chapter 14 is all about making sure IoT systems are safe. It gives tips on protecting software, stopping malware, keeping data safe, and much, much more. Lastly, Chapter 15 gives you more resources to learn about topics like creating devi‐ ces, managing data, and working with edge solutions, which are all things talked about in this book. Throughout this book, you’ll encounter hands-on examples that will show you how the technology all works together. Some of the examples have many parts and build on examples from previous chapters. There’s no way I can do an example for every‐ thing, but what I have chosen exemplifies the concepts and patterns. That’s the important part. At the end of the day, whatever your IoT solution looks like, I hope that you will use this book as a resource to help inform the decisions you make along the way. There is nothing simple about IoT, but investing the time to read and consider architecture up front can save tons of time and money down the road. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program ele‐ ments such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user. Constant width italic Shows text that should be replaced with user-supplied values or by values deter‐ mined by context. Preface | xiii
- 20. Using Code Examples Supplemental material (code examples, exercises, etc.) is available for download at https://oreil.ly/supp-AIoT. If you have a technical question or a problem using the code examples, please send email to [email protected]. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Architecting IoT Solu‐ tions on Azure by Blaize Stewart (O’Reilly). Copyright 2024 Blaize Stewart, 978-1-098-14286-5.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at [email protected]. O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit https://oreilly.com. xiv | Preface
- 21. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-889-8969 (in the United States or Canada) 707-829-7019 (international or local) 707-829-0104 (fax) [email protected] https://www.oreilly.com/about/contact.html We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/ArchitectingIoT. For news and information about our books and courses, visit https://oreilly.com. Find us on LinkedIn: https://linkedin.com/company/oreilly-media. Follow us on Twitter: https://twitter.com/oreillymedia. Watch us on YouTube: https://youtube.com/oreillymedia. Acknowledgments First, I would like to thank all of my reviewers, Marvin Garcia, Ken Muse, and Robert Stackowiak. These amazing people have deep knowledge and experience in IoT and data; they helped fill in a lot of the places I was missing and even pointed out things that were probably less than ideal. I’d also like to thank my outstanding editor, Rita Fernando, who helped keep things on track and offered valuable feedback on making the book better all the way around. And I’d like to thank Aaron Black and Jon Hassell at O’Reilly for helping this project get started. I’d like to thank my loving wife who put up with me during the many long hours of writing this book. It’s no simple task, yet she helped me through it. And lastly, I’d like to thank Dad, Blaize Stewart I (I’m Blaize II) who got me into com‐ puting at a young age. Who knew that the plunky Commodore 64 would set the tra‐ jectory of my career? Preface | xv
- 23. CHAPTER 1 The IoT Landscape When contemplating the Internet of Things (IoT), you might initially envision the countless devices seamlessly interwoven into our modern landscape. IoT has perme‐ ated practically every part of our lives, from the simplest light bulb to orchestrations on a monumental scale within the contours of smart cities. These devices are aptly dubbed the “things” in the IoT realm. To fathom the entirety of this intricate tapestry, one must not simply ascend to the metaphorical 40,000-foot vantage from an air‐ plane. This perspective reveals the devices and the elaborate symphony of parts, all orchestrated in harmonious collaboration. It’s the IoT Landscape—a panoramic view that captures the collective essence of every element and orchestration that conspires to set these IoT devices into remarkable motion. It gives a taxonomy to each individ‐ ual part, assigning purpose and place within the grand design. In this book, I’ll take you through the IoT Landscape that is specific to Azure, which offers a plethora of Microsoft-managed cloud services, edge components, and software development kits (SDKs) to choose from. So, without further ado, take your first look at the Azure IoT Landscape shown in Figure 1-1. 1
- 24. Figure 1-1. The IoT Landscape for Azure This book explores each facet within this landscape, delving into the interplay of each part within the framework of a comprehensive IoT solution. Envision this as a voy‐ age—an expedition starting from the leftward side (with devices) that generally moves to the right. In navigating this trajectory, you’ll traverse significant junctures, including edge computing, IoT management and messaging, data pathways, data persistence, data servicing, and the expansive sphere of data consumers. However, to acclimate you to this voyage, let us embark with an overview of each point on the journey. This strate‐ gic prelude will chart your course, starting with devices. Off Azure The first section of the Azure IoT Landscape is, oddly enough, the things that exist outside of Azure: devices and edge computing. IoT Devices It’s hard to define an IoT device exactly because it’s not as simple as one might think. A textbook definition for the “Internet of Things” would be a network of intercon‐ nected devices that can collect, transmit, and receive data without direct human intervention. Sure, some things are pretty obvious. A smart light bulb or connected appliance is an IoT device, but that line gets a little blurry with things like televisions or media players. Quite the opposite might be true with some devices. They may be purpose-built to work with humans, like an excavator or smart toothbrushes. Notice what I said: they are purpose-built, like moving dirt or cleaning your teeth. The device has a specialized purpose rather than having a general purpose like a laptop or smartphone. 2 | Chapter 1: The IoT Landscape
- 25. IoT devices generally, though, have some common characteristics. With devices, you almost always have connectivity for getting the device to talk to the internet, sensors for collecting data, data processing on the device to deal with the collected data, and actuators for responding to commands. Combining these things can create a complex or simple device, depending on the context. Because there’s a huge variety of devices ranging from light bulbs to traffic cameras, to bulldozers, to heart monitors, it’s impossible to address all of them. It’s even daunt‐ ing to attempt to generalize because, inevitably, there seem to be exceptions. In Chap‐ ter 2, I’ll take you through the Azure-centric hardware offerings from Microsoft and we will study how they work together for different purposes. I’ll also show you the different software offerings for constrained devices (also known as resource- constrained or edge devices, IoT devices with limitations in computing power, mem‐ ory, energy, and communication capabilities) and unconstrained devices (cloud- connected or gateway devices, with higher computational power, memory, and communication capabilities compared to constrained devices). These devices are designed to perform specific tasks with minimal resources and are often deployed in remote or challenging environments where frequent maintenance or resource replen‐ ishment might be difficult. In Chapter 3, I’ll show you how to get started with exploring IoT devices without needing to invest in hardware right away. Devices ultimately connect to the internet. They can do that directly in some cases, but in others, they may use a proxy that assists the devices in their work. Such a proxy would be part of edge computing. Edge Computing Edge computing is a decentralized approach to data processing where computations occur closer to the data source, reducing latency and enhancing real-time decision making. It involves processing data at or near the devices or local servers rather than sending all data to a centralized cloud server. Edge computing is extending the cloud into a data center rather than thinking of the cloud as an extension to the data center, which is more of a hybrid model. It is espe‐ cially valuable for applications that require quick responses, low latency, efficient use of network bandwidth, and offline and intermittent connected computing scenarios. Edge computing suits scenarios where immediate data processing is essential, such as industrial automation, remote monitoring, and smart cities. IoT, therefore, is a perti‐ nent topic in the scope of edge computing. As of the writing of this book, edge computing is a hot topic, with a flurry of activity in Microsoft space attempting to bring Azure closer to the edge. Within this context, Microsoft has two major initiatives encompassing many different resources. Off Azure | 3
- 26. Specifically for the IoT, Azure offers Azure IoT Edge runtime. It’s a lightweight con‐ tainer orchestration platform for building modular IoT devices and edge appliances. Beyond that, Azure offers Azure Arc, a general-purpose tool for extending the Azure control plane into off-Azure environments. One of its biggest abilities is its capacity to manage Kubernetes clusters. Users can deploy services like stream processing, SQL Server, machine learning, and messaging to the edge through Arc. Because Arc lever‐ ages Kubernetes, Azure Arc and Azure IoT Edge both use Docker containers, so you can’t ignore that! Containers, in any case, are a part of these solutions. Chapter 6 goes into detail about the edge options for Azure. These different technologies make edge computing one of the most exciting parts of IoT solutions. Even if you don’t plan on using edge, some things about edge comput‐ ing can be incorporated into your devices. Regardless of how you connect your devi‐ ces to the internet, they will need a place online to manage them. That’s the next part of the IoT Landscape. Azure Now, we come to the IoT Landscape that exists in Azure. Here, we have IoT messag‐ ing and management, data processing, data persistence, and data presentation. IoT Messaging and Management IoT messaging and management encompasses a massive set of interrelated topics. A cloud solution like Azure IoT Hub combines them into a platform-as-a-service (PaaS) offering that you can leverage as a solution builder. Here’s what Azure IoT Hub and its related services provide for your devices in terms of IoT messaging and management: Device registration and provisioning Devices must be registered within the IoT Hub to establish secure communica‐ tion. Azure Device Provisioning Services (DPS) provides mechanisms for auto‐ matic device provisioning, enabling devices to join the network securely. Secure communication Devices connect to Azure IoT Hub using secure communication protocols such as MQTT, AMQP, or HTTPS. End-to-end security over the internet is main‐ tained through device authentication, authorization, and encryption over one of these communication protocols. Device twins and desired properties Twinning is a way of tracking the settings of devices on the cloud without having to query each device in the cloud. A “device twin” maintained by Azure IoT Hub represents the device’s state and metadata. Device twins store desired properties that can be set by the cloud application and reported properties that devices update. 4 | Chapter 1: The IoT Landscape
- 27. Cloud-to-device (C2D) messaging Azure IoT Hub enables cloud applications to send commands or messages to individuals or groups of devices. It provides APIs and routing capabilities for these messages. Devices receive these messages and can take action accordingly. Device-to-cloud (D2C) messaging Devices can send telemetry data, status updates, and other information to the cloud. Azure IoT Hub provides many ways to move this data onto downstream data processing engines for analysis and monitoring, like databases, Azure Func‐ tions, Logic Apps, and Stream Analytics. Device management and monitoring Azure IoT Hub provides tools to monitor the health and status of connected devices. Devices can report their state, and cloud applications can query this information for troubleshooting and maintenance. Edge device management We already discussed Azure IoT Edge, but Azure IoT Hub supports managing IoT devices at the edge, where data processing occurs closer to the data source. Azure IoT Edge extends this capability for deploying and managing container‐ ized workloads on edge devices. Scalability and performance Azure IoT Hub is designed to handle large numbers of devices and high message throughput. With thousands of devices connected at a time, the potential data deluge is daunting. It offers tiered options for scaling to accommodate diverse IoT scenarios. Chapter 4 dives into all things related to device lifecycle management. This process encompasses everything from a device’s inception to the day it’s offloaded and no longer used. It entails device design, device provisioning, device maintenance, and finally, device decommissioning and disposal. Chapter 5 covers what happens during a device’s main sequence, where it communi‐ cates with the cloud through messaging from the device to the cloud and messages from the cloud to the device. As you can see, there’s no one thing that IoT Hub does for managing messaging and devices at scale. It’s the central nervous system of your IoT solution. It brokers the rela‐ tionship between your potentially thousands of devices and the downstream systems that handle the telemetry and data coming off those devices through data processing. Data Processing Data processing is a crucial component in IoT solutions that involves managing, pre‐ paring, and analyzing the vast amounts of data generated by IoT devices. Data Azure | 5
- 28. processes ensure data is collected, transformed, and utilized effectively to extract insights and drive meaningful actions. Data processing involves gathering data from these devices, integrating data from diverse sources into datasets, ensuring the collected data is stored appropriately in databases or data lakes, cleaning and transforming to ensure its quality and consis‐ tency, and ultimately presenting the data with well-designed data schemas so data can be queried and analyzed. There are a ton of ways to do this, though. You can think of this as real-time process‐ ing and batch processing. Real-time processing involves immediate analysis of streaming data from IoT devices, enabling quick responses to anomalies and trigger‐ ing alerts or actions. Batch processing analyzes historical data in predefined batches to uncover trends, patterns, and correlations that offer insights into past events. This approach is particularly valuable for long-term analysis. In both cases, data is summarized and consolidated through aggregation, reducing its volume while retaining its informative content. More advanced techniques like machine learning and artificial intelligence are employed to perform complex analyt‐ ics, identify hidden patterns, and predict future outcomes. The outcome of data pro‐ cessing is actionable insights that guide decision making, optimize operations, and enhance overall efficiency. These are part of data architecture. Data architecture provides a general framework for how one thinks about data pro‐ cessing. It spans from data collection to insightful outcomes. It encompasses method‐ ologies, tools, and strategies for efficiently handling large volumes of data from sources like IoT devices. It looks at data sources, how data is received, how data is integrated, how data is transformed, how data is processed, and how data is presen‐ ted. When designing data processing architecture, factors such as real-time versus batch processing, data latency, data quality, integration with existing systems, and scalability need to be considered. Chapter 7 starts the last half of the book, shifting focus to creating scalable data solu‐ tions on Azure. Chapter 8 extends the processing concepts by looking at three differ‐ ent architectures commonly used in IoT data. While I just named two broad categories—real-time and batch-style processing— some more nuance can be applied to this context with hot, warm, and cold paths. Each of these takes a different approach to data processing within data architecture. Hot paths “Hot path” data refers to the portion of data within a system that requires immediate or real-time processing to facilitate quick decision making, rapid responses, and timely actions. (Hot path is usually thought of as “fast” processing, but it should not be confla‐ ted with speed because speed is not the only concern.) Hot path data is processed in 6 | Chapter 1: The IoT Landscape
- 29. real time or near real time, allowing for rapid analysis and decision making. It often contains critical information that, when acted upon swiftly, can prevent or mitigate issues, optimize processes, or enhance user experiences. These actions could be auto‐ mated responses, user notifications, or system parameter adjustments. In the context of IoT, hot path data entails telemetry events that feed into systems that provide real-time and near real-time responses to data. A canonical example of a hot path telemetry event may be a smoke detector sending a signal to get the fire department’s attention. Azure provides tooling for managing this through options like Stream Analytics, Azure Functions, Event Grids, Event Hubs, Service Buses, and other real-time pro‐ cessing capabilities. These all work well together to make complete solutions for IoT hot paths. I’ll talk about hot paths in more detail in Chapter 9. Still, there are more than hot paths. There’s something way cooler: warm paths. Warm paths Warm path data represents a category that holds a position between “hot path” and “cold path” data regarding its urgency and processing frequency. It is characterized by requiring faster processing than cold path data but not as immediate as hot path data. Like a hot path, speed is important, but it’s not the only factor. Warm path data pro‐ vides insights and support decisions that can tolerate a brief processing delay. Unlike hot path data that demands immediate action, warm path data is processed with moderate latency, falling into near real-time or slightly delayed analysis. Its pur‐ pose is to contribute insights that aid ongoing monitoring, optimization efforts, and decisions requiring timely attention without needing instant reactions. Warm path data is utilized in scenarios where efficiency and optimization are priorities. These situations benefit from insights that are prompt enough to drive action but don’t require an immediate response. Regarding data processing architecture, warm path data is typically processed using techniques like stream processing that balance real-time insights and a slight delay in processing. This ensures that the data is transformed into actionable insights within a timeframe that aligns with the needs of operational decision making and ongoing optimization. In essence, warm path data plays a crucial role in enabling organiza‐ tions to balance the urgency of hot path data and the longer-term insights derived from cold path data. For IoT, a warm path helps you gain insights into ongoing oper‐ ations, monitoring, and optimizations within a timeframe that balances prompt anal‐ ysis with acceptable delay. A warm path provides situational awareness by providing insights into changes or trends in IoT device behavior, performance, or environmen‐ tal conditions. See Chapter 10 for more details on warm paths. Azure supports warm paths through time-series data. Azure Data Explorer and Azure Stream Analytics are the primary services that accomplish this. Data Explorer is more of a database service but has the compute capacity and integrated services to make it Azure | 7
- 30. useful for warm paths. Stream Analytics can look at historical data in time windows, too. But for large windows of time or when data does not need immediate processing, you can consider the next type: cold paths. Cold paths Cold path data is a category in data processing and analytics characterized by its focus on long-term storage and historical analysis. It’s not exactly synonymous with batch processing, but it often is implemented as such. Unlike data in the hot or warm path, cold path data is stored and processed with a lower frequency of access. It serves purposes such as historical reporting, trend analysis, and gaining deep insights that don’t demand rapid responses. Processing of cold path data doesn’t require immedi‐ ate urgency and is subject to longer intervals, typically ranging from hours to days. Its primary utility lies in retrospective analysis, identifying trends, historical patterns, and anomalies that have occurred over a substantial period. It’s also useful as an inte‐ gration pattern for more traditional workloads that leverage more conventional extract, transform, and load (ETL) styles of integrations. Organizations tend to store cold path data using cost-effective storage solutions, as frequent and fast access is not a central requirement. It is used when regulatory com‐ pliance is essential, as industries need to retain historical data for auditing purposes. For businesses, cold path data contributes to generating historical reports, tracking performance over time, and making informed decisions based on a comprehensive historical context. For IoT, cold path data’s contribution extends to predictive analytics, such as when machine learning models are trained on historical patterns for forecasting. It aids in root cause analysis, enabling detailed investigations into past incidents or anomalies. Moreover, businesses utilize cold path data to plan for the future by understanding the consequences of past decisions and events. I focus on cold paths in Chapter 10. In data processing architectures, Azure Data Factory, Azure Synapse, Azure Data‐ bricks, and HDInsights provide the compute resources to work with the storage to make cold paths work. Cold path data is stored in data warehouses such as Azure Synapse, Azure Data Lake, specialized archival systems, or something as simple as object storage like Azure Blob Storage. All of these are part of persistence. Data Persistence Unless you deal only with transient data, you will need to store it somewhere. Even transient data systems, like message queues, will persist data for a while. Data persistence in the context of data processing refers to the methods and mecha‐ nisms used to store and maintain data beyond its initial processing or input. It ensures that data remains accessible, retrievable, and usable for various purposes such 8 | Chapter 1: The IoT Landscape
- 31. as analysis, reporting, and future processing. Different kinds of data persistence serve various needs in data processing architectures: Temporary or volatile storage Temporary storage involves holding data in memory or cache for immediate pro‐ cessing. This type of persistence is short-lived and is primarily used during the immediate processing stage. Once the processing is complete, data in temporary storage is discarded. It’s suitable for holding data that needs to be accessed fre‐ quently and quickly during processing, but it’s not intended for long-term reten‐ tion. You’ll encounter this storage for things like Azure Service Bus, Event Grids, and stream processing technologies like Azure Stream Analytics. Stream storage is used for managing real-time data streams generated by IoT devices, sensors, or other sources. This type of persistence allows for the immediate storage and pro‐ cessing of streaming data, enabling real-time analytics and decision making. Raw storage or data lakes Data lakes are repositories that store raw, unprocessed data in its original format. This form of persistence is ideal for storing diverse data types, such as structured, semi-structured, and unstructured data, without immediate structuring or trans‐ formation. Data lakes facilitate future processing and analysis, enabling organiza‐ tions to derive insights from data as needs arise. Batch processing and ETL tools use data lakes and raw storage like Blob Storage as part of their processing. Structured storage or data warehouses Structured storage, commonly found in data warehouses, involves storing data in a structured format that is optimized for query and analysis. This type of persis‐ tence is particularly suitable for processed and transformed data that is ready for reporting, analytics, and business intelligence. Data warehouses provide opti‐ mized performance for querying large datasets and generating insights. Data‐ bases are useful for querying data for transactional workloads. For IoT, there’s no one database that does everything well, but Azure Cosmos DB and Azure Data Explorer provide some level of transactional storage while Azure Synapse pro‐ vides more analytic storage. While no one chapter is dedicated to data persistence, this book talks about persis‐ tence quite a bit, especially in the context of data processing. In an IoT solution, you’re likely to run into many different kinds of data storage. You’ll use these as part of your data movement and processing. In data processing architectures, especially IoT, combining these data persistence methods creates different processing and analysis stages. Temporary storage aids immediate processing, data lakes store raw data, structured storage enables efficient querying, and archival storage ensures data retention. All these, at some level, how‐ ever, support a data presentation. Azure | 9
- 32. Data Presentation Layer In software architecture and system design, a presentation layer, sometimes called a servicing layer, refers to a component or set of features that facilitate the interaction between different parts of a system. It is an intermediary that handles various communication-related tasks, data processing, and exposure to functionality. The pri‐ mary purpose of a servicing layer is to provide a unified and standardized interface for different clients, allowing them to access services and resources consistently. This layer helps abstract the underlying complexity of the system’s components and pro‐ vides a cohesive interface for users and other systems to interact with. By acting as a bridge between clients (consumers) and core functionalities, the servicing layer abstracts technical intricacies, allowing users to interact without needing to understand the inner workings. It manages the exchange of data between these entities, ensuring smooth communication through various protocols and transformations. It’s, in effect, a façade pattern atop complex data processing. A servicing layer provides security for data, APIs for serving data, pushes for data subscribers, data caches for download, and many other things that data consumers use. This layer also addresses cross-cutting concerns such as logging, monitoring, and error handling, which are essential to maintaining a stable and well-monitored envi‐ ronment. The servicing layer can be designed for scalability, enabling the system to handle increased traffic and demand without compromising performance. Its role is vital in creating a simple, secure, and efficient bridge between the data and consum‐ ers. Chapter 11 dives into the servicing layer in great detail, where it talks about all the nuances of each of these styles of data exposure. No one service in Azure provides a data presentation layer; however, the services used for this are typically forward-facing. For push-style deliveries, you may use some low- level protocols like FTP. Still, push-style for web apps and data integrations can be accomplished through webhooks, Azure Web PubSub, and Azure SignalR integration. Other integrations also may use APIs like those exposed through Azure Functions or an API service or with a tool like Azure Data API Builder. All these tools and integra‐ tion points provide a way for consumers to get the data they need for their purposes. Data Consumers In the expansive landscape of IoT, data consumers encompass a diverse set of entities and systems that find value in the data generated by IoT devices. These consumers are instrumental in translating data into meaningful insights and actionable informa‐ tion. Several distinct categories of data consumers emerge within this ecosystem: 10 | Chapter 1: The IoT Landscape
- 33. • Some consumers demand immediate results from real-time data, acting swiftly on the incoming information to make instantaneous decisions and trigger responses. These real-time analytics and decision systems are essential for pre‐ dictive maintenance, security breach detection, and event-driven operations. These systems consume APIs and push-style data integrations. • Operational dashboards and monitoring tools, like Power BI, present real-time data and reporting. These tools provide an easy-to-comprehend snapshot of ongoing operations and performance metrics, supporting operational teams in managing resources effectively and identifying anomalies in a timely manner. They facilitate informed choices and strategy formulation by delivering insights from real-time, historical, and aggregated IoT data. • Dataset consumers, such as predictive analytics and machine learning models, tap into historical and real-time IoT data, deducing patterns, trends, and insights that serve as a foundation for future predictions. These consumers are vital in optimizing processes, anticipating maintenance requirements, and guiding stra‐ tegic planning. • External applications and APIs utilize IoT data for integration into other software systems, applications, or services. They range from third-party applications enhancing functionality to data marketplaces offering access to specific IoT data‐ sets. They are likely to consume APIs and real-time data feeds. Industry apps and consumer apps empower users with insights into their data. These applications allow users to make informed choices and remotely manage their IoT devices. Each type of data consumer possesses unique demands and purposes. A robust data architecture and a well-crafted servicing layer must be in place to effectively serve this diverse array. This ensures that the data generated by IoT devices is accessible, rele‐ vant, and available in formats tailored to the needs of each consumer, enabling them to extract maximum value from the IoT ecosystem. Chapter 12 dives into this topic and gives you some practical examples of each. Monitoring, Logging, and Security The IoT Landscape encompasses different domains within the context of IoT, but monitoring, logging, and security are cross-cutting concerns on the Azure IoT Land‐ scape. They’re like fertilizer, water, and sunlight that make crops grow well. In the context of Microsoft Azure and IoT deployments on the Azure platform, mon‐ itoring, logging, and security are three crucial aspects to consider. These aspects are fundamental to ensuring the reliability, visibility, and protection of IoT systems and data. You not only monitor your devices. You need to monitor your software systems and Azure itself. Monitoring, Logging, and Security | 11
- 34. Monitoring in an Azure IoT solution refers to the ongoing observation and measure‐ ment of system components, device behavior, and data flows. It involves using tools and services to track the performance, availability, and health of IoT resources. IoT Hub provides the data, but tools like Azure Monitor provide various monitoring capabilities, including real-time telemetry data visualization. Azure provides Defender for IoT, Azure Security Center, and Azure Sentinel to gain insights into device activities, detect anomalies, and ensure that your IoT solution functions as expected. By monitoring IoT devices and infrastructure, you can proactively identify issues, optimize performance, and provide a smooth user experience. Monitoring is based on logging, which systematically records events, activities, and interactions within an IoT solution. Azure IoT solutions can generate substantial data, and effective logging is essential for troubleshooting, auditing, and understanding sys‐ tem behavior. Azure provides Azure Log Analytics and Azure Monitor, which enables you to capture and store logs centrally. These logs can offer valuable insights into device behavior, application interactions, and system events, helping you diagnose problems and analyze historical data for improvements. Chapter 13 talks about how to monitor your Azure estate and how the available tooling captures and logs data. Security is a cross-cutting concern in IoT deployments, and Azure offers robust fea‐ tures to protect your IoT solution. Azure IoT provides secure device provisioning, identity management, and authentication using technologies like X.509 certificates or device keys. Role-based access control (RBAC) allows you to control access to resources based on user roles, ensuring that only authorized individuals can manage and interact with IoT components. As mentioned, Microsoft provides Azure Security Center and Sentinel for threat detection and protection capabilities, helping you iden‐ tify and address potential security vulnerabilities in your IoT environment. Chap‐ ter 14 enumerates different threats related to IoT devices and offers strategies to mitigate these threats. Monitoring, logging, and security are integral to Azure IoT deployments. Monitoring empowers you to oversee the health and performance of your solution in real time, logging enables effective troubleshooting and historical analysis, and security meas‐ ures safeguard your IoT environment from unauthorized access and potential threats. These aspects collectively contribute to a reliable, well-managed, and secure IoT eco‐ system on the Azure platform. Conclusion The last few pages have given you a distilled version of what this book entails. Each component illuminated has its role and significance within the IoT Landscape. There’s much to consider, from devices to data pathways and edge computing to data servicing. Welcome to the forefront of innovation, where Azure IoT offerings wait and stand as your gateway to possibilities. 12 | Chapter 1: The IoT Landscape
- 35. CHAPTER 2 Azure-Centric IoT Devices Chapter 1 provided the view of the world of IoT solutions from 40,000 feet. By now, you have the gist of what architecting an IoT solution on Azure is all about. But hold tight, because the real adventure is just getting started. In Chapter 2, you come down from the bird’s-eye view straight into the nitty-gritty world of devices. Before you get too carried away, though, understand there’s no way to cram every detail about every device into a single chapter. The focus here is on Azure-centric devices, where hard‐ ware, supporting software, and cloud services all shake hands. Speaking of devices, my first internet-connected device was an IP camera. I wasn’t thinking “IoT” back then—I just wanted a camera that let me peek at video streams online for home security. It was just another device, right? Fast forward, now I have a smart thermostat, Chromecast, internet-linked home alarm, robot vacuum, you name it. It wasn’t long before my house was a full-blown IoT hub alongside my trusty phones and gadgets. But here’s the kicker. As I kept adding gear, my inner network security guy started ringing alarm bells. I eyed those IoT devices with a bit of caution. They came with mysteries and challenges. Heck, I even set up a separate access point for my IoT squad, just to keep them at arm’s length from my computers and tablets. While my experience is certainly my own, I do not think it is all that different from most folks. There are countless IoT devices and endless applications for IoT. Even some of my less tech-savvy friends have more devices than I do. Everything that you might install as part of a “smart home,” such as light bulbs, thermostats, and micro‐ waves, is an IoT device. Entertainment devices, like Alexa speakers and Chromecasts, are IoT devices. Beyond the walls of the home, IoT devices are found in stores, hospi‐ tals, as part of the electrical grid, in factories, in trains, in planes, in ships, and practi‐ cally anywhere else you might want to install a device to monitor and interact with an environment while connected to the internet. They are everywhere. They have inva‐ ded our lives and world to the point that more IoT devices exist on the planet than 13
- 36. 1 Lionel Sujay Vailshery, “Number of Internet of Things (IoT) connected devices worldwide from 2019 to 2023, with forecasts from 2022 to 2030”, Statista. people. There are now 13 billion devices, and that number is projected to grow to over 29 billion by 2030.1 The IoT device invasion was silent. Phones, on the other hand, were loud. Apple made sure everyone knew that the iPhone was not like the trusty old Nokia or Razr phones of the early 2000s. Marketing hype made sure everyone knew that the iPhone and its competitors were the most revolutionary thing since the invention of the wheel. Contrast that with IoT. Most people, like me, probably bought their first IoT devices without even realizing they could be categorized as IoT. Folks are probably most familiar with devices as part of IoT, but they’re really only a small part of the picture. On the IoT Landscape, devices occupy a small part of an IoT solution, as seen in Figure 2-1. But while an IoT solution might be thought of as a single system, devices are prolific with potentially millions of devices as part of a device eco‐ system. So much of the work for IoT happens outside the immediate context of devices. Much of the rest of this book talks about everything that’s to the right of devices in the IoT Landscape. You can generally think of these domains as being in “the cloud.” Figure 2-1. Devices in the IoT Landscape IoT Devices as the Nexus of Three Domains Zooming in on devices reveals more nuances, however, and that they aren’t merely big buckets of things. These devices have a life of their own. Fundamentally though, you can think of IoT devices as the nexus of three different domains: hardware, soft‐ ware, and cloud, as seen in Figure 2-2. 14 | Chapter 2: Azure-Centric IoT Devices
- 37. Figure 2-2. IoT devices as the nexus of hardware, software, and cloud All three of these parts of IoT interact to create network-connected devices as part of a comprehensive IoT device. Hardware Hardware in the context of IoT is pretty straightforward: it composes the physical “thing.” IoT device hardware varies depending on the context in which it is used. It can be as simple as a microphone attached to a WiFi network or something much more sophisticated, such as a piece of medical imaging equipment that does 3D brain scans. At the core of every IoT device is some kind of compute capacity—a composition of a processor, memory, and storage. This is the central “brain” that translates interactions with an analog world into digital signals that can be processed, packaged, and trans‐ mitted over a network and reverses that process for signals coming off a network into something that the device does. The level of sophistication of the compute hardware, like the devices they control, can vary from something rather simple to something incredibly complex. IoT devices fit into one of two large categories, although the line between these is a bit blurry. One category contains the most basic compute hardware. These are con‐ strained devices, with extremely limited compute capacity, available resources, and RAM. This sort of compute is typically a tiny computer on a small board and is referred to as a microcontroller unit (MCU). These devices have benefits such as low costs and minimal power draw measured in single-digit milliwatts or even micro‐ watts. MCUs can run an operating system such as a real-time operating system (dis‐ cussed in “Azure IoT Device SDKs” on page 29) but an operating system is not required. Instead, the device can run embedded code, typically written in a low-level language like C, that runs right on top of the MCU. MCUs have all kinds of applica‐ tions, such as home appliances, sensors, and tracking tags, among many other things. Constrained devices compose the majority of IoT devices. IoT Devices as the Nexus of Three Domains | 15
- 38. Outside of constrained devices, the second category, unconstrained devices, contains devices more recognizable as what one might consider a computer. These devices, like the Raspberry Pi, are also small, but they have more capable hardware that can drive displays, run a full-blown operating system like Linux, run multiple applications simultaneously, and perform more CPU-intensive operations that make the device more intelligent. Some have compute capacity more or less equivalent to desktop and laptop computers, and they are more capable of performing tasks. In some cases, these IoT devices are PCs embedded into a machine. Some of these devices may even include specialty hardware, such as dedicated circuitry for processing AI models. Unconstrained devices also include devices containing server-grade compute. These devices are typically edge devices that interact with other IoT devices. They provide services to other devices that might typically run in the cloud closer to the devices they serve. Software Software in the context of IoT provides the instruction set for how a device interacts with its environment. In its simplest form on IoT devices, the software simply reads data, packages it up, and sends it to the cloud. More sophisticated devices, however, can do more. Some of these devices interact physically with the environment. ATMs, for instance, receive and dispense money. Self-checkout kiosks scan groceries and accept payments. All these applications, from the simple to the sophisticated, require software at some level to collect data, transmit data, receive data, and react to events. Cloud Technically, the cloud is hardware and software, too. But from the perspective of an IoT device, the cloud is just some external entity that the device sends data to and receives data from. The device understands the inputs and sends outputs that the cloud responds to. The interactions between devices and cloud services are one of the defining characteristics of IoT devices. Sometimes, a high concentration of devices in a building or area warrants the use of something collocated with the devices to man‐ age them, assist with data-related tasks, or help get them connected to the internet. Such tasks are managed by edge computing. For the discussion here, all cloud-side services and edge computing are lumped into this category, but these will be picked apart in later chapters. The cloud components and services that are pertinent to a specific kind of device are discussed as part of the device’s cloud offerings. IoT devices are the nexus of hardware, software, and cloud services. Microsoft, being a tech company that makes hardware, software, and cloud services, has much to offer in this space. It is sometimes good to look at where you have come from to 16 | Chapter 2: Azure-Centric IoT Devices
- 39. understand where you are going. From here, the journey continues with a look at Microsoft as a company, and how that positions them as an IoT solutions provider. IoT Devices and Microsoft From a technical perspective, this book concerns itself with Microsoft Azure for cloud services. The chapters discussing cloud-side components will talk about Azure specifically. For this reason, it is a good idea to get acquainted with Microsoft and what positions Microsoft as a leader in IoT solutions and innovations. When most people think of Microsoft, they do not think of a hardware company. Microsoft’s success as a software company most certainly overshadows its attempts in hardware, but a closer look reveals success in hardware, too. Microsoft the Software Company Microsoft was one of the first companies solely focused on software in a time when most people thought of software as something that came with the computer, not as a thing in itself. By the 1990s Microsoft solidified its position as a mainstay in enter‐ prise software with products like Windows Server, SQL Server, and Great Plains. By the turn of the century, Microsoft was venturing into entertainment with games and media. Microsoft’s expansion into practically every niche in the software market posi‐ tioned them well to extend many of these offerings into the cloud. Microsoft’s success is in part due to how it has tried to make tools and products that developers want to use. BASIC, one of its first commercial products, was a program‐ ming language designed to make writing programs for computers simpler. Microsoft built on BASIC over time with an IDE for GW BASIC and QBasic that the company bundled with early versions of DOS. This IDE, while text-based, incorporated many of the features and motifs still found even in modern IDEs. For Windows, Microsoft created Visual Studio, a product that they continued to evolve and expand over 25 years after its original release. Visual Studio 2002 marked the first version of the product for use with the .NET Framework, which became the backbone for desktop and web development on the Microsoft platform. .NET’s rich ecosystem has been widely adopted by enterprises and adapted to numerous other platforms beyond its original inception to include mobile, Linux, and IoT. Microsoft focused on a closed-source model for most of its existence. However, when cloud computing arose, Microsoft also started pivoting toward open source by offer‐ ing Linux as a key Azure operating system, opening up .NET, and providing numer‐ ous open source tools and SDKs for developers to use with Azure. Microsoft also expanded beyond .NET with its SDKs to include many popular open source lan‐ guages like Python, JavaScript via Node.js, and Java, and many of these include sup‐ port for Azure’s IoT offerings. IoT Devices and Microsoft | 17
- 40. Microsoft’s commitment to developers and its commitment to software is in the com‐ pany’s DNA. As a developer, I have worked with numerous platforms and tools over my career, but Microsoft tools and technologies have always been some of the best and most used tools in my development toolbox. Azure’s IoT tools are among these, and they integrate well as part of the larger Microsoft development ecosystem. Microsoft as a Hardware Company While Microsoft is primarily known as a software company, Microsoft has a long his‐ tory as a hardware company, too. This history dates back to 1980, merely five years after the company started. Microsoft developed an expansion card for the Apple II called the SoftCard, which effectively turned an Apple II into a platform capable of running the CP/M operating system, an OS that Microsoft had invested heavily into for apps and software development. The SoftCard is a forgotten success. Microsoft soon turned its attention to operating systems, and produced DOS, which became the dominant PC platform in the world by the 1990s. Microsoft did make a few peripherals for PCs, like mice and keyboards in the 1980s and speaker sets and other PC peripherals in the 1990s, but they did not branch out beyond the PC peripherals until at least the late 1990s with phone systems. The big‐ gest change happened, however, when the company pivoted from hardware that interacted with a PC to hardware that was another platform altogether. These plat‐ forms not only ran Microsoft software but also invited developers and companies to include their own apps and software. In the early 2000s, the company made a foray into devices in the gaming space with the Xbox and in the mobile space. For mobile, Microsoft produced a version of Windows for handheld computers and the Kin mobile phone. Microsoft attempted to create a media player to compete with the iPod called Zune. By the end of the decade, Microsoft had a fledgling mobile platform with Windows Phone and even made in-house Lumia line handsets for the platform. In the 2010s Microsoft abandoned its mobile and media player ambitions but created its own line of PCs with surface tablets and laptops that is still going strong. Xbox con‐ tinues to be a success. Microsoft is developing the HoloLens platform as well. Not everything Microsoft created for hardware and software was a success. Despite this, the company has found a sweet spot with many of its products. Even with Microsoft’s long history with hardware, these platforms inevitably always came with a set of tools and SDKs to encourage developers to build apps, games, and solutions for the platforms. Moreover, Microsoft has maintained long-term working relationships with a network of original equipment manufacturers (OEMs) that make PCs and hardware used by PCs. Whether Microsoft is designing and building hardware itself or working with others, the company has historically created a ubiquitous set of tools and platforms. 18 | Chapter 2: Azure-Centric IoT Devices
- 41. Microsoft’s pivot to the cloud with Azure has positioned the company to consider other opportunities to enable different kinds of hardware to leverage its platform, including IoT workloads. To this end, Microsoft has created numerous, parallel streams that empower and enable organizations to leverage the cloud and develop solutions with Microsoft technologies. Some of these solutions are hardware- oriented, while others are more software-oriented. As with most of Microsoft’s other hardware platforms, each of these comes with ways to create solutions that both lev‐ erage the IoT device platform and the supporting cloud-side components. Microsoft as a Cloud Company Microsoft’s cloud journey started in the late 2000s under the code name Project Red Dog, which later became Azure. Released in 2008, Azure initially had very few offer‐ ings. It could run web apps, host SQL databases, and run virtual machines. Since that time, it has exploded to include a swath of services for almost every compute need imaginable. Microsoft has expanded its Azure operations all over the globe with regional data centers on every continent except Antarctica. The suite of tools, tech‐ nologies, and offerings is as broad as it is deep, and the platform has become one of Microsoft’s largest revenue producers. While Azure is a focus of this book, Microsoft’s cloud also supports the Dynamics 365 platform and the Office 365 platform. Dynamics integrates numerous enterprise resource planning (ERP) features and applications to create a system for managing business resources and processes. Office 365 is Microsoft’s productivity suite evolved in the cloud that offers tools for managing business data, collaboration, and communication. Like Office 365 and Dynamics 365, Azure owes part of its success to how well it inte‐ grates with existing enterprise systems. Azure, since its inception, has provided tools and technology to create hybrid cloud computing environments that enable organiza‐ tions to share data and compute between cloud resources on Azure and on premises. Many of the technologies offer ways to extend on-premises data centers into the cloud and also bring cloud services back to on premises. This bidirectional integration is cru‐ cial for many IoT workloads that need Azure-style compute on the edge of networks. IoT on Azure: Microsoft’s Combination of Software, Hardware, and Cloud IoT on Azure brings together three things that Microsoft has been successful at in the past: software, hardware, and cloud, the cornerstones for building a successful IoT device. Microsoft has created a number of IoT-specific services in the Azure cloud and Azure-centric hardware platforms for interacting with those services. Addition‐ ally, Microsoft provides tools and software built on many of the successful Microsoft offerings that allow developers to bring solutions to life. IoT Devices and Microsoft | 19
- 42. Now that you understand where Microsoft has been, it’s time to take the plunge and explore exactly what devices Microsoft has to offer, starting with Azure Sphere. Azure Sphere It is hard to talk about Azure Sphere as a device—it’s so much more than that. Azure Sphere is really an entire ecosystem designed for Azure and IoT. It contains its own specialized hardware, device-specific software, and a set of cloud services that sup‐ port it. The nexus in Figure 2-3 shows how Azure Sphere combines cloud services to support hardware tailor-made for Azure integrations. The value added by Azure Sphere comes from how it automates large portions of device lifecycle management with the services and adds security features through cloud services. Figure 2-3. Azure Sphere IoT nexus Azure Sphere Hardware “Devices as a service” are probably not possible, but Azure Sphere is about as close to that as you can get. Azure Sphere is both a security-hardened device combined with security and management services on Azure. Azure Sphere boards vary depending on the manufacturer, but they are, generally speaking, a system on a chip (SoC) with an ARM Cortex 7 CPU and modest amounts of RAM and storage. The boards all run the same Azure Sphere operating system and support the same kind of apps regard‐ less of the board manufacturer. Although Azure Sphere runs a custom Linux OS, for practical purposes the device is a constrained device. Azure Sphere Software The operating system for an Azure Sphere device is a Microsoft-maintained Linux distribution that creates a secure container for user applications. The operating sys‐ tem provides a set of APIs for interacting with the device’s hardware while protecting the internals of the operating system and the device from user applications. This strict 20 | Chapter 2: Azure-Centric IoT Devices
- 43. separation of concerns exists to ensure that the code running in the container does not harm the device or access anything the device connects to beyond what is exposed through the APIs. Azure Sphere provides development tools for the C language with specific extensions for Azure Sphere in Visual Studio Code and Visual Studio. It supports two different kinds of applications for different kinds of programming: high-level apps and real- time apps. High-level apps behave more like an application where users might expect an application wait state and manage a state machine, even if the application is per‐ forming tasks in the background. A real-time application does not reflect a wait mode but rather continuously runs and responds to events as they happen. Real-time apps run closer to the metal than high-level apps. These applications require closer man‐ agement of the application’s logic, resources, and other internals of the application’s logic flow, while high-level applications take care of many of those details for devel‐ opers. Real-time apps can also run via a real-time operating system, which provides a framework for real-time development for applications. Both high-level and real-time application paradigms require that developers familiar‐ ize themselves with the pros and cons of each pattern, and select the one that best suits deployment needs. Azure Sphere Cloud Services The security services for Azure Sphere devices maintain the devices through cloud services, including regular security patches for the operating system, device authenti‐ cation, and device attestation. Users are still responsible for maintaining the code in the container, but much of the work that typically falls to IoT device makers is han‐ dled through Microsoft’s IoT services. Users pay a license for these services on a per- device basis. What’s It For? Azure Sphere is a general-purpose platform for IoT workloads. It provides value- added services for devices with stringent security requirements, a desire to outsource some of the device lifecycle management, or both. Like the Azure MXChip, which I’ll discuss in the next section, the hardware for Azure Sphere is modest and therefore will not work for devices that require a rich user experience. Azure Sphere, however, can be a part of a more complex system that uses Azure Sphere for its connectivity and integration capabilities. Beyond the Azure Sphere services, Azure Sphere devices can interact with all other Azure IoT Services. Azure Sphere | 21
- 44. Azure Sphere has a slight learning curve for adoption. It uses the C language for development, and the app paradigms and tools are provided by Microsoft for devel‐ opers. Likewise, Azure Sphere is more of a closed ecosystem that is coupled with Microsoft services. The development community for Azure Sphere is smaller than other platforms but still vibrant enough to find help and qualified individuals. What Makes It Unique? Azure Sphere takes care of much of the device lifecycle management through its offerings for Azure Sphere by patching and updating the devices. This is not a com‐ plete solution, but it does substantially mitigate the device lifecycle management bur‐ den. Azure Sphere IoT devices are for teams that want to focus more on bringing a device to life but do not have the human resources to develop and maintain the oper‐ ating system, authentication, and attestation for IoT devices. Azure MXChip The Azure MXChip is a small, Arduino-compatible, IoT-centric device especially designed to work with Azure solutions. The MXChip was co-developed by Microsoft and the company MXChip for a wide array of IoT applications. The value MXChip brings is an Azure-focused Arduino board with special software and hardware for Azure, illustrated by the nexus shown in Figure 2-4. Figure 2-4. MXChip IoT nexus MXChip Hardware The MXChip comes with many sensors already on board, including a humidity and temperature sensor, a pressure sensor, a magnetometer, a motion sensor, and a 28-pin general-purpose input/output (GPIO). The small board has a 128×64 OLED display as well as an RGB LED, a user LED, and another system LED to show connectivity to WiFi (also built-in) and Azure. The device also has two built-in user-programmable 22 | Chapter 2: Azure-Centric IoT Devices
- 45. buttons. While the chip boasts a wide array of built-in hardware, its CPU and storage are rather modest, with only a 100 Mhz ARM CPU, 256 K of RAM, and 3 MB of stor‐ age that is split between user storage and the device’s storage. Like all Arduino devi‐ ces, this is a constrained device. MXChip Software The MXChip itself is compatible with Arduino SDKs and tools. Arduino is an open- source platform that allows anyone to manufacture Arduino chips on the general public license (GPL) that governs the chip. It has a large, vibrant community that makes finding developers and help straightforward for users of the platform. These factors can help accelerate development. The MXChip emulator pictured in Figure 2-5 offers an environment for developing solutions for the MXChip without the need to install anything in a local dev environ‐ ment. Code runs on the emulator, which can also interact with Azure services. Figure 2-5. The MXChip online emulator for development work The MXChip requires developers to use C. For developers used to working in lan‐ guages like C#, Java, or Node.js, C can be a challenge because many of the features Azure MXChip | 23
- 46. built into C#, Java, and Node.js are not available in C. For instance, developers are required to manage the memory they use more explicitly rather than relying on something like a garbage collector. MXChip Cloud Services There are no unique services for MXChip like Azure Sphere beyond the standard Azure IoT services discussed in Chapter 1. MXChip, however, makes connecting to Azure services easier through the provided classes and built-in network support. What’s It For? The MXChip board is intended to provide a way to quickly prototype devices for use with Azure IoT solutions. The devices using the MXChip can be productized using Arduino-compatible devices. MXChip does provide some user controls and a small display, but the kind of solutions for the MXChip are not solutions that demand much compute power. The device is principally designed to create a solution that, once created, will remain on the device without much human intervention. These sol‐ utions will basically read data from whatever sensor is needed from the built-in sen‐ sors or sensors attached via GPIO, bundle the sensor data in a message, then send that message to solutions on Azure. The device can receive messages from the cloud as well either as twinning changes or direct messages. See Chapter 3 for how to update twinning data and send messages from the cloud to the device. One of the main advantages of the Azure MXChip is its built-in support for use with Azure. The chip’s SDKs are written for C and offer many prebuilt libraries for inclu‐ sion in the C project. Projects for MXChip can use Visual Studio or Azure’s own MXChip emulator. Likewise, because of the device’s limited compute, MXChip will not be able to do much computing on the device. Data will need to be sent to the Azure cloud or an IoT Edge device with more compute capacity to handle the heavier compute needs. Edge computing and IoT Edge are covered in more depth in Chapter 6. What Makes It Unique? While the MXChip is not particularly unique, its combination of Arduino-compatible compute, built-in sensors, and connectivity to networks and Azure accelerates the development of hardware solutions. 24 | Chapter 2: Azure-Centric IoT Devices
- 47. Kinect Strictly speaking, Kinect is not an Azure-centric IoT device like Azure Sphere or MXChip, but it is worth mentioning because it is intended for many IoT-based appli‐ cations. Its nexus combines PC class hardware with the Kinect device, specialized SDKs, and Azure Services for speech and vision (Figure 2-6). Historically, Kinect was used for entertainment purposes by creating augmented reality for games. The use cases of Kinect, however, have been expanded beyond games to include any number of applications that can take advantage of augmented reality. Figure 2-6. Kinect IoT nexus Kinect Hardware Azure Kinect contains a suite of AI-assisted sensors that capture video, sound, and motion. These applications collect data that can be processed by a connected PC or sent to the cloud for processing on Azure. Kinect itself needs a computer of some kind running Windows or Linux to work. The camera on the device is coupled with a depth-of-field camera to enable multidimensional applications. The microphone array offers multipoint audio that enables the detection of directional sounds and more accurate filtering and processing than a mono or stereo microphone. Kinect Software Kinect needs a PC to work, so any application running on Kinect connects to the devices from the PC using USB. Kinect has two primary SDKs available for applica‐ tions: a more general-purpose sensor SDK and a body-tracking SDK. The sensor SDK gives applications access to the camera, microphones, and motion sensors on the devices. The body tracking SDK tracks movement using the camera and depth- of-field camera on the device. Kinect | 25
- 48. Kinect Cloud Services Azure does not provide any services unique to Kinect, but Kinect can integrate with Azure Speech Services and Azure Vision Services. Speech services can render recor‐ ded speech to text or translate the speech. Vision services include object recognition, optical character recognition (OCR) for extracting text from video, and spatial analy‐ sis that extracts multidimensional data from the capture of two-dimensional images. Speech and Vision Services from Azure can run on Azure or be containerized and run on an Azure IoT Edge deployment. Azure IoT Edge can either run on the same computer connected to the Kinect hardware or another computer. See Chapter 6 for more information about IoT Edge. What’s It For? Kinect as a device is principally for applications that need more than a basic camera. It has a body detection API, making it useful as a human-interface device for things like exercise equipment. The advanced camera and microphone array also give the devices more special awareness, so they can detect movement and sound from differ‐ ent angles. Combined with AI for sound and image processing, it has uses in indus‐ trial IoT for things like quality control on an assembly line. What Makes It Unique? Kinect provides an accelerator for applications that want to create intelligent vision and audio applications. The hardware and SDKs combined with cloud services pro‐ vide a robust platform for creating augmented reality applications or applying the sensors for purposes like logistics or medical applications. Device integrators can use Kinect with a PC platform to create many different kinds of applications. Windows for IoT Windows for IoT is a special SKU for Windows intended for IoT workloads. It creates a nexus around the Windows operating system with its software, cloud, and sup‐ ported hardware for IoT applications (Figure 2-7). 26 | Chapter 2: Azure-Centric IoT Devices
- 49. Figure 2-7. Windows IoT nexus Windows IoT Software Windows IoT is a special SKU of Windows 10 and Windows 11. It is the spiritual suc‐ cessor to the venerable Windows Embedded operating system that was intended for applications that were not considered general-purpose computers. Organizations cannot purchase Windows IoT and install it; rather it is available only to original equipment manufacturers, and it is sold and distributed with the device it runs on. Early versions of Windows IoT were otherwise identical to enterprise versions of Windows, but more recent versions contain changes to reduce storage requirements for the operating system. More changes in the future may happen. Windows IoT offers a few different SKUs, depending on the need. Windows IoT Core Windows IoT Core brings Windows to ARM-based compute, such as Raspberry Pis, but can still run on x86 hardware. The OS provides a minimal Windows experience on more modest hardware. ARM CPUs are not capable of running code written for x86 CPUs without an emulation layer. Windows apps written using Windows UWP, however, can run on Windows IoT Core. IoT Core allows only one app at a time to run. The cloud service offerings for IoT Core are more limited, and it cannot be domain-joined to a traditional Active Directory domain. Microsoft last provided downloadable images from IoT Core in 2018, but at the time of this writing, images integrated with Azure will continue to receive updates to the oper‐ ating system under the Long-Term Servicing Channel (LTSC) until 2029. Windows IoT Core can still be configured and installed on flash media using the Windows IoT Core Dashboard. The Dashboard application downloads images and writes them to SD cards for use with dev boards like Raspberry Pis or NXP dev boards from Intel. Windows for IoT | 27
- 50. Windows 11 has been ported to ARM processors, but as of this writing, there does not appear to be any ARM-based “core” version of the OS intended for IoT usage. Windows IoT Enterprise Windows IoT Enterprise is available in Windows 10 and Windows 11 versions. It supports both x86 and ARM-based architectures, depending on the SKU. ARM-based support limits the kinds of apps that can run on the devices to UWP applications, but the platform is not as limited as IoT Core in the number of applications it can run simultaneously. Likewise, these devices can be domain-joined and support other stan‐ dard enterprise features not available to IoT Core. Otherwise, Windows IoT Enterprise on x86 hardware is functionally identical to a standard Windows SKU. It supports the same hardware models, driver models, appli‐ cation models, and management models. It integrates with existing Windows envi‐ ronments, such as Windows domains, and when becoming part of a mobile device management solution. Windows Server IoT Like Windows IoT Enterprise, Windows Server IoT 2019 and 2022 provide similar features as the general-purpose versions of Windows Server. These editions are avail‐ able only to OEMs for use with devices that ship with Server preinstalled. The intent of Windows Server IoT is to provide services for edge computing for IoT devices. It integrates with many cloud offerings, such as Azure Arc, to create hybrid cloud solutions for IoT devices. For more about edge computing for IoT, see Chapter 6. Windows for IoT Hardware Windows IoT does not come with any prescribed hardware, so therefore it is truly “bring your own hardware.” The intended use cases for Windows IoT, however, are applications that need a richer user experience not possible with more modest hard‐ ware, such as kiosks, rich screen-based user interfaces, and monitoring. Some exam‐ ples include point-of-sale systems, medical imaging equipment, and infotainment solutions. None of these are intended as general-use computers and some attach specialty hardware for their environment, such as barcode scanners and payment terminals. 28 | Chapter 2: Azure-Centric IoT Devices
- 51. Windows for IoT Cloud Services Because Windows IoT Enterprise is fundamentally Windows, it can take advantage of solutions to manage Windows deployments on and off Azure. Windows IoT can authenticate using Azure AD, integrate with an existing Windows domain, and use any number of other services that manage Windows clients. Windows IoT Core comes with a tailored set of services for managing Windows 10 IoT Core devices on Azure with Windows 10 IoT Core Services. These services pro‐ vide an Azure-based control panel and integrated services that update the device’s operating system. What’s It For? Windows IoT operating systems are intended to create more powerful IoT devices than what is possible with more constrained devices like Azure Sphere and Azure MXChip. Compute resources for the platform are similar to those found in desktops, laptops, and servers. In many cases, devices running Windows IoT include common PC hardware but will have a specific purpose in mind. One of the canonical use cases for Windows IoT would be something like a point-of-sale system or a self-checkout kiosk. These have a single-app user experience with attachments, like scales, scanners, and payment terminals. The user drives the device through a touch screen. Windows as an operating system provides a foundation for connecting devices to the central compute through one of its many supported hardware interfaces. Apps bring these devices together to create complete solutions. What Makes It Unique? Windows IoT brings together the Windows ecosystem with IoT devices and supports these devices with all the same services that support standard Windows deployments. In some cases, such as Windows 10 IoT Core Services, Microsoft provides specialized services to manage devices at scale. Azure IoT Device SDKs Whether or not you use hardware from Microsoft, if you want to use Azure resources, you will need a way to talk to those services, and this is orchestrated by software. While it is possible to manually implement everything against Microsoft’s APIs, Microsoft has gone to great lengths to ensure that the APIs have coverage with SDKs, which makes interacting with the APIs much simpler than trying to write everything manually. The support they provide creates the nexus (Figure 2-8) between cloud services and hardware where they live. Azure IoT Device SDKs | 29
- 52. Figure 2-8. Nexus for IoT SDKs Microsoft SDKs for Azure IoT services work by providing wrappers around many of the common protocols used by IoT devices, such as HTTP, MQTT, and AMQP (see Chapter 5 on device messaging to see the pros and cons of each of these protocols). The wrappers provide a common interface that abstracts the underlying protocol from the implementation. This allows the protocols to seamlessly swap with one another should one protocol not be feasible for the particular need. Microsoft’s device SDKs provide coverage for the common tasks that devices perform against the Microsoft services during a device’s lifecycle, covered in Chapter 4: Device provisioning Device provisioning SDKs interact with Azure Device Provisioning Service, which brokers a device’s first-time connection to the cloud by generating creden‐ tials and assigning the device to an IoT Hub. Securing device communication The SDKs provide inflight message encryption and a client that brokers authenti‐ cation between a device and an IoT Hub. Device updates Device updates are provided through Azure IoT Hub. The SDK checks for updates to a device and brokers applying the update from the cloud to the device. Messaging to and from the cloud Messaging services are the backbone of any IoT platform. The SDKs provide robust support for sending and receiving telemetry, sending files, receiving com‐ mands, and sending events. 30 | Chapter 2: Azure-Centric IoT Devices
- 53. Device twinning Part of maintaining a fleet of devices includes maintaining a digital twin of a device’s configuration in the cloud. The SDKs provide seamless integration to notify the cloud when configurations change and also allows the cloud to push down changes. The Azure Device SDKs principally aim at providing clients for connecting to Azure IoT services. Not all of these services have to be used, so there are different SDKs for each context, depending on the need. The SDKs mentioned here are generic SDKs, and therefore should not be confused with other SDKs that are more specific to devices and device-centric services, such as the Azure Sphere SDK or Kinect SDK. Moreover, the SDKs do not provide any assis‐ tance for interacting with hardware. These SDKs, as mentioned, are for working with the cloud services and will therefore need to be amended with device-specific SDKs or code to interact with the hardware. Supported Languages and Platforms Azure IoT supplies SDKs for .NET, Python, Node.js, Java, and C. The SDKs are pro‐ vided as prebuilt packages that can be readily downloaded and included with the package manager for the platform, such as Pip for Python, Nuget for .NET, or NPM for Node.js. The source code for the packages is all available as open source on Git‐ Hub if for some reason you need to build the SDKs yourself. Even if you do not use the SDK source code from GitHub, the GitHub repos are an excellent source for code samples on how to implement the SDKs. Real-Time Operating System (RTOS) The standard device SDKs are not for constrained devices such as MXChip or Azure Sphere. For these, Microsoft provides an RTOS with special libraries for embedded systems. An RTOS is essentially a tiny operating system for embedded microcontrol‐ lers. It provides features similar to those of a more robust operating system but at a much more essential level for the device it runs on. An RTOS typically uses only 1 to 2 kilobytes of memory. An RTOS can manage threading, CPU cycles, memory, and access to I/O stacks like USB or TCP. An RTOS is designed to accelerate development and make managing code simpler for constrained devices. For more constrained devices, Microsoft provides the Azure RTOS. The Azure RTOS works with MXChip, Azure Sphere, and other approved devices. Beyond the com‐ mon tasks like multithreading, Azure RTOS provides connectivity to Azure IoT serv‐ ices. Additionally, Microsoft provides middleware for FreeRTOS and open source projects ported to a number of different microcontrollers. Azure IoT Device SDKs | 31
- 54. An RTOS is not necessary for embedded devices. For some applications, the C libra‐ ries for the embedded microcontroller are sufficient. When All Else Fails, Use the APIs If for some reason you are unable to use the SDKs, Microsoft does expose APIs that can be called through any of the supported protocols, and devices can use non- Microsoft implementations of clients to connect to these APIs, such as general- purpose MQTT clients or HTTP clients. Because the SDKs wrap much of the complexity, implementing these services without the SDKs will be a more substantial undertaking. All in all, using SDKs makes development easier. One way to learn the SDKs and apply them quickly in your context is to use a device simulator, which I’ll talk about in Chapter 3. Summary Throughout this chapter, you’ve learned about all kinds of things related to devices. The chapter started by looking at devices and ecosystems from Microsoft, including Azure Sphere, MXChip, and Kinect. Beyond these physical devices, the chapter cov‐ ered other Microsoft IoT ecosystems like Windows IoT and SDKs. Table 2-1 contains a summary. Table 2-1. A summary of Azure-centric IoT device solutions Solution Hardware Software Cloud Purpose Azure Sphere Azure Sphere devices Azure Sphere OS with high- level apps, low-level apps, and RTOS implementation Azure Sphere services for security and OS maintenance Value added services for security and device lifecycle management MXChip Arduino-compatible sensor loaded dev board Arduino tools, Azure libraries for Azure services, MXChip Simulator Standard IoT services A general-purpose dev board for Azure-centric IoT workloads Kinect AI-assisted cameras and microphones connected to a Windows or Linux PC Kinect SDK for sensors and body motion, and other dev tools for applications run by the connected PC Standard IoT services Applications that can take advantage of the spatial sensors on the Kinect device Windows IoT PC-class hardware connected to IoT peripherals Standard Windows SDK and driver models for hardware Standard Azure IoT services, Azure Windows IoT Core services, Windows management services, Azure Arc IoT application for integration with a Windows ecosystem or applications that need a more robust UX 32 | Chapter 2: Azure-Centric IoT Devices
- 55. Random documents with unrelated content Scribd suggests to you:
- 59. The Project Gutenberg eBook of The Waterloo Campaign, 1815
- 60. This ebook is for the use of anyone anywhere in the United States and most other parts of the world at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this ebook or online at www.gutenberg.org. If you are not located in the United States, you will have to check the laws of the country where you are located before using this eBook. Title: The Waterloo Campaign, 1815 Author: William Siborne Author of introduction, etc.: Edward Arber Release date: November 11, 2018 [eBook #58268] Language: English Credits: Produced by Brian Coe, Graeme Mackreth and the Online Distributed Proofreading Team at http://www.pgdp.net (This file was produced from images generously made available by The Internet Archive) *** START OF THE PROJECT GUTENBERG EBOOK THE WATERLOO CAMPAIGN, 1815 ***
- 61. THE WATERLOO CAMPAIGN 1815 WILLIAM SIBORNE Captain, Half Pay, Unattached, Constructor of the Waterloo Model FIFTH EDITION WESTMINSTER ARCHIBALD CONSTABLE AND CO., LTD. 1900
- 62. PREFACE. BY common consent, this Work is regarded as the best comprehensive account in the English language of the Waterloo Campaign. Even those who differ from the Author upon particular points, most cordially admit the general accuracy and fulness of his History. It is charmingly written, is graphic yet precise, and abundantly witnesses to the Author's most strenuous endeavour to do justice to every one who took part in that great Conflict. This Work will henceforth be a household book amongst the Teutonic race; and all who read it will gain a very clear insight into the methods of Military Strategy as they were practised by the great Captains of that Age. It is impossible to repress one's admiration of the heroic bravery displayed in this brief Campaign: whether amongst the Allies at Quatre Bras and Waterloo, or by the Imperial Guard at Planchenoit, or by the Prussians at Ligny, Wavre, and Le Chesnay. The reader must be good enough to observe that a Prussian Brigade then equalled in numbers a French or an English Division. This Work has extended to such great length that one half of the Appendix (see pages 42 to 44) and nearly all the Notes have been, most unwillingly, omitted. Only those Foot Notes have been inserted which are absolutely essential to the Text. Room has however been found, at pages 798 to 826, for the Nominal Lists of Officers at Waterloo, &c. One would most earnestly wish that Wars may cease until the end of Time; but if that may not be, then may they be as bravely fought as was this War of Twenty Days: from the 15th June, when Napoleon crossed the Sambre; to the 4th July 1815, the day on which the Allies took possession of Paris. EDWARD ARBER.
- 63. Edgbaston, Birmingham. THE TITLE PAGE OF THE THIRD EDITION. TO THE QUEEN'S MOST EXCELLENT MAJESTY. Madam, IN graciously deigning to accept the Dedication of these pages, Your Majesty has afforded the greatest possible encouragement to my humble endeavours to record, with simplicity, impartiality, and truth, the incidents of an eventful War, resulting in a long enduring Peace; a War which shed a new and brighter lustre on the valour and discipline of the British Army, and once more called forth the consummate sagacity and far-extending prescience of that illustrious Chief, whom Your Majesty, with wise appreciation and a just pride, retains at its head. Earnestly hoping that the result of those endeavours may prove not altogether undeserving of Your Majesty's approbation,
- 64. I have the honour to be, With profound respect, Madam, Your Majesty's most humble And most devoted servant, WILLIAM SIBORNE, Captain Unattached. PREFACE TO THE THIRD EDITION. IN offering to the Public this Third Edition, I feel called upon to state, by way of explanation, in what respect it differs from the two former Editions. During the interval which has elapsed, I have not failed to avail myself of every opportunity to correct and improve any points which further investigation rendered desirable; and I have been much gratified in finding that the general plan and arrangement of the Work, together with the elucidation of the military operations, and the views of their tendency and effect, have been generally borne out and approved; and that, consequently, in these respects little alteration has been required. The exceptions, which consist principally in details, and amount in number to only four or five, have been rectified in this edition. They are chiefly the result of discussions which have appeared in the pages of the United Service Magazine; and relate to a portion of the proceedings of Sir Colin Halkett's and Sir Denis Pack's Brigades at Quatre Bras and Waterloo. Through the kindness of His Excellency the Prussian Ambassador, Chevalier Bunsen, and of the Prussian Generals von Canitz and von Krauseneck, and of Major Gerwien of the Prussian Head Quarters Staff; I have obtained additional interesting details connected with the
- 65. Prussian operations; more especially as regards the opening of the Campaign. A Dutch work published, apparently under authority, by Major Van Löben Sels, Aide de Camp to his Royal Highness Prince Frederick of the Netherlands, and entitled B dragen tot de Kr gsgeschiedenis van Napoleon Bonaparte, of which I was not previously in possession, has enabled me to give additional particulars respecting the movements and dispositions of the most advanced portion of the Dutch-Belgian troops, on the first advance of the Enemy; and also to explain particular circumstances and qualify some observations respecting those troops which appeared in former Editions. The Editor of an Article in The Quarterly Review, No. CLI., entitled Marmont, Siborne, and Alison, having, in his comments upon this Work, denied the accuracy of one or two important facts therein stated; I have, in notes at pages 57 and 152,[2] entered into more minute details, which explain the grounds that warrant me in adhering to the original statements. The observations made in the Preface of a Volume of "Murray's Home and Colonial Library," entitled The Story of Waterloo, and the palpable embodiment of the present Work into the pages of the latter, have been such as could scarcely fail to attract attention; and I have accordingly appended to this Edition, in a separate form, some remarks upon that publication.[3] Public opinion (if I may judge by the unanimous consent of the press) having so distinctly pronounced its acknowledgment of the value of my Work, as one of History; I could not disregard the conduct of a Writer, who, in the first place endeavours to depreciate that value, and then unblushingly makes the most ample and unlicensed use of it for his own purposes. W. SIBORNE. 18th June 1848.
- 66. FOOTNOTES: [2] Omitted in this Fourth Edition.—E.A. [3] Omitted in this Fourth Edition.—E.A. The Duke of Wellington. PREFACE TO THE SECOND EDITION. THE circumstance of the First Edition having been sold off within a very few days, combined with the highly favourable notices taken of the Work by professional as well as other critics, and, I may be permitted to add, the very flattering encomiums which have been pronounced upon it by so many who, from their position, are the most competent to form an opinion on its merits, cannot fail to afford proofs, the most satisfactory to the Public, and, at the same
- 67. time, the most gratifying to the Author, that, in the production of these Volumes,[4] upon a subject of such stirring national interest, neither the expectations of the former have been altogether disappointed, nor the labours of the latter bestowed in vain. The present Edition contains corrections on one or two points of trivial importance, to which my attention has been directed; and I shall be happy to receive further information from surviving Eye Witnesses who may discover any instances in which the facts related appear either inaccurately or insufficiently explained. W. SIBORNE. August 23rd, 1844. FOOTNOTES: [4] The First and Second Editions of this Work were each published in Two Volumes.—E.A. PREFACE. SOME years ago, when constructing a Model of the Field of Waterloo, at a particular period of the Battle; I found it necessary to make great exertions to procure that detailed information for which I had sought in vain in the already numerous published accounts of the military transactions of 1815. Anxious to ensure the rigorous accuracy of my work, I ventured to apply for information to nearly all the surviving Eye Witnesses of the incidents which my Model was intended to represent. In every quarter, and among Officers of all ranks, from the General to the Subaltern, my applications were
- 68. responded to in a most liberal and generous spirit; and the result did indeed surprise me, so greatly at variance was this historical evidence with the general notions which had previously prevailed on the subject. Thus was suggested the present Work. I was induced by the success of this experiment to embrace a wider field, and to extend my enquiries over the entire Battle; and, ultimately, throughout the Campaign itself, from its commencement to its close. Having become the depository of such valuable materials, I felt it a duty to the honourable profession of which I am a humble member, to submit to it, and to the World, a true and faithful account of this memorable epoch in the history of Britain's military greatness. Though not so presumptuous as to imagine that I have fully supplied so absolute a desideratum; yet I consider myself fortunate in being the instrument of withdrawing so far the veil from Truth. One of my Waterloo correspondents has humorously remarked, that "if ever truth lies at the bottom of a well, she does so immediately after a great Battle; and it takes an amazingly long time before she can be lugged out." The time of her emerging appears to have at length arrived; but, while I feel that I have brought to light much that was involved in obscurity, I cannot but be sensible that I may have fallen into errors. Should such be the case, I shall be most ready, hereafter, to make any corrections that may appear requisite, on my being favoured, by Eye Witnesses, with further well authenticated information. I take this opportunity of returning my sincere thanks to the numerous Officers of the British Army, who have so kindly committed to my keeping their recollections of the events which I have attempted to describe. Similar thanks are likewise due to the Officers of the King's German Legion and Hanoverian Subsidiary Corps; as also to the General Officers who respectively furnished me with such information as related to the troops of Brunswick and Nassau. I beg also to express my obligations to the Prussian Minister of War, and the Officers of the Prussian General Staff in Berlin, for the
- 69. readiness and liberality with which they have supplied me with such details concerning the dispositions and movements of the troops of their Sovereign, as were essential to me in prosecuting the task I had undertaken. Having briefly explained the circumstances that led to the construction of the Work which I thus venture to place before the Public, I have now only to express a hope that my labours may be crowned with usefulness. Should such a result occur, I shall then have obtained the only fame I seek. W. SIBORNE. March 1844. TABLE OF CONTENTS. CHAPTER I. PAGE Landing of Napoleon Buonaparte in France after his escape from Elba 47 Flight of Louis XVIII. 47 Decision of the Congress of Vienna 48 Preparations on the part of the Allied Powers for opening a Campaign against Napoleon 49 Great Britain and Prussia occupy Belgium 49 Advance of the Russians towards the French frontier 51
- 70. Advance of the Austrians 52 The troops of Bavaria, Baden, Würtemburg, and of Hesse, assemble upon the Upper Rhine 52 Preparations on the part of Napoleon 53 General aspect of France 57 Spirit of the French Army 58 Public Opinion and state of Parties in France 59 CHAPTER II. Belgium again destined to become the Theatre of War 62 The British Army 62 The Duke of Wellington 63 The Prussian Army 67 Prince Blücher von Wahlstadt 67 The King's German Legion; the Hanoverian, Brunswick, Dutch, Belgian, and Nassau troops 67 Napoleon and the French Army 68 Prospect of a severe struggle 69 CHAPTER III. Strength, composition, and distribution of the Anglo-Allied Army under Wellington 71 Its projected concentration in the event of Napoleon's advance 75 Strength, composition, and distribution of the Prussian Army under Blücher 76
- 71. Its projected concentration in the event of Napoleon's advance 79 The line on which Wellington's Left and Blücher's Right rested, selected by Napoleon for the direction of his attack 82 Strength, composition, and distribution of the French Army under Napoleon 82 Necessity under which the French Emperor is placed of opening the Campaign without awaiting the further development of his resources 87 Slight retrospect of the Campaign of 1814 88 Napoleon's prospect of success 88 His preparations for the commencement of hostilities 90 Wellington receives information from his Outposts in front of Tournai, of the assembling of French troops on the frontier; but delays the concentration of the Anglo-Allied troops until certain of the object and direction of Napoleon's main operation 91 Concentration of the French Army 91 Napoleon joins the latter in person 92 Ordre du Jour of the 14th of June 93 CHAPTER IV.
- 72. Zieten ascertains and communicates to the Allied Commanders the assembling of French troops in his front, and that there is every probability of an attack by the Enemy on the 14th or 15th of June 94 Blücher's dispositions 96 Extent of information gained by Wellington and Blücher immediately previous to the commencement of hostilities 97 Position of the First Prussian Corps d'Armée under Zieten 97 Advance of the French Army into Belgium on the 15th of June 98 The French force the Prussian Outposts; cross the Sambre, and gain possession of Charleroi 98 Retreat of the different Brigades of Zieten's Corps upon Fleurus 104 Affair at Gilly 106 Zieten's Corps concentrates in position between Ligny and St Arnaud 110 Losses experienced by this Corps on the 15th 111 The Second and Third Prussian Corps d'Armée, under Pirch and Thielemann, concentrate and bivouac on the night of the 15th; the former between Onoz and Mazy not far 111
- 73. from Sombref, the latter in and around Namur Bülow is desired to concentrate the Fourth Prussian Corps d'Armée at Hannut 112 Cause of this operation being deferred until the 16th 113 Ney joins the French Army, and receives from Napoleon the command of a detached Corps destined to operate by the Brussels road from Charleroi 114 The Advanced Post at Frasne, upon the extreme Left of the Duke of Wellington's Army, receives intelligence of the French attack 115 Consequent movements of de Perponcher's Dutch-Belgian Division 115 The Anglo-Allied Post at Frasne is driven in by the Advanced Guard of Ney's Corps; the progress of which is checked by Prince Bernhard of Saxe Weimar's Dutch-Belgian Brigade in front of Quatre Bras 116 Disposition of Ney's forces in the night of the 15th of June 118 Wellington is informed of Napoleon's advance, and makes his dispositions accordingly 119 Order of the movements of the Anglo-Allied Army 120
- 74. Disposition of the Centre and Right Columns of the French Army during the night of the 15th 123 Remarks on the result of Napoleon's operations on the 15th of June 123 CHAPTER V. On the morning of the 16th, Wellington's troops are in movement upon Nivelles and Quatre Bras 129 The Dutch-Belgian Detachment at the latter point is reinforced, and becomes engaged with the French Advanced Guard 129 The Prince of Orange arrives, and succeeds in forcing back the French upon Frasne 131 Ney's views and dispositions 131 Wellington arrives in person at Quatre Bras 134 He proceeds to the Prussian Head Quarters for the purpose of holding a conference with Blücher 134 Adopted Plan of Operations 135 Instructions received by Ney from Napoleon 135 Ney's advance 143 The Prince of Orange's dispositions to meet it 143 Relative strength 143
- 75. The Prince of Orange retires towards Quatre Bras, occupies the Wood of Bossu, and endeavours to maintain the Post of Gemioncourt 144 Arrival of Picton's Division 145 Conspicuous gallantry of the Prince of Orange 147 Arrival of van Merlen's Light Cavalry Brigade 148 Van Merlen advances in support of Perponcher's Infantry 148 Both are driven back: the former to Quatre Bras; the latter into the Wood of Bossu, which is now attacked by the French 148 The latter occupy Gemioncourt and Piermont 148 Ney's position 149 Arrival of the principal portion of the Brunswick troops 149 Relative strength 150 Part of the Brunswick Corps posted between the Charleroi road and the Wood of Bossu 151 French attack 152 Wellington decides on meeting it 153 Advance of Picton with the Fifth British Division 153 The French Infantry gallantly repulsed by the British 154
- 76. Attack upon the Brunswickers 155 The Duke of Brunswick makes an ineffectual charge at the head of his Lancers 157 Retreat of the Brunswickers 157 Fall of the Duke of Brunswick 158 Conspicuous gallantry of the 42nd and 44th British Regiments 159 The French Cavalry advances as far as Quatre Bras 162 Is checked by the 92nd Highlanders 162 Kellermann joins Ney with L'Heritier's Cavalry Division 163 The French Cavalry attacks the British Squares 164 Picton advances his Infantry into the midst of the French Cavalry 166 Remarkable steadiness of the British Squares 167 Manner in which the charges of the French Cavalry were executed 167 The French are rapidly gaining possession of the entire Wood of Bossu, are reinforcing their Light Troops in Piermont, and are preparing to renew their attack upon Quatre Bras 172 Alten joins Wellington with two Infantry Brigades of the Third Division 173
- 77. Ney is joined by the remaining Division of Kellermann's Corps of Heavy Cavalry 173 Relative strength 173 Ney, after despatching an Order to d'Erlon to join him without delay, commences another general attack 174 Two French Foot Batteries suddenly open a fire from the edge of the Wood of Bossu upon the Brunswick Infantry 174 Gallant conduct of Lloyd's British Foot Battery 174 Advance of Halkett's British Infantry Brigade posted between the Wood of Bossu and the Charleroi road 175 Kielmansegge's Hanoverian Infantry Brigade advances along the Namur road to reinforce and support Picton's Division 175 Advance of French Infantry against Quatre Bras 176 The latter gallantly charged and pursued by the 92nd Highlanders 176 Halkett's Brigade posted between the Wood of Bossu and the Charleroi road 177 The 69th British Regiment is attacked and dispersed by French Cuirassiers 178 Vigorous assault along the whole of the Anglo-Allied Line 180
- 78. Arrival of British and German Artillery 181 French Cuirassiers driven back in confusion from Quatre Bras 182 Ney receives intelligence that d'Erlon's Corps has been ordered by Napoleon to march towards the Prussian Extreme Right on the Field of Ligny; and shortly afterwards a despatch reaches him, requiring him to attack and repulse whatever Enemy may be in his front, and then to fall upon the Prussian Right 182 Vigorous attack upon the Left of Wellington's Line successfully repelled 184 The French Cavalry continues its attacks upon the central portion of the Anglo-Allied Army 184 Ney receives a further despatch from the Emperor, urging him to comply immediately with the instructions previously given 185 Arrival of Brunswick reinforcement 185 Also of the First British Division under Cooke 186 Relative strength 186 Halkett is again attacked by French Cavalry, after which he makes a further advance of his Brigade 187 The British Guards succeed in forcing the French out of the Wood 188
- 79. of Bossu Signal defeat of French Cavalry by the British Guards and the Brunswick Guard Battalion 189 Wellington's victorious advance 191 Ney withdraws the whole of his forces to the Heights of Frasne, on which they bivouac for the night 191 D'Erlon joins Ney after the termination of the action 191 Losses in killed and wounded 193 Remarks upon the Battle 193 CHAPTER VI. Blücher decides upon accepting battle in the position in rear of Fleurus 199 The position of Ligny strategically considered 200 The position itself described 201 Distribution of Zieten's Corps on the morning of the 16th of June 201 At eleven o'clock Pirch's Corps is posted as a Reserve to Zieten's 203 Thielemann's Corps reaches Sombref about noon 204 Its distribution on the Field 204 General view of Blücher's dispositions 204 About ten o'clock the foremost of the French troops debouch in two 204
- 80. Columns from the Wood of Fleurus, and draw up in front of this town Napoleon's views and dispositions 205 At two o'clock he communicates to Ney his intention to commence his attack upon the Prussians, and desires that Marshal also to attack the Enemy in his front 206 The French Light Troops gain possession of Fleurus 206 The Cavalry of Zieten's Corps falls back upon the position of Ligny 206 The French Army advances and takes up a position preparatory to its attack 207 Strength of the French forces under Napoleon 208 Strength of the Prussian forces under Blücher 209 Blücher's arrangements 209 He moves Thielemann's Corps into his Front Line, of which it then forms the Left Wing 210 Blücher's views and dispositions 211 Tactical defects of the position of Ligny 213 Napoleon commences the Battle with an attack by Vandamme's Corps upon St Amand 213 Gérard's Corps attacks Ligny 214 Contest in these Villages 215
- 81. The French carry St Amand 216 Renewed attack upon Ligny 217 Nature of the contest between Thielemann's and Grouchy's Corps 217 Girard's Division gains possession of St Amand la Haye 218 Blücher's dispositions for retaking this Village, securing Wagnelé, and impeding any further advance from the French Left 218 Failure of the Prussian attack upon St Amand la Haye 219 Blücher decides on a renewed attack upon this Village, as a diversion in favour of his projected movement against the French Left 219 Napoleon reinforces this Flank 220 The Prussians retake St Amand la Haye 220 Blücher reinforces his extreme Right with Cavalry 221 Prussian attack upon Wagnelé unsuccessful 222 The French regain St Amand la Haye 223 Continued contest at Ligny 223 Blücher reinforces his troops employed in the defence of this Village 224 Long and desperate struggle in the Villages of St Amand la Haye, 227
- 82. Wagnelé, and the Hameau de St Amand Napoleon, perceiving that Blücher has scarcely any Reserve remaining at his disposal, resolves upon attacking the Prussian Centre 230 He suspends his meditated attack in consequence of a large Column advancing apparently from Frasne towards his Left Rear 231 This Column is discovered to be d'Erlon's Corps d'Armée 234 This circumstance explained 234 Thielemann detaches a portion of his Cavalry with some guns across the Ligny, along the Fleurus road 237 They are attacked and driven back by part of Grouchy's Cavalry 237 Disposition and state of the Prussian troops at the moment Napoleon advances with a formidable Reserve across the Ligny 239 The Prussian Infantry forced to evacuate Ligny 242 Failure of Prussian Cavalry attacks upon the advancing Column of French Infantry 243 Blücher's horse is killed, and the Prince thrown under him 245 Critical situation of the Prussian Commander 246
- 83. He is removed from the Field 246 Retreat of Prussian Infantry upon Bry 247 Contest at Sombref 249 Retreat of the Prussians from St Amand and St Amand la Haye 250 Zieten's and Pirch's Corps retire by Marbais and Tilly 251 Thielemann's Corps retains its position 252 Close of the Battle 253 Distribution of the French troops 254 Disposition of the Prussian troops 254 Bülow's Corps reaches Gembloux during the night 255 Losses sustained by both Armies 255 Consequences of the Prussian defeat 255 Remarks upon the Battle 256 CHAPTER VII. An engagement of short duration, and originating accidentally, takes place between the French and Anglo-Allied Picquets on the Field of Quatre Bras, about an hour before daylight of the 17th June 259 Wellington detaches a Patrol to his Left for the purpose of gaining intelligence concerning Blücher's movements 261
- 84. The Patrol finds the Prussians at Tilly 262 Upon its return Wellington decides on retrograding his forces to the position in front of Waterloo 263 Order of Movement 263 Communications between Blücher and Wellington 264 Retreat of the Anglo-Allied Infantry; masked from the Enemy 264 Ney's views and dispositions 266 Napoleon communicates to Ney the result of the Battle of Ligny; and proposes, should the Enemy's force at Quatre Bras advance against him, to co-operate with the Marshal in a combined attack upon the Anglo-Allied Army 267 Tardiness of Napoleon's movements 267 Simultaneous advance of Napoleon and Ney against Wellington 268 Uxbridge's dispositions for the retreat of the British Cavalry 270 Brilliant Cavalry Affair at Genappe 281 Retreat continued to the Waterloo position 282 Napoleon's advance checked on his reaching La Belle Alliance 282 Remarks on the retreat 283 Blücher's promised support 285
- 85. Wellington's disposition of his detached troops under Sir Charles Colville and Prince Frederick of Orange 285 The French and Anglo-Allied Armies establish their respective bivouacs for the night 286 CHAPTER VIII. At daybreak of the 17th, the Prussian Army commences its retreat upon Wavre 287 Zieten's Corps retires by Mont St Guibert, and reaches Wavre about mid day 287 Pirch's Corps follows the same route, and takes post upon the right bank of the Dyle 287 Thielemann, having collected together the Brigades of his Corps, begins to retire from the Field of Ligny at two o'clock in the morning 288 He halts in rear of Gembloux 289 Bülow retires by Walhain and Corbaix to Dion le Mont, near which he takes up a position 290 Thielemann resumes his march at two o'clock in the afternoon, and arrives at the position of Wavre late in the evening 290 Prussian Head Quarters established at Wavre 291
- 86. Blücher receives a message from Wellington 291 While the Prussians are effecting their retreat during the early part of the morning, the French continue quietly in their bivouac 292 Pajol, with the Light Cavalry Division, seeks the Prussians along the Namur road; followed by Lieutenant General Teste's Infantry Division, in support 292 Other troops detached towards Gembloux, near which traces of the Prussian retreat are discovered 293 Remarks upon the extraordinary degree of inactivity on the part of Napoleon 293 About noon, Napoleon proceeds to collect, in advance of Marbais, on the high road to Quatre Bras, a portion of the troops that had fought at Ligny; and detaches the remainder, under Grouchy, in pursuit of the Prussians 296 Napoleon's instructions to Grouchy 297 The troops assembled near Marbais advance upon Quatre Bras, which they reach about two o'clock 298 The Corps of Vandamme and Gérard do not reach Gembloux until late in the evening 299 Grouchy's dispositions 300
- 87. Welcome to our website – the perfect destination for book lovers and knowledge seekers. We believe that every book holds a new world, offering opportunities for learning, discovery, and personal growth. That’s why we are dedicated to bringing you a diverse collection of books, ranging from classic literature and specialized publications to self-development guides and children's books. More than just a book-buying platform, we strive to be a bridge connecting you with timeless cultural and intellectual values. With an elegant, user-friendly interface and a smart search system, you can quickly find the books that best suit your interests. Additionally, our special promotions and home delivery services help you save time and fully enjoy the joy of reading. Join us on a journey of knowledge exploration, passion nurturing, and personal growth every day! ebookbell.com