Production Kubernetes: Building Successful Application Platforms 1st Edition Josh Rosso

Read Anytime Anywhere Easy Ebook Downloads at ebookmeta.com
Production Kubernetes: Building Successful
Application Platforms 1st Edition Josh Rosso
https://ebookmeta.com/product/production-kubernetes-
building-successful-application-platforms-1st-edition-josh-
rosso/
OR CLICK HERE
DOWLOAD EBOOK
Visit and Get More Ebook Downloads Instantly at https://ebookmeta.com

Recommended digital products (PDF, EPUB, MOBI) that
you can download immediately if you are interested.
Kubernetes Best Practices Blueprints for Building
Successful Applications on Kubernetes 1st Edition Brendan
Burns
https://ebookmeta.com/product/kubernetes-best-practices-blueprints-
for-building-successful-applications-on-kubernetes-1st-edition-
brendan-burns/
ebookmeta.com
Kubernetes Best Practices: Blueprints for Building
Successful Applications on Kubernetes, 2nd Edition Brendan
Burns
https://ebookmeta.com/product/kubernetes-best-practices-blueprints-
for-building-successful-applications-on-kubernetes-2nd-edition-
brendan-burns/
ebookmeta.com
Cloud Native Integration with Apache Camel: Building Agile
and Scalable Integrations for Kubernetes Platforms 1st
Edition Guilherme Camposo
https://ebookmeta.com/product/cloud-native-integration-with-apache-
camel-building-agile-and-scalable-integrations-for-kubernetes-
platforms-1st-edition-guilherme-camposo/
ebookmeta.com
Outlook For Dummies 1st Edition Faithe Wempen
https://ebookmeta.com/product/outlook-for-dummies-1st-edition-faithe-
wempen-2/
ebookmeta.com

Secret Project 1 Brandon Sanderson
https://ebookmeta.com/product/secret-project-1-brandon-sanderson/
ebookmeta.com
Neighbourhoods in Transition Brownfield Regeneration in
European Metropolitan Areas 1st Edition Emmanuel Rey
https://ebookmeta.com/product/neighbourhoods-in-transition-brownfield-
regeneration-in-european-metropolitan-areas-1st-edition-emmanuel-rey/
ebookmeta.com
Myofascial Induction Andrzej Pilat
https://ebookmeta.com/product/myofascial-induction-andrzej-pilat/
ebookmeta.com
The Cambridge Handbook of Health Research Regulation 1st
Edition Graeme Laurie
https://ebookmeta.com/product/the-cambridge-handbook-of-health-
research-regulation-1st-edition-graeme-laurie/
ebookmeta.com
Shadow Working in Project Management Understanding and
Addressing the Irrational and Unconscious in Groups 1st
Edition Joana Bértholo
https://ebookmeta.com/product/shadow-working-in-project-management-
understanding-and-addressing-the-irrational-and-unconscious-in-
groups-1st-edition-joana-bertholo/
ebookmeta.com

Heron Streaming Fundamentals Applications Operations and
Insights Huijun Wu & Maosong Fu
https://ebookmeta.com/product/heron-streaming-fundamentals-
applications-operations-and-insights-huijun-wu-maosong-fu/
ebookmeta.com

Production
Kubernetes
Building Successful Application Platforms
Josh Rosso, Rich Lander,
Alexander Brand & John Harris

Production Kubernetes: Building Successful Application Platforms 1st Edition Josh Rosso

Josh Rosso, Rich Lander,
Alexander Brand, and John Harris
Production Kubernetes
Building Successful Application Platforms
Boston Farnham Sebastopol Tokyo
Beijing Boston Farnham Sebastopol Tokyo
Beijing

978-1-492-09230-8
[LSI]
Production Kubernetes
by Josh Rosso, Rich Lander, Alexander Brand, and John Harris
Copyright © 2021 Josh Rosso, Rich Lander, Alexander Brand, and John Harris. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.
O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are
also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional
sales department: 800-998-9938 or corporate@oreilly.com.
Acquisitions Editor: John Devins
Development Editor: Jeff Bleiel
Production Editor: Christopher Faucher
Copyeditor: Kim Cofer
Proofreader: Piper Editorial Consulting, LLC
Indexer: Ellen Troutman
Interior Designer: David Futato
Cover Designer: Karen Montgomery
Illustrator: Kate Dullea
March 2021: First Edition
Revision History for the First Edition
2021-03-16: First Release
See http://oreilly.com/catalog/errata.csp?isbn=9781492092308 for release details.
The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Production Kubernetes, the cover
image, and related trade dress are trademarks of O’Reilly Media, Inc.
The views expressed in this work are those of the authors, and do not represent the publisher’s views.
While the publisher and the authors have used good faith efforts to ensure that the information and
instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility
for errors or omissions, including without limitation responsibility for damages resulting from the use of
or reliance on this work. Use of the information and instructions contained in this work is at your own
risk. If any code samples or other technology this work contains or describes is subject to open source
licenses or the intellectual property rights of others, it is your responsibility to ensure that your use
thereof complies with such licenses and/or rights.
This work is part of a collaboration between O’Reilly and VMware Tanzu. See our statement of editorial
independence.

Table of Contents
Foreword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv
1. A Path to Production. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Defining Kubernetes 1
The Core Components 2
Beyond Orchestration—Extended Functionality 4
Kubernetes Interfaces 5
Summarizing Kubernetes 7
Defining Application Platforms 7
The Spectrum of Approaches 8
Aligning Your Organizational Needs 10
Summarizing Application Platforms 11
Building Application Platforms on Kubernetes 12
Starting from the Bottom 13
The Abstraction Spectrum 15
Determining Platform Services 16
The Building Blocks 17
Summary 21
2. Deployment Models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Managed Service Versus Roll Your Own 24
Managed Services 24
Roll Your Own 24
Making the Decision 25
Automation 26
Prebuilt Installer 26
iii

Custom Automation 27
Architecture and Topology 28
etcd Deployment Models 28
Cluster Tiers 29
Node Pools 31
Cluster Federation 32
Infrastructure 35
Bare Metal Versus Virtualized 36
Cluster Sizing 39
Compute Infrastructure 41
Networking Infrastructure 42
Automation Strategies 44
Machine Installations 46
Configuration Management 46
Machine Images 46
What to Install 47
Containerized Components 49
Add-ons 50
Upgrades 52
Platform Versioning 52
Plan to Fail 53
Integration Testing 54
Strategies 55
Triggering Mechanisms 60
Summary 61
3. Container Runtime. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
The Advent of Containers 64
The Open Container Initiative 65
OCI Runtime Specification 65
OCI Image Specification 67
The Container Runtime Interface 69
Starting a Pod 70
Choosing a Runtime 72
Docker 73
containerd 74
CRI-O 75
Kata Containers 76
Virtual Kubelet 77
Summary 78
iv | Table of Contents

4. Container Storage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
Storage Considerations 80
Access Modes 80
Volume Expansion 81
Volume Provisioning 81
Backup and Recovery 81
Block Devices and File and Object Storage 82
Ephemeral Data 83
Choosing a Storage Provider 83
Kubernetes Storage Primitives 83
Persistent Volumes and Claims 83
Storage Classes 86
The Container Storage Interface (CSI) 87
CSI Controller 88
CSI Node 89
Implementing Storage as a Service 89
Installation 90
Exposing Storage Options 92
Consuming Storage 94
Resizing 96
Snapshots 97
Summary 99
5. Pod Networking. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
Networking Considerations 102
IP Address Management 102
Routing Protocols 104
Encapsulation and Tunneling 106
Workload Routability 108
IPv4 and IPv6 109
Encrypted Workload Traffic 109
Network Policy 110
Summary: Networking Considerations 112
The Container Networking Interface (CNI) 112
CNI Installation 114
CNI Plug-ins 116
Calico 117
Cilium 120
AWS VPC CNI 123
Multus 125
Additional Plug-ins 126
Summary 126
Table of Contents | v

6. Service Routing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
Kubernetes Services 128
The Service Abstraction 128
Endpoints 135
Service Implementation Details 138
Service Discovery 148
DNS Service Performance 151
Ingress 152
The Case for Ingress 153
The Ingress API 154
Ingress Controllers and How They Work 156
Ingress Traffic Patterns 157
Choosing an Ingress Controller 161
Ingress Controller Deployment Considerations 162
DNS and Its Role in Ingress 165
Handling TLS Certificates 166
Service Mesh 169
When (Not) to Use a Service Mesh 169
The Service Mesh Interface (SMI) 170
The Data Plane Proxy 173
Service Mesh on Kubernetes 175
Data Plane Architecture 179
Adopting a Service Mesh 181
Summary 184
7. Secret Management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
Defense in Depth 188
Disk Encryption 189
Transport Security 190
Application Encryption 190
The Kubernetes Secret API 191
Secret Consumption Models 193
Secret Data in etcd 196
Static-Key Encryption 198
Envelope Encryption 201
External Providers 203
Vault 203
Cyberark 203
Injection Integration 204
CSI Integration 208
Secrets in the Declarative World 210
Sealing Secrets 211
vi | Table of Contents

Sealed Secrets Controller 211
Key Renewal 214
Multicluster Models 215
Best Practices for Secrets 215
Always Audit Secret Interaction 215
Don’t Leak Secrets 216
Prefer Volumes Over Environment Variables 216
Make Secret Store Providers Unknown to Your Application 216
Summary 217
8. Admission Control. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
The Kubernetes Admission Chain 220
In-Tree Admission Controllers 222
Webhooks 223
Configuring Webhook Admission Controllers 225
Webhook Design Considerations 227
Writing a Mutating Webhook 228
Plain HTTPS Handler 229
Controller Runtime 231
Centralized Policy Systems 234
Summary 241
9. Observability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
Logging Mechanics 244
Container Log Processing 244
Kubernetes Audit Logs 247
Kubernetes Events 249
Alerting on Logs 250
Security Implications 251
Metrics 251
Prometheus 251
Long-Term Storage 253
Pushing Metrics 253
Custom Metrics 253
Organization and Federation 254
Alerts 255
Showback and Chargeback 257
Metrics Components 260
Distributed Tracing 269
OpenTracing and OpenTelemetry 269
Tracing Components 270
Application Instrumentation 272
Table of Contents | vii

Service Meshes 272
Summary 272
10. Identity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
User Identity 274
Authentication Methods 275
Implementing Least Privilege Permissions for Users 285
Application/Workload Identity 288
Shared Secrets 289
Network Identity 289
Service Account Tokens (SAT) 293
Projected Service Account Tokens (PSAT) 297
Platform Mediated Node Identity 299
Summary 311
11. Building Platform Services. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
Points of Extension 314
Plug-in Extensions 314
Webhook Extensions 315
Operator Extensions 316
The Operator Pattern 317
Kubernetes Controllers 317
Custom Resources 318
Operator Use Cases 323
Platform Utilities 323
General-Purpose Workload Operators 324
App-Specific Operators 324
Developing Operators 325
Operator Development Tooling 325
Data Model Design 329
Logic Implementation 331
Extending the Scheduler 347
Predicates and Priorities 348
Scheduling Policies 348
Scheduling Profiles 350
Multiple Schedulers 350
Custom Scheduler 350
Summary 351
12. Multitenancy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353
Degrees of Isolation 354
Single-Tenant Clusters 354
viii | Table of Contents

Multitenant Clusters 355
The Namespace Boundary 357
Multitenancy in Kubernetes 358
Role-Based Access Control (RBAC) 358
Resource Quotas 360
Admission Webhooks 361
Resource Requests and Limits 363
Network Policies 368
Pod Security Policies 370
Multitenant Platform Services 374
Summary 375
13. Autoscaling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377
Types of Scaling 378
Application Architecture 379
Workload Autoscaling 380
Horizontal Pod Autoscaler 380
Vertical Pod Autoscaler 384
Autoscaling with Custom Metrics 387
Cluster Proportional Autoscaler 388
Custom Autoscaling 389
Cluster Autoscaling 389
Cluster Overprovisioning 393
Summary 395
14. Application Considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397
Deploying Applications to Kubernetes 398
Templating Deployment Manifests 398
Packaging Applications for Kubernetes 399
Ingesting Configuration and Secrets 400
Kubernetes ConfigMaps and Secrets 400
Obtaining Configuration from External Systems 403
Handling Rescheduling Events 404
Pre-stop Container Life Cycle Hook 404
Graceful Container Shutdown 405
Satisfying Availability Requirements 407
State Probes 408
Liveness Probes 409
Readiness Probes 410
Startup Probes 411
Implementing Probes 412
Pod Resource Requests and Limits 413
Table of Contents | ix

Resource Requests 413
Resource Limits 414
Application Logs 415
What to Log 415
Unstructured Versus Structured Logs 416
Contextual Information in Logs 416
Exposing Metrics 416
Instrumenting Applications 417
USE Method 419
RED Method 419
The Four Golden Signals 419
App-Specific Metrics 419
Instrumenting Services for Distributed Tracing 420
Initializing the Tracer 420
Creating Spans 421
Propagate Context 422
Summary 423
15. Software Supply Chain. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425
Building Container Images 426
The Golden Base Images Antipattern 428
Choosing a Base Image 429
Runtime User 430
Pinning Package Versions 430
Build Versus Runtime Image 431
Cloud Native Buildpacks 432
Image Registries 434
Vulnerability Scanning 435
Quarantine Workflow 437
Image Signing 438
Continuous Delivery 439
Integrating Builds into a Pipeline 440
Push-Based Deployments 443
Rollout Patterns 445
GitOps 446
Summary 448
16. Platform Abstractions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 449
Platform Exposure 450
Self-Service Onboarding 451
The Spectrum of Abstraction 453
Command-Line Tooling 454
x | Table of Contents

Abstraction Through Templating 455
Abstracting Kubernetes Primitives 458
Making Kubernetes Invisible 462
Summary 464
Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 465
Table of Contents | xi

Foreword
It has been more than six years since we publicly released Kubernetes. I was there at
the start and actually submitted the first commit to the Kubernetes project. (That isn’t
as impressive as it sounds! It was a maintenance task as part of creating a clean repo
for public release.) I can confidently say that the success Kubernetes has seen is some‐
thing we had hoped for but didn’t really expect. That success is based on a large com‐
munity of dedicated and welcoming contributors along with a set of practitioners
who bridge the gap to the real world.
I’m lucky enough to have worked with the authors of Production Kubernetes at the
startup (Heptio) that I cofounded with the mission to bring Kubernetes to typical
enterprises. The success of Heptio is, in large part, due to my colleagues’ efforts in
creating a direct connection with real users of Kubernetes who are solving real prob‐
lems. I’m grateful to each one of them. This book captures that on-the-ground experi‐
ence to give teams the tools they need to really make Kubernetes work in a
production environment.
My entire professional career has been based on building systems aimed at applica‐
tion teams and developers. It started with Microsoft Internet Explorer and then con‐
tinued with Windows Presentation Foundation and then moved to cloud with Google
Compute Engine and Kubernetes. Again and again I’ve seen those building platforms
suffer from what I call “The Platform Builder’s Curse.” The people who are building
the platforms are focused on a longer time horizon and the challenge of building a
foundation that will, hopefully, last decades. But that focus creates a blind spot to the
problems that users are having right now. Oftentimes we are so busy building a thing
we don’t have the time and problems that lead us to actually use the thing we are
building.
xiii

The only way to defeat the platform builder’s curse is to actively seek information
from outside our platform-builder bubble. This is what the Heptio Field Engineering
team (and later the VMware Kubernetes Architecture Team—KAT) did for me.
Beyond helping a wide variety of customers across industries be successful with
Kubernetes, the team is a critical window into the reality of how the “theory” of our
platform is applied.
This problem is only exacerbated by the thriving ecosystem that has been built up
around Kubernetes and the Cloud Native Computing Foundation (CNCF). This
includes both projects that are part of the CNCF and those that are in the larger orbit.
I describe this ecosystem as “beautiful chaos.” It is a rainforest of projects with vary‐
ing degrees of overlap and maturity. This is what innovation looks like! But, just like
exploring a rainforest, exploring this ecosystem requires dedication and time, and it
comes with risks. New users to the world of Kubernetes often don’t have the time or
capacity to become experts in the larger ecosystem.
Production Kubernetes maps out the parts of that ecosystem, when individual tools
and projects are appropriate, and demonstrates how to evaluate the right tool for the
problems the reader is facing. This advice goes beyond just telling readers to use a
particular tool. It is a larger framework for understanding the problem a class of tools
solves, knowing whether you have that problem, being familiar with the strengths
and weaknesses to different approaches, and offering practical advice for getting
going. For those looking to take Kubernetes into production, this information is gold!
In conclusion, I’d like to send a big “Thank You” to Josh, Rich, Alex, and John. Their
experience has made many customers directly successful, has taught me a lot about
the thing that we started more than six years ago, and now, through this book, will
provide critical advice to countless more users.
— Joe Beda
Principal Engineer for VMware Tanzu,
Cocreator of Kubernetes,
Seattle, January 2021
xiv | Foreword

Preface
Kubernetes is a remarkably powerful technology and has achieved a meteoric rise in
popularity. It has formed the basis for genuine advances in the way we manage soft‐
ware deployments. API-driven software and distributed systems were well estab‐
lished, if not widely adopted, when Kubernetes emerged. It delivered excellent
renditions of these principles, which are foundational to its success, but it also deliv‐
ered something else that is vital. In the recent past, software that autonomously con‐
verged on declared, desired state was possible only in giant technology companies
with the most talented engineering teams. Now, highly available, self-healing,
autoscaling software deployments are within reach of every organization, thanks to
the Kubernetes project. There is a future in front of us where software systems accept
broad, high-level directives from us and execute upon them to deliver desired out‐
comes by discovering conditions, navigating changing obstacles, and repairing prob‐
lems without our intervention. Furthermore, these systems will do it faster and more
reliably than we ever could with manual operations. Kubernetes has brought us all
much closer to that future. However, that power and capability comes at the cost of
some additional complexity. The desire to share our experiences helping others navi‐
gate that complexity is why we decided to write this book.
You should read this book if you want to use Kubernetes to build a production-grade
application platform. If you are looking for a book to help you get started with Kuber‐
netes, or a text on how Kubernetes works, this is not the right book. There is a wealth
of information on these subjects in other books, in the official documentation, and in
countless blog posts and the source code itself. We recommend pairing the consump‐
tion of this book with your own research and testing for the solutions we discuss, so
we rarely dive deeply into step-by-step tutorial style examples. We try to cover as
much theory as necessary and leave most of the implementation as an exercise to the
reader.
xv

Throughout this book, you’ll find guidance in the form of options, tooling, patterns,
and practices. It’s important to read this guidance with an understanding of how the
authors view the practice of building application platforms. We are engineers and
architects who get deployed across many Fortune 500 companies to help them take
their platform aspirations from idea to production. We have been using Kubernetes
as the foundation for getting there since as early as 2015, when Kubernetes reached
1.0. We have tried as much as possible to focus on patterns and philosophy rather
than on tools, as new tooling appears quicker than we can write! However, we inevi‐
tably have to demonstrate those patterns with the most appropriate tool du jour.
We have had major successes guiding teams through their cloud native journey to
completely transform how they build and deliver software. That said, we have also
had our doses of failure. A common reason for failure is an organization’s misconcep‐
tion of what Kubernetes will solve for. This is why we dive so deep into the concept
early on. Over this time we’ve found several areas to be especially interesting for our
customers. Conversations that help customers get further on their path to produc‐
tion, or even help them define it, have become routine. These conversations became
so common that we decided maybe it’s time to write a book!
While we’ve made this journey to production with organizations time and time again,
there is only one key consistency across them. This is that the road never looks the
same, no matter how badly we sometimes want it to. With this in mind, we want to
set the expectation that if you’re going into this book looking for the “5-step program”
for getting to production or the “10 things every Kubernetes user should know,”
you’re going to be frustrated. We’re here to talk about the many decision points and
the traps we’ve seen, and to back it up with concrete examples and anecdotes when
appropriate. Best practices exist but must always be viewed through the lens of prag‐
matism. There is no one-size-fits-all approach, and “It depends” is an entirely valid
answer to many of the questions you’ll inevitably confront on the journey.
That said, we highly encourage you to challenge this book! When working with clients
we’re always encouraging them to challenge and augment our guidance. Knowledge is
fluid, and we are always updating our approaches based on new features, informa‐
tion, and constraints. You should continue that trend; as the cloud native space con‐
tinues to evolve, you’ll certainly decide to take alternative roads from what we
recommended. We’re here to tell you about the ones we’ve been down so you can
weigh our perspective against your own.
xvi | Preface

Conventions Used in This Book
The following typographical conventions are used in this book:
Italic
Indicates new terms, URLs, email addresses, filenames, and file extensions.
Constant width
Used for program listings, as well as within paragraphs to refer to program ele‐
ments such as variable or function names, databases, data types, environment
variables, statements, and keywords.
Constant width bold
Shows commands or other text that should be typed literally by the user.
Constant width italic
Shows text that should be replaced with user-supplied values or by values deter‐
mined by context.
Kubernetes kinds are capitalized, as in Pod, Service, and StatefulSet.
This element signifies a tip or suggestion.
This element signifies a general note.
This element indicates a warning or caution.
Using Code Examples
Supplemental material (code examples, exercises, etc.) is available for download and
discussion at https://github.com/production-kubernetes.
If you have a technical question or a problem using the code examples, please send
email to bookquestions@oreilly.com.
Preface | xvii

This book is here to help you get your job done. In general, if example code is offered
with this book, you may use it in your programs and documentation. You do not
need to contact us for permission unless you’re reproducing a significant portion of
the code. For example, writing a program that uses several chunks of code from this
book does not require permission. Selling or distributing examples from O’Reilly
books does require permission. Answering a question by citing this book and quoting
example code does not require permission. Incorporating a significant amount of
example code from this book into your product’s documentation does require
permission.
We appreciate, but generally do not require, attribution. An attribution usually
includes the title, author, publisher, and ISBN. For example: “Production Kubernetes
by Josh Rosso, Rich Lander, Alexander Brand, and John Harris (O’Reilly). Copyright
2021 Josh Rosso, Rich Lander, Alexander Brand, and John Harris,
978-1-492-09231-5.”
If you feel your use of code examples falls outside fair use or the permission given
above, feel free to contact us at permissions@oreilly.com.
O’Reilly Online Learning
For more than 40 years, O’Reilly Media has provided technol‐
ogy and business training, knowledge, and insight to help
companies succeed.
Our unique network of experts and innovators share their knowledge and expertise
through books, articles, and our online learning platform. O’Reilly’s online learning
platform gives you on-demand access to live training courses, in-depth learning
paths, interactive coding environments, and a vast collection of text and video from
O’Reilly and 200+ other publishers. For more information, visit http://oreilly.com.
How to Contact Us
Please address comments and questions concerning this book to the publisher:
O’Reilly Media, Inc.
1005 Gravenstein Highway North
Sebastopol, CA 95472
800-998-9938 (in the United States or Canada)
707-829-0515 (international or local)
707-829-0104 (fax)
xviii | Preface

We have a web page for this book, where we list errata, examples, and any additional
information. You can access this page at https://oreil.ly/production-kubernetes.
Email bookquestions@oreilly.com to comment or ask technical questions about this
book.
For news and information about our books and courses, visit http://oreilly.com.
Find us on Facebook: http://facebook.com/oreilly
Follow us on Twitter: http://twitter.com/oreillymedia
Watch us on YouTube: http://youtube.com/oreillymedia
Acknowledgments
The authors would like to thank Katie Gamanji, Michael Goodness, Jim Weber, Jed
Salazar, Tony Scully, Monica Rodriguez, Kris Dockery, Ralph Bankston, Steve Sloka,
Aaron Miller, Tunde Olu-Isa, Alex Withrow, Scott Lowe, Ryan Chapple, and Kenan
Dervisevic for their reviews and feedback on the manuscript. Thanks to Paul Lundin
for encouraging the development of this book and for building the incredible Field
Engineering team at Heptio. Everyone on the team has contributed in some way by
collaborating on and developing many of the ideas and experiences we cover over the
next 450 pages. Thanks also to Joe Beda, Scott Buchanan, Danielle Burrow, and Tim
Coventry-Cox at VMware for their support as we initiated and developed this project.
Finally, thanks to John Devins, Jeff Bleiel, and Christopher Faucher at O’Reilly for
their ongoing support and feedback.
The authors would also like to personally thank the following people:
Josh: I would like to thank Jessica Appelbaum for her absurd levels of support, specif‐
ically blueberry pancakes, while I dedicated my time to this book. I’d also like to
thank my mom, Angela, and dad, Joe, for being my foundation growing up.
Rich: I would like to thank my wife, Taylor, and children, Raina, Jasmine, Max, and
John, for their support and understanding while I took time to work on this book. I
would also like to thank my Mum, Jenny, and my Dad, Norm, for being great role
models.
Alexander: My love and thanks to my amazing wife, Anais, who was incredibly sup‐
portive as I dedicated time to writing this book. I also thank my family, friends, and
colleagues who have helped me become who I am today.
John: I’d like to thank my beautiful wife, Christina, for her love and patience during
my work on this book. Also thanks to my close friends and family for their ongoing
support and encouragement over the years.
Preface | xix

CHAPTER 1
A Path to Production
Over the years, the world has experienced wide adoption of Kubernetes within organ‐
izations. Its popularity has unquestionably been accelerated by the proliferation of
containerized workloads and microservices. As operations, infrastructure, and devel‐
opment teams arrive at this inflection point of needing to build, run, and support
these workloads, several are turning to Kubernetes as part of the solution. Kubernetes
is a fairly young project relative to other, massive, open source projects such as Linux.
Evidenced by many of the clients we work with, it is still early days for most users of
Kubernetes. While many organizations have an existing Kubernetes footprint, there
are far fewer that have reached production and even less operating at scale. In this
chapter, we are going to set the stage for the journey many engineering teams are on
with Kubernetes. Specifically, we are going to chart out some key considerations we
look at when defining a path to production.
Defining Kubernetes
Is Kubernetes a platform? Infrastructure? An application? There is no shortage of
thought leaders who can provide you their precise definition of what Kubernetes is.
Instead of adding to this pile of opinions, let’s put our energy into clarifying the prob‐
lems Kubernetes solves. Once defined, we will explore how to build atop this feature
set in a way that moves us toward production outcomes. The ideal state of “Produc‐
tion Kubernetes” implies that we have reached a state where workloads are success‐
fully serving production traffic.
The name Kubernetes can be a bit of an umbrella term. A quick browse on GitHub
reveals the kubernetes organization contains (at the time of this writing) 69 reposito‐
ries. Then there is kubernetes-sigs, which holds around 107 projects. And don’t get
us started on the hundreds of Cloud Native Compute Foundation (CNCF) projects
that play in this landscape! For the sake of this book, Kubernetes will refer exclusively
1

to the core project. So, what is the core? The core project is contained in the kuber‐
netes/kubernetes repository. This is the location for the key components we find in
most Kubernetes clusters. When running a cluster with these components, we can
expect the following functionality:
• Scheduling workloads across many hosts
• Exposing a declarative, extensible, API for interacting with the system
• Providing a CLI, kubectl, for humans to interact with the API server
• Reconciliation from current state of objects to desired state
• Providing a basic service abstraction to aid in routing requests to and from
workloads
• Exposing multiple interfaces to support pluggable networking, storage, and more
These capabilities create what the project itself claims to be, a production-grade con‐
tainer orchestrator. In simpler terms, Kubernetes provides a way for us to run and
schedule containerized workloads on multiple hosts. Keep this primary capability in
mind as we dive deeper. Over time, we hope to prove how this capability, while foun‐
dational, is only part of our journey to production.
The Core Components
What are the components that provide the functionality we have covered? As we have
mentioned, core components reside in the kubernetes/kubernetes repository. Many
of us consume these components in different ways. For example, those running man‐
aged services such as Google Kubernetes Engine (GKE) are likely to find each compo‐
nent present on hosts. Others may be downloading binaries from repositories or
getting signed versions from a vendor. Regardless, anyone can download a Kuber‐
netes release from the kubernetes/kubernetes repository. After downloading and
unpacking a release, binaries may be retrieved using the cluster/get-kube-
binaries.sh command. This will auto-detect your target architecture and download
server and client components. Let’s take a look at this in the following code, and then
explore the key components:
$ ./cluster/get-kube-binaries.sh
Kubernetes release: v1.18.6
Server: linux/amd64 (to override, set KUBERNETES_SERVER_ARCH)
Client: linux/amd64 (autodetected)
Will download kubernetes-server-linux-amd64.tar.gz from https://dl.k8s.io/v1.18.6
Will download and extract kubernetes-client-linux-amd64.tar.gz
Is this ok? [Y]/n
2 | Chapter 1: A Path to Production

Inside the downloaded server components, likely saved to server/kubernetes-server-$
{ARCH}.tar.gz, you’ll find the key items that compose a Kubernetes cluster:
API Server
The primary interaction point for all Kubernetes components and users. This is
where we get, add, delete, and mutate objects. The API server delegates state to a
backend, which is most commonly etcd.
kubelet
The on-host agent that communicates with the API server to report the status of
a node and understand what workloads should be scheduled on it. It communi‐
cates with the host’s container runtime, such as Docker, to ensure workloads
scheduled for the node are started and healthy.
Controller Manager
A set of controllers, bundled in a single binary, that handle reconciliation of
many core objects in Kubernetes. When desired state is declared, e.g., three repli‐
cas in a Deployment, a controller within handles the creation of new Pods to sat‐
isfy this state.
Scheduler
Determines where workloads should run based on what it thinks is the optimal
node. It uses filtering and scoring to make this decision.
Kube Proxy
Implements Kubernetes services providing virtual IPs that can route to backend
Pods. This is accomplished using a packet filtering mechanism on a host such as
iptables or ipvs.
While not an exhaustive list, these are the primary components that make up the core
functionality we have discussed. Architecturally, Figure 1-1 shows how these compo‐
nents play together.
Kubernetes architectures have many variations. For example, many
clusters run kube-apiserver, kube-scheduler, and kube-controller-
manager as containers. This means the control-plane may also run
a container-runtime, kubelet, and kube-proxy. These kinds of
deployment considerations will be covered in the next chapter.
Defining Kubernetes | 3

Figure 1-1. The primary components that make up the Kubernetes cluster. Dashed bor‐
ders represent components that are not part of core Kubernetes.
Beyond Orchestration—Extended Functionality
There are areas where Kubernetes does more than just orchestrate workloads. As
mentioned, the component kube-proxy programs hosts to provide a virtual IP (VIP)
experience for workloads. As a result, internal IP addresses are established and route
to one or many underlying Pods. This concern certainly goes beyond running and
scheduling containerized workloads. In theory, rather than implementing this as part
of core Kubernetes, the project could have defined a Service API and required a plug-
in to implement the Service abstraction. This approach would require users to choose
between a variety of plug-ins in the ecosystem rather than including it as core
functionality.
This is the model many Kubernetes APIs, such as Ingress and NetworkPolicy, take.
For example, creation of an Ingress object in a Kubernetes cluster does not guarantee

action is taken. In other words, while the API exists, it is not core functionality.
Teams must consider what technology they’d like to plug in to implement this API.
For Ingress, many use a controller such as ingress-nginx, which runs in the cluster. It
implements the API by reading Ingress objects and creating NGINX configurations
for NGINX instances pointed at Pods. However, ingress-nginx is one of many
options. Project Contour implements the same Ingress API but instead programs
instances of envoy, the proxy that underlies Contour. Thanks to this pluggable model,
there are a variety of options available to teams.
Kubernetes Interfaces
Expanding on this idea of adding functionality, we should now explore interfaces.
Kubernetes interfaces enable us to customize and build on the core functionality. We
consider an interface to be a definition or contract on how something can be interac‐
ted with. In software development, this parallels the idea of defining functionality,
which classes or structs may implement. In systems like Kubernetes, we deploy plug-
ins that satisfy these interfaces, providing functionality such as networking.
A specific example of this interface/plug-in relationship is the Container Runtime
Interface (CRI). In the early days of Kubernetes, there was a single container runtime
supported, Docker. While Docker is still present in many clusters today, there is
growing interest in using alternatives such as containerd or CRI-O. Figure 1-2 dem‐
onstrates this relationship with these two container runtimes.
Figure 1-2. Two workload nodes running two different container runtimes. The kubelet
sends commands defined in the CRI such as CreateContainer and expects the runtime
to satisfy the request and respond.
In many interfaces, commands, such as CreateContainerRequest or PortForwardRe
quest, are issued as remote procedure calls (RPCs). In the case of CRI, the communi‐
cation happens over GRPC and the kubelet expects responses such as
CreateContainerResponse and PortForwardResponse. In Figure 1-2, you’ll also
notice two different models for satisfying CRI. CRI-O was built from the ground up
as an implementation of CRI. Thus the kubelet issues these commands directly to it.
Defining Kubernetes | 5

containerd supports a plug-in that acts as a shim between the kubelet and its own
interfaces. Regardless of the exact architecture, the key is getting the container run‐
time to execute, without the kubelet needing to have operational knowledge of how
this occurs for every possible runtime. This concept is what makes interfaces so pow‐
erful in how we architect, build, and deploy Kubernetes clusters.
Over time, we’ve even seen some functionality removed from the core project in favor
of this plug-in model. These are things that historically existed “in-tree,” meaning
within the kubernetes/kubernetes code base. An example of this is cloud-provider
integrations (CPIs). Most CPIs were traditionally baked into components such as the
kube-controller-manager and the kubelet. These integrations typically handled con‐
cerns such as provisioning load balancers or exposing cloud provider metadata.
Sometimes, especially prior to the creation of the Container Storage Interface (CSI),
these providers provisioned block storage and made it available to the workloads run‐
ning in Kubernetes. That’s a lot of functionality to live in Kubernetes, not to mention
it needs to be re-implemented for every possible provider! As a better solution, sup‐
port was moved into its own interface model, e.g., kubernetes/cloud-provider, that
can be implemented by multiple projects or vendors. Along with minimizing sprawl
in the Kubernetes code base, this enables CPI functionality to be managed out of
band of the core Kubernetes clusters. This includes common procedures such as
upgrades or patching vulnerabilities.
Today, there are several interfaces that enable customization and additional function‐
ality in Kubernetes. What follows is a high-level list, which we’ll expand on through‐
out chapters in this book:
• The Container Networking Interface (CNI) enables networking providers to
define how they do things from IPAM to actual packet routing.
• The Container Storage Interface (CSI) enables storage providers to satisfy intra-
cluster workload requests. Commonly implemented for technologies such as
ceph, vSAN, and EBS.
• The Container Runtime Interface (CRI) enables a variety of runtimes, common
ones including Docker, containerd, and CRI-O. It also has enabled a proliferation
of less traditional runtimes, such as firecracker, which leverages KVM to provi‐
sion a minimal VM.
• The Service Mesh Interface (SMI) is one of the newer interfaces to hit the Kuber‐
netes ecosystem. It hopes to drive consistency when defining things such as traf‐
fic policy, telemetry, and management.
• The Cloud Provider Interface (CPI) enables providers such as VMware, AWS,
Azure, and more to write integration points for their cloud services with Kuber‐
netes clusters.

• The Open Container Initiative Runtime Spec. (OCI) standardizes image formats
ensuring that a container image built from one tool, when compliant, can be run
in any OCI-compliant container runtime. This is not directly tied to Kubernetes
but has been an ancillary help in driving the desire to have pluggable container
runtimes (CRI).
Summarizing Kubernetes
Now we have focused in on the scope of Kubernetes. It is a container orchestrator,
with a couple extra features here and there. It also has the ability to be extended and
customized by leveraging plug-ins to interfaces. Kubernetes can be foundational for
many organizations looking for an elegant means of running their applications. How‐
ever, let’s take a step back for a moment. If we were to take the current systems used
to run applications in your organization and replace them with Kubernetes, would
that be enough? For many of us, there is much more involved in the components and
machinery that make up our current “application platform.”
Historically, we have witnessed a lot of pain when organizations hold the view of hav‐
ing a “Kubernetes” strategy—or when they assume that Kubernetes will be an ade‐
quate forcing function for modernizing how they build and run software. Kubernetes
is a technology, a great one, but it really should not be the focal point of where you’re
headed in the modern infrastructure, platform, and/or software realm. We apologize
if this seems obvious, but you’d be surprised how many executive or higher-level
architects we talk to who believe that Kubernetes, by itself, is the answer to problems,
when in actuality their problems revolve around application delivery, software devel‐
opment, or organizational/people issues. Kubernetes is best thought of as a piece of
your puzzle, one that enables you to deliver platforms for your applications. We have
been dancing around this idea of an application platform, which we’ll explore next.
Defining Application Platforms
In our path to production, it is key that we consider the idea of an application plat‐
form. We define an application platform as a viable place to run workloads. Like most
definitions in this book, how that’s satisfied will vary from organization to organiza‐
tion. Targeted outcomes will be vast and desirable to different parts of the business—
for example, happy developers, reduction of operational costs, and quicker feedback
loops in delivering software are a few. The application platform is often where we find
ourselves at the intersection of apps and infrastructure. Concerns such as developer
experience (devx) are typically a key tenet in this area.
Application platforms come in many shapes and sizes. Some largely abstract underly‐
ing concerns such as the IaaS (e.g., AWS) or orchestrator (e.g., Kubernetes). Heroku is
a great example of this model. With it you can easily take a project written in lan‐
guages like Java, PHP, or Go and, using one command, deploy them to production.
Defining Application Platforms | 7

Alongside your app runs many platform services you’d otherwise need to operate
yourself. Things like metrics collection, data services, and continuous delivery (CD).
It also gives you primitives to run highly available workloads that can easily scale.
Does Heroku use Kubernetes? Does it run its own datacenters or run atop AWS?
Who cares? For Heroku users, these details aren’t important. What’s important is del‐
egating these concerns to a provider or platform that enables developers to spend
more time solving business problems. This approach is not unique to cloud services.
RedHat’s OpenShift follows a similar model, where Kubernetes is more of an imple‐
mentation detail and developers and platform operators interact with a set of abstrac‐
tions on top.
Why not stop here? If platforms like Cloud Foundry, OpenShift, and Heroku have
solved these problems for us, why bother with Kubernetes? A major trade-off to
many prebuilt application platforms is the need to conform to their view of the world.
Delegating ownership of the underlying system takes a significant operational weight
off your shoulders. At the same time, if how the platform approaches concerns like
service discovery or secret management does not satisfy your organizational require‐
ments, you may not have the control required to work around that issue. Addition‐
ally, there is the notion of vendor or opinion lock-in. With abstractions come
opinions on how your applications should be architected, packaged, and deployed.
This means that moving to another system may not be trivial. For example, it’s signif‐
icantly easier to move workloads between Google Kubernetes Engine (GKE) and
Amazon Elastic Kubernetes Engine (EKS) than it is between EKS and Cloud Foundry.
The Spectrum of Approaches
At this point, it is clear there are several approaches to establishing a successful appli‐
cation platform. Let’s make some big assumptions for the sake of demonstration and
evaluate theoretical trade-offs between approaches. For the average company we work
with, say a mid to large enterprise, Figure 1-3 shows an arbitrary evaluation of
approaches.
In the bottom-left quadrant, we see deploying Kubernetes clusters themselves, which
has a relatively low engineering effort involved, especially when managed services
such as EKS are handling the control plane for you. These are lower on production
readiness because most organizations will find that more work needs to be done on
top of Kubernetes. However, there are use cases, such as teams that use dedicated
cluster(s) for their workloads, that may suffice with just Kubernetes.

Figure 1-3. The multitude of options available to provide an application platform to
developers.
In the bottom right, we have the more established platforms, ones that provide an
end-to-end developer experience out of the box. Cloud Foundry is a great example of
a project that solves many of the application platform concerns. Running software in
Cloud Foundry is more about ensuring the software fits within its opinions. Open‐
Shift, on the other hand, which for most is far more production-ready than just
Kubernetes, has more decision points and considerations for how you set it up. Is this
flexibility a benefit or a nuisance? That’s a key consideration for you.
Lastly, in the top right, we have building an application platform on top of Kuber‐
netes. Relative to the others, this unquestionably requires the most engineering effort,
at least from a platform perspective. However, taking advantage of Kubernetes exten‐
sibility means you can create something that lines up with your developer, infrastruc‐
ture, and business needs.

Aligning Your Organizational Needs
What’s missing from the graph in Figure 1-3 is a third dimension, a z-axis that dem‐
onstrates how aligned the approach is with your requirements. Let’s examine another
visual representation. Figure 1-4 maps out how this might look when considering
platform alignment with organizational needs.
Figure 1-4. The added complexity of the alignment of these options with your organiza‐
tional needs, the z-axis.
In terms of requirements, features, and behaviors you’d expect out of a platform,
building a platform is almost always going to be the most aligned. Or at least the most
capable of aligning. This is because you can build anything! If you wanted to re-
implement Heroku in-house, on top of Kubernetes, with minor adjustments to its
capabilities, it is technically possible. However, the cost/reward should be weighed
out with the other axes (x and y). Let’s make this exercise more concrete by consider‐
ing the following needs in a next-generation platform:
• Regulations require you to run mostly on-premise
• Need to support your baremetal fleet along with your vSphere-enabled
datacenter
• Want to support growing demand for developers to package applications in
containers

• Need ways to build self-service API mechanisms that move you away from
“ticket-based” infrastructure provisioning
• Want to ensure APIs you’re building atop of are vendor agnostic and not going to
cause lock-in because it has cost you millions in the past to migrate off these
types of systems
• Are open to paying enterprise support for a variety of products in the stack, but
unwilling to commit to models where the entire stack is licensed per node, core,
or application instance
We must understand our engineering maturity, appetite for building and empowering
teams, and available resources to qualify whether building an application platform is
a sensible undertaking.
Summarizing Application Platforms
Admittedly, what constitutes an application platform remains fairly gray. We’ve
focused on a variety of platforms that we believe bring an experience to teams far
beyond just workload orchestration. We have also articulated that Kubernetes can be
customized and extended to achieve similar outcomes. By advancing our thinking
beyond “How do I get a Kubernetes” into concerns such as “What is the current
developer workflow, pain points, and desires?” platform and infrastructure teams will
be more successful with what they build. With a focus on the latter, we’d argue, you
are far more likely to chart a proper path to production and achieve nontrivial adop‐
tion. At the end of the day, we want to meet infrastructure, security, and developer
requirements to ensure our customers—typically developers—are provided a solution
that meets their needs. Often we do not want to simply provide a “powerful” engine
that every developer must build their own platform atop of, as jokingly depicted in
Figure 1-5.
Figure 1-5. When developers desire an end-to-end experience (e.g., a driveable car), do
not expect an engine without a frame, wheels, and more to suffice.

Building Application Platforms on Kubernetes
Now we’ve identified Kubernetes as one piece of the puzzle in our path to production.
With this, it would be reasonable to wonder “Isn’t Kubernetes just missing stuff
then?” The Unix philosophy’s principle of “make each program do one thing well” is a
compelling aspiration for the Kubernetes project. We believe its best features are
largely the ones it does not have! Especially after being burned with one-size-fits-all
platforms that try to solve the world’s problems for you. Kubernetes has brilliantly
focused on being a great orchestrator while defining clear interfaces for how it can be
built on top of. This can be likened to the foundation of a home.
A good foundation should be structurally sound, able to be built on top of, and pro‐
vide appropriate interfaces for routing utilities to the home. While important, a foun‐
dation alone is rarely a habitable place for our applications to live. Typically, we need
some form of home to exist on top of the foundation. Before discussing building on
top of a foundation such as Kubernetes, let’s consider a pre-furnished apartment as
shown in Figure 1-6.
Figure 1-6. An apartment that is move-in ready. Similar to platform as a service options
like Heroku. Illustration by Jessica Appelbaum.
This option, similar to our examples such as Heroku, is habitable with no additional
work. There are certainly opportunities to customize the experience inside; however,
many concerns are solved for us. As long as we are comfortable with the price of rent
and are willing to conform to the nonnegotiable opinions within, we can be success‐
ful on day one.

Circling back to Kubernetes, which we have likened to a foundation, we can now look
to build that habitable home on top of it, as depicted in Figure 1-7.
Figure 1-7. Building a house. Similar to establishing an application platform, which
Kubernetes is foundational to. Illustration by Jessica Appelbaum.
At the cost of planning, engineering, and maintaining, we can build remarkable plat‐
forms to run workloads throughout organizations. This means we’re in complete con‐
trol of every element in the output. The house can and should be tailored to the needs
of the future tenants (our applications). Let’s now break down the various layers and
considerations that make this possible.
Starting from the Bottom
First we must start at the bottom, which includes the technology Kubernetes expects
to run. This is commonly a datacenter or cloud provider, which offers compute, stor‐
age, and networking. Once established, Kubernetes can be bootstrapped on top.
Within minutes you can have clusters living atop the underlying infrastructure. There
are several means of bootstrapping Kubernetes, and we’ll cover them in depth in
Chapter 2.
From the point of Kubernetes clusters existing, we next need to look at a conceptual
flow to determine what we should build on top. The key junctures are represented in
Figure 1-8.
Building Application Platforms on Kubernetes | 13

Figure 1-8. A flow our teams may go through in their path to production with
Kubernetes.
From the point of Kubernetes existing, you can expect to quickly be receiving ques‐
tions such as:
• “How do I ensure workload-to-workload traffic is fully encrypted?”
• “How do I ensure egress traffic goes through a gateway guaranteeing a consistent
source CIDR?”
• “How do I provide self-service tracing and dashboards to applications?”
• “How do I let developers onboard without being concerned about them becom‐
ing Kubernetes experts?”
This list can be endless. It is often incumbent on us to determine which requirements
to solve at a platform level and which to solve at an application level. The key here is
to deeply understand exiting workflows to ensure what we build lines up with current
expectations. If we cannot meet that feature set, what impact will it have on the devel‐
opment teams? Next we can start the building of a platform on top of Kubernetes. In
doing so, it is key we stay paired with development teams willing to onboard early
and understand the experience to make informed decisions based on quick feedback.
After reaching production, this flow should not stop. Platform teams should not
expect what is delivered to be a static environment that developers will use for deca‐
des. In order to be successful, we must constantly be in tune with our development
groups to understand where there are issues or potential missing features that could
increase development velocity. A good place to start is considering what level of inter‐
action with Kubernetes we should expect from our developers. This is the idea of how
much, or how little, we should abstract.

The Abstraction Spectrum
In the past, we’ve heard posturing like, “If your application developers know they’re
using Kubernetes, you’ve failed!” This can be a decent way to look at interaction with
Kubernetes, especially if you’re building products or services where the underlying
orchestration technology is meaningless to the end user. Perhaps you’re building a
database management system (DBMS) that supports multiple database technologies.
Whether shards or instances of a database run via Kubernetes, Bosh, or Mesos proba‐
bly doesn’t matter to your developers! However, taking this philosophy wholesale
from a tweet into your team’s success criteria is a dangerous thing to do. As we layer
pieces on top of Kubernetes and build platform services to better serve our custom‐
ers, we’ll be faced with many points of decision to determine what appropriate
abstractions looks like. Figure 1-9 provides a visualization of this spectrum.
Figure 1-9. The various ends of the spectrum. Starting with giving each team its own
Kubernetes cluster to entirely abstracting Kubernetes from your users, via a platform as
a service (PaaS) offering.
This can be a question that keeps platform teams up at night. There’s a lot of merit in
providing abstractions. Projects like Cloud Foundry provide a fully baked developer
experience—an example being that in the context of a single cf push we can take an
application, build it, deploy it, and have it serving production traffic. With this goal
and experience as a primary focus, as Cloud Foundry furthers its support for running
on top of Kubernetes, we expect to see this transition as more of an implementation
detail than a change in feature set. Another pattern we see is the desire to offer more
than Kubernetes at a company, but not make developers explicitly choose between
technologies. For example, some companies have a Mesos footprint alongside a
Kubernetes footprint. They then build an abstraction enabling transparent selection
of where workloads land without putting that onus on application developers. It also
prevents them from technology lock-in. A trade-off to this approach includes build‐
ing abstractions on top of two systems that operate differently. This requires signifi‐
cant engineering effort and maturity. Additionally, while developers are eased of the
burden around knowing how to interact with Kubernetes or Mesos, they instead need
to understand how to use an abstracted company-specific system. In the modern era
of open source, developers from all over the stack are less enthused about learning
systems that don’t translate between organizations. Lastly, a pitfall we’ve seen is an
obsession with abstraction causing an inability to expose key features of Kubernetes.

Over time this can become a cat-and-mouse game of trying to keep up with the
project and potentially making your abstraction as complicated as the system it’s
abstracting.
On the other end of the spectrum are platform groups that wish to offer self-service
clusters to development teams. This can also be a great model. It does put the respon‐
sibility of Kubernetes maturity on the development teams. Do they understand how
Deployments, ReplicaSets, Pods, Services, and Ingress APIs work? Do they have a
sense for setting millicpus and how overcommit of resources works? Do they know
how to ensure that workloads configured with more than one replica are always
scheduled on different nodes? If yes, this is a perfect opportunity to avoid over-
engineering an application platform and instead let application teams take it from the
Kubernetes layer up.
This model of development teams owning their own clusters is a little less common.
Even with a team of humans that have a Kubernetes background, it’s unlikely that
they want to take time away from shipping features to determine how to manage the
life cycle of their Kubernetes cluster when it comes time to upgrade. There’s so much
power in all the knobs Kubernetes exposes, but for many development teams, expect‐
ing them to become Kubernetes experts on top of shipping software is unrealistic. As
you’ll find in the coming chapters, abstraction does not have to be a binary decision.
At a variety of points we’ll be able to make informed decisions on where abstractions
make sense. We’ll be determining where we can provide developers the right amount
of flexibility while still streamlining their ability to get things done.
Determining Platform Services
When building on top of Kubernetes, a key determination is what features should be
built into the platform relative to solved at the application level. Generally this is
something that should be evaluated at a case-by-case basis. For example, let’s assume
every Java microservice implements a library that facilitates mutual TLS (mTLS)
between services. This provides applications a construct for identity of workloads and
encryption of data over the network. As a platform team, we need to deeply under‐
stand this usage to determine whether it is something we should offer or implement
at a platform level. Many teams look to solve this by potentially implementing a tech‐
nology called a service mesh into the cluster. An exercise in trade-offs would reveal
the following considerations.
Pros to introducing a service mesh:
• Java apps no longer need to bundle libraries to facilitate mTLS.
• Non-Java applications can take part in the same mTLS/encryption system.
• Lessened complexity for application teams to solve for.

Cons to introducing a service mesh:
• Running a service mesh is not a trivial task. It is another distributed system with
operational complexity.
• Service meshes often introduce features far beyond identity and encryption.
• The mesh’s identity API might not integrate with the same backend system as
used by the existing applications.
Weighing these pros and cons, we can come to the conclusion as to whether solving
this problem at a platform level is worth the effort. The key is we don’t need to, and
should not strive to, solve every application concern in our new platform. This is
another balancing act to consider as you proceed through the many chapters in this
book. Several recommendations, best practices, and guidance will be shared, but like
anything, you should assess each based on the priorities of your business needs.
The Building Blocks
Let’s wrap up this chapter by concretely identifying key building blocks you will have
available as you build a platform. This includes everything from the foundational
components to optional platform services you may wish to implement.
The components in Figure 1-10 have differing importance to differing audiences.
Figure 1-10. Many of the key building blocks involved in establishing an application
platform.

Some components such as container networking and container runtime are required
for every cluster, considering that a Kubernetes cluster that can’t run workloads or
allow them to communicate would not be very successful. You are likely to find some
components to have variance in whether they should be implemented at all. For
example, secret management might not be a platform service you intend to imple‐
ment if applications already get their secrets from an external secret management
solution.
Some areas, such as security, are clearly missing from Figure 1-10. This is because
security is not a feature but more so a result of how you implement everything from
the IAAS layer up. Let’s explore these key areas at a high level, with the understanding
that we’ll dive much deeper into them throughout this book.
IAAS/datacenter and Kubernetes
IAAS/datacenter and Kubernetes form the foundational layer we have called out
many times in this chapter. We don’t mean to trivialize this layer because its stability
will directly correlate to that of our platform. However, in modern environments, we
spend much less time determining the architecture of our racks to support Kuber‐
netes and a lot more time deciding between a variety of deployment options and top‐
ologies. Essentially we need to assess how we are going to provision and make
available Kubernetes clusters.
Container runtime
The container runtime will faciliate the life cycle management of our workloads on
each host. This is commonly implemented using a technology that can manage con‐
tainers, such as CRI-O, containerd, and Docker. The ability to choose between these
different implementations is thanks to the Container Runtime Interface (CRI). Along
with these common examples, there are specialized runtimes that support unique
requirements, such as the desire to run a workload in a micro-vm.
Container networking
Our choice of container networking will commonly address IP address management
(IPAM) of workloads and routing protocols to facilitate communication. Common
technology choices include Calico or Cilium, which is thanks to the Container Net‐
working Interface (CNI). By plugging a container networking technology into the
cluster, the kubelet can request IP addresses for the workloads it starts. Some plug-ins
go as far as implementing service abstractions on top of the Pod network.

Exploring the Variety of Random
Documents with Different Content

Marie Antoinette and Mary, Queen of Scots, stirred his imagination
most of all, and to the ill-fated Queen of Louis XVI he reverted so
often that it seemed the book was likely to be over-weighted with
matter dealing with her sad career, to the exclusion of so much else
of vital importance to our handbook.
Whenever he stood in front of the decapitated head of Marie
Antoinette he always contemplated it in silence—and invariably
passed from it without making any remark, as if it were a subject
too sad for ordinary comment.
“I have done the Marie Antoinette biography,” greeted me long
before the work had been definitely agreed upon, and six or seven
pages of essay were pressed into my hands as an accomplished
undertaking that positively left no room for further consideration.
This matter was printed in full in our Catalogue, and remained there
until the difficulty in procuring paper during the war necessitated its
temporary elimination. It is, perhaps, the best thing, from a purely
literary point of view, that Sala ever wrote.
It is reprinted as the following chapter.

GEORGE AUGUSTUS SALA
From a photograph.
CHAPTER XXXVII
G. A. SALA ON MARIE ANTOINETTE
The Royal Family—The Queen—Her “trial,”
condemnation and death—The Sansons—Sala’s
impressions.
There are some stories so dreadful
in the immensity of human misery
which they reveal—there are some
tragedies of which the catastrophe is
one of such unmitigated horror, that
the reader who has general
impressions of what will be the end of
the dismal tale, but who is unfamiliar
with its particular circumstances, is
unable to follow, without some kind of
impatience, the opening scenes of the
drama. He has continually in his
mind’s eye the awful falling of the
curtain on anguish and despair and
death. Half unconsciously he hastens
on in his perusal, and slurs over minor
episodes and seemingly trifling facts,
forgetting that these are subsidiary
and auxiliary to the terrible consummation which he so anxiously
awaits. “Toutes choses meuvent vers leur fin,” Rabelais has said; but
the little things—the slender fibres of a story—are gathered up as it
proceeds, into bundles; and, acquiring importance from
consolidation, are ultimately merged in the final and tremendous
whole.

Thus there have been many records of human life and action, now
real, now artificial, in reading which we have to encounter an almost
uncontrollable impulse to turn to the end, and ascertain whether
that of which we have had, at the beginning, a vague forecast, will
really come to pass. Who, if he will only have the candour to
acknowledge it, has not had to struggle with such an impulse in
reading, say, the Electra of Sophocles, the Faust of Goethe, and the
Bride of Lammermoor of Scott?—three of the most perfectly tragic
dramas, I take it, ever fashioned by the hand of mortal genius. And
so it is with numerous tragedies of superhuman structure and
ordinance. In both cases we pant for the last scene of all, which is to
end the strange eventful history. What will be the fate of Aegisthus,
and the doom of Clytemnestra? Who, if anyone, will rescue Gretchen
from a shameful death? How will Edgar Ravenswood bear his
immeasurable sorrow?
These are the problems which agitate us in the study of fiction,
and irresistibly impel us to hasten from the prologue to the epilogue
—from the exordium to the peroration. And to speed as quickly is
usually our desire when we are confronted with the tragedies of
history, or with the vouched-for chronicles of human passion and
crime. Throw down on the floor Clarendon’s History of the Rebellion,
it has been said, and the volume will open, automatically, at the
page where the execution of Charles I is described. Try to
concentrate your thoughts on the history of Marie Stuart; and,
coldly, clearly, sternly distinct in the midst of a whirligig of scenes
and events—the Louvre, Holyrood, the Kirk of Field, Lochleven and
what not—there stands out the image of the Hall at Fotheringay, the
black scaffold, the block, the masked headsman; the Dean of
Peterborough drearily homilising, and the Puritan Earl of Kent
ranting; while the weeping tire-women disrobe the royal victim, her
little pet dog snuggling by her, not without difficulty when the axe
has fallen to be dislodged from the corse of the kind mistress he
loved so well, and who has been stricken down by cruel men, he
knows not why. See this, as I see it.

It is my purpose to write something on the eventful life and
dreadful ending of Queen Marie Antoinette. I try, when I remember
the sunshine of her early days—her youth, her beauty, her grace—to
put myself in a cheerful frame of mind. I wish to look, at least for a
little while, on the bright side of a career which began so splendidly
and so happily. I would fain picture to myself the daughter of Maria
Theresa, as Edmund Burke saw her at Versailles—smiling, radiant,
adored. I would fain hear the clash of the thirty thousand swords
which should have leaped from their scabbards to avenge the
slightest affront to the peerless consort of the King of France and
Navarre.
I take from my shelves the Journal de Madame Eloff—the ledger
containing the milliner and dressmaker’s bills of a perhaps too
extravagant young Queen—an endless catalogue of taffetas and
satins, gauze and ribbons, high-heeled shoes and embroidered
gloves, scent-bottles, reticules, feathers, artificial flowers and fans.
From an old Boule cabinet I lift tenderly a dainty little coffee-cup of
Sèvres egg-shell porcelain, adorned with an exquisite miniature of
her, painted when she had only been two years the wife of the
hapless Louis. The cup is half embedded in a setting of velvet bleu
du Roi; and, alas! when I draw the ceramic gem delicately from the
case I see that the cup has no handle.
A maimed relic, this porcelain trifle, possibly of a priceless
breakfast set, wantonly shattered by a howling mob of poissardes
and red night-capped “patriots” who had sacked one of the Royal
Palaces. A crowd of memories are conjured up by this morsel of
dismembered Sèvres. I see, as in a glass darkly, the Galerie des
Glaces and the Œil-de-Boeuf at Versailles. I see the toy Dairy at the
Petit Trianon; the banquet of the Gardes du Corps in the Great
Theatre of the Palace; the King and Queen: the Royal Princesses
circulating among the guests and distributing white cockades among
them; while the musicians make the hall resound with the strains of
“Oh, Richard! Oh, mon Roi!”

No, surely, the age of Chivalry is not past, and thrice ten thousand
glaives will leap into the light to vindicate the outraged Majesty of
France. There’s no such thing! A confused picture—a panorama all
torn to shreds and splashed with mud and flecked with blood flows
before me. The Etats Genéraux have wed: the nobility sparkling in
velvet and plumes and golden broideries; the clergy brave in copes
and mitres and point lace: the “Tiers Etat,” all in sombre black,
short-cloaked, slouch-hatted, grave, preoccupied, looking
unutterable things. Among them looms, very real and portentous
indeed, a thick-set, pock-marked man, with an eye of fire. This is
Honore Gabriel Riquetti, rightly Comte de Mirabeau, but who has
broken with his order, and styling himself “Mirabeau Marchand de
Draps”—a retail clothier from Marseilles, forsooth! of about forty-
eight hours’ commercial standing—stalks among country notaries
and shopkeepers, farmers and shopkeepers as a Deputy of the Third
Estate.
But all these fade away from my field of vision. I set to studying
and balancing my rambling thoughts. I have to deal with Marie
Antoinette, Josephe-Jeanne de Lorraine, wife of Louis XVI, and who
was born, you will remember, at Vienna, on the 2nd of November,
1755, the very day of that earthquake at Lisbon in the occurrence of
which Dr. Johnson for a long time so resolutely refused to believe.
Would the doctor, I wonder, had he lived in 1793, have declined to
place credence in a newspaper report of what is now to be narrated
—an upheaval more dreadful and disastrous than any physical
convulsion of the earth’s crust? The tattered, muddy, gory panorama
fades into a murky nothingness. Then, out of the Valley of Shadows
there arises, terribly distinct and substantial, THIS—
It is a raw, chilly, marrow-searching day in the month of October,
1793. A spacious hall, known in this new and blessed era of
Universal Regeneration, and Unlimited Throat-Cutting, as the Salle
de la Liberté, in the Palais de Justice, hard by the prison of the
Conciergerie, has been swept and garnished for the trial of the
discrowned and desolate widow of “Louis Capet,” murdered on the
scaffold in the Place de la Révolution last January. In a dark and

filthy dungeon of that same Conciergerie Marie Antoinette has been
immured since August. The walls of the Salle de la Liberté have been
newly whitewashed—no voluptuous frescoes or oil painting in this
abode of Republican simplicity, if you please: only patriotic lime-
whiting and democratic glue—and the almost blinding glare of the
stark walls brings out in strong relief the dark green canopy
suspended over the heads of the Judges of the Revolutionary
Tribunal, who are five in number, the President being one, Hermann.
Above this precious conclave are the busts of Brutus—save the
mark!—and two recent Revolutionary notorieties: the infamous
Marat, deservedly done to death by Charlotte Corday and the
member of the Convention, Lepelletier de St. Fargeau, who had
voted for the death sentence on Louis XVI, and who immediately
afterwards was stabbed to death by an ex-Garde du Corps in an
eating house in the Palais National—once Palais Royal. The busts are
crowned with scarlet caps of liberty, adorned with monstrous tri-
coloured cockades, and are flanked by two huge oil lamps. There will
be need of the lamps; for the deliberation of the tribunal will
probably last far into the night.
The judges sit at a long table which, although shabby, is
somewhat pretentious in its upholstering, since the legs are of
mahogany, and fluted, and the brazen feet are fashioned in the
shape of griffin’s claws, and exhibit some traces of bygone gilding.
This table is yet extant, and forms part of the furniture of the Court
of Cassation, which at present holds its sittings in the old Salle de la
Liberté. The Public Accuser has his place in front of the President;
the jury—yes, this monstrous tribunal has a jury!—is to the left of
the judges; and to the right is the desk of the Counsel for the
defence. Behind him is the seat for the prisoners. A breast-high
balustrade separates the Court from the space set apart for the
public, which is ample enough, and is thronged, this dreary October
morning, by a motley crew of sans culottes, mechanics,
lamplighters, bargemen and coarse, loud-voiced women from the
markets, some of them known as “Tricoteuses” and “Furies of the
Guillotine.”

Between the balustrade and the body of the Court runs a long
gangway, at one extremity of which is a door, communicating by
means of a narrow staircase with the Gaol of the Conciergerie.
Up this staircase and through this door, and along this gangway,
and so through an opening of the balustrade into the criminal dock,
there is brought, between two gendarmes, a woman of middle age,
with abundant hair which has turned quite grey lately, and features
which retain a few—a very few—traces of former comeliness. She is
barely eight-and-thirty, and she looks full fifty. She is miserably clad
in an old, patched, threadbare gown of black serge, which has been
mended for her innumerable times by a compassionate girl named
Rosalie, the daughter of the gaoler. Her shoes are old, full of holes,
and down at heel. She wears black cotton stockings, and about her
shoulders is arranged a kind of tippet, or pelérine, of frayed white
muslin. As yet she wears no cap; and her long tresses have been
carefully dressed and oiled this morning by the pitying Rosalie.
Obviously, she is in mourning for her husband, sometime King of
France and Navarre; but the Revolutionary Tribunal knows nothing of
such titles, and in the Act of Accusation, which is read in a
monotonous sing-song by the Greffier, the prisoner is arraigned as
“Marie Antoinette, of Austria and Lorraine, widow of Louis Capet.”
The indictment goes on to say that the widow Capet has by her
crimes rendered herself the worthy compeer of Brunéhaut,
Fredegonde, and Catherine de Medicis; that since she has had her
abode in France she has been the scourge and bloodsucker of her
adopted country; and that even before “the Happy Revolution which
gave the French their sovereignty” she entered into political
correspondence with “the man calling himself King of Bohemia and
Hungary”—this is the Emperor of Austria her brother—that, in
conjunction with the brothers of Louis Capet, and “the execrable and
infamous Calonne” she had squandered the resources of France (the
fruit of the sweat of the people) in a dreadful manner, “to satisfy
inordinate pleasures and to pay the agents of her criminal intrigues.”

In another count of the indictment she is charged with being “an
adept in all sorts of crimes.” One of these “crimes” is, that on the
evening of the famous banquet to the Garde du Corps, and the
Regiment de Flanders, in the Opera House at Versailles, she, with
the King and a numerous and brilliant following, had passed
between the lines of tables, distributing white cockades to the
officers and encouraging them to trample the national or tri-coloured
cockade under foot.
“Prisoner,” thunders the President, “were you there when the band
played the air, ‘Oh, Richard, oh mon Roi’?”
“I do not recollect,” replies the Queen.
“Were you there when the toast of ‘The Nation’ was proposed and
refused?”
“I do not think that I was.”
“Did not your husband read his speech to the representatives to
you half-an-hour before he delivered it?”
“My husband had great confidence in me, and that made him read
his speech to me; but I made no observations.”
Fancy cutting a poor woman’s head off because her husband read
her a speech which he was about to deliver in public! Does Mr.
Gladstone, does Lord Randolph Churchill, does Sir William Harcourt,
I wonder, ever favour the domestic circle with such “fore-lectures” as
Dr. Furnival might call them?
A remarkable witness against Marie Antoinette is a ruffian named
Roussillon, who deposes that on the fatal Tenth of August when the
Tuileries was stormed by the mob, he saw under the Queen’s bed a
number of empty wine-bottles, “from which,” adds Roussillon, “I
concluded that she had herself distributed wine to the Swiss soldiers,
that these wretches in their intoxication might assassinate the
people.”
Another witness testifies that among the effects of the ex-Queen
found at the prison of the Temple was a satin riband bearing the gilt

image of a Heart with the inscription “Cor Jesu miserere nobis.”
Other testimony is to the effect that while the Queen and the
children were incarcerated in the Temple, after the execution of
Louis, the poor little Dauphin was placed at the top of the table by
his mother, and was served first; thus justifying the inference that
she ignored the Republic, One and Indivisible, and recognised her
young son as Louis XVII, and the successor of his murdered sire.
Another charge, an abominable charge, and one so monstrous as
to make it scarcely credible that it should be launched against a
woman and a mother, is that she had systematically sought to
corrupt the mind of the poor young prince. To this horrible allegation
she makes at first no answer. At length, when the charge is
repeated, she is moved to noble indignation, and exclaims: You
accuse me of an impossibility: “J’en appelle à toutes les mères.” I
appeal to all mothers. But the instinct of maternity seems to be dead
in all that hall of blood, and the beldames in the public tribunes only
yell and gibe at her.
Less revolting, but equally preposterous, is the evidence of one
Renée Mullet, a chambermaid who has been in service at Versailles,
and this hussey swears that one day, “in a moment of good humour,”
she asked the ci-devant Duc de Coigny whether the Emperor still
continued to wage war against the Turks; as in that case France
would soon be ruined, the Queen having sent her brother no less
than two hundred millions of livres, wherewith to carry on hostilities.
To this, according to the gossiping waiting woman, the Duke made
answer: “Thou art right enough. Two hundred millions have already
been spent, and we are not at the end of it yet.”
It is on such evidence as this—evidence not heavy enough to
detach a feather from a pigeon’s wing, not convincing enough to
prove a forty shilling debt, the wretched Marie Antoinette is at length
convicted. The President sums up, furiously, against her. The
advocates who defend her, Chauveau and Tronçon-Ducoudray have
little to say, to the point, and can only feebly plead for clemency to
be extended to her; and the jury, after deliberating for fifty-five

minutes, return a verdict affirming all the charges submitted to
them. Hermann calls on the accused to declare whether she has any
objection to make to the sentence of the law demanded by the
Public Accuser. Marie Antoinette bows her head in token of a
negative.
Then the tribunal, putting their bloodthirsty heads together for a
few minutes, condemn Marie Antoinette of Austria and Lorraine,
widow of Louis Capet to the punishment of Death, “and the
confiscation of all her property for the benefit of the Republic, the
sentence to be executed in the Square of the Revolution.” The
confiscation of all her property! When she was dead, an inventory
was taken of the few rags which she had left behind her in her cell
in the Conciergerie, and they were appraised at the magnificent sum
of nine livres, about seven and sixpence sterling. Nine livres all told!
In the second year of her marriage it was computed that the roll and
butter served every morning to each of her ladies of honour, cost
two thousand livres, or eighty pounds a year; and five thousand
livres was the annual charge for the bouillon, or beef-tea, kept hot
by day and by night for Madame Royale, who was a weakly child.
During the earlier portion of her imprisonment the unhappy Queen
had been supplied with body linen by the compassionate care of the
Marchioness of Stafford, the wife of the British Ambassador in Paris,
but there was no kindly Ambassadress to succour her in her last and
darkest days, and the only hand held forth in pity to this forlorn
daughter of the Cæsars was that of a gaoler’s daughter.
It was half past four on the morning of the sixteenth of October
when this infernal tribunal adjourned, and the Queen was conducted
back to her prison. Throughout the whole of her trial she had not
ceased to maintain a calm countenance; but at times she seemed to
be giving way to a feeling of sheer weary listlessness, and moved
her fingers on the bar of the dock before her, as though she was
playing on the harpsichord When she heard the sentence
pronounced, her features did not shew the slightest alteration; and
she walked from the hall erect and seemingly unmoved, gendarmes
with drawn swords before and behind her, and the beldames of the

fish-market and the rag-shops cursing and shrieking at her, just as
you may see them in Paul Delaroche’s noble picture.
So they took her back to a dungeon twelve feet long, eight feet
broad, four feet underground, with a grated window on a level with
the pavement. Into this wretched hole some scraps of the coarsest
food were brought her; but she was left under the incessant
supervision of a female prisoner and two soldiers. It is said that she
snatched a little sleep. On waking she asked one of the gendarmes
who had been present at the trial whether she had replied “with too
much dignity” to the question put to her. “I ask,” she added,
“because I overheard a woman say, See how haughty she still is.”
The woman who could have made such an observation must have
been one of the hags that Delaroche has painted.
At seven o’clock in the morning, the entire garrison of Paris was
under arms. Cannon were placed in all the public places; and at the
foot of every bridge from the Quay of the Conciergerie to the Place
de la Révolution, that magnificent area between the gardens of the
Tuileries, originally called the Place Louis XV, and now know as the
Place de la Concorde. At half-past eleven Marie Antoinette, dressed
in a white linen déshabille, was brought out from the prison. As
though she had been the commonest of malefactors she was made
to mount the charette, or open cart, the appointed tumbril of infamy.
At least the murderers of her husband had had the decency to allow
him the “luxury” of a hackney coach, when he was taken from the
Temple to the scaffold. Her hair had been cut short ere she left the
gaol, and what remained of her formerly luxuriant tresses was
tucked under a white mob-cap. Her hands were tied behind her
back.
Of the Queen in this deplorable plight there exists a very beautiful
statue executed by Lord Ronald Gower. On the right, in the tumbril,
was seated Sanson, the executioner, and on the left a
“constitutional” priest, that is to say, one who had taken the oath of
fealty to the Republic. To the ministrations of this “patriotic” cleric,
who was dressed in light grey coat and a bob-wig, Marie Antoinette

had in the first instance declined to listen; but she occasionally
spoke to him on her way to the fatal Place de la Révolution.
An immense mob, in which women were revoltingly numerous,
crowded the streets throughout the entire line of route insulting the
Queen and vociferating “Long live the Republic!” She seldom cast
her eyes on the populace, but from time to time looked with some
curiosity on the prodigious military force surrounding the cart.
Otherwise her attitude throughout this last dismal pilgrimage was
one of half torpid indifference.
As the cart traversed the Rue St. Honoré, the numbed faculties of
the Queen seemed momentarily to revive; and she examined with
some attention the multitudinous inscriptions of “Liberty” and
“Equality” over the shop-fronts.
It was as the vehicle turned the corner of the Rue St. Honoré into
that which is now the Rue Royale that the famous painter, David,
who, during the Reign of Terror, was a furious Jacobin and a friend
of Robespierre, but who was destined to become a Baron of the
Empire, and to paint the Coronation of Napoleon at Notre Dame,
was able from the balcony which he occupied in company with the
wife of a member of the Convention to make a sketch of Marie
Antoinette. The drawing has come down to us. The features of the
Martyr Queen are sharp and pinched, exhibiting no traces whatever
of former comeliness, and she looks fifty years of age. It may here
be mentioned that the illustrious and pure-minded English sculptor,
John Flaxman, when he visited Paris, after the Peace of Amiens,
resolutely refused to meet the artist who made the last sketch of
Marie Antoinette, and always spoke of him disdainfully as “David of
the bloodstained brush.”
The historians are divided in opinion as to the demeanor of Marie
Antoinette on the scaffold. Some say that she laid herself down on
the fatal plank with calm deliberation, and met her death with noble
fortitude, recalling Andrew Marvell’s superb lines on the execution of
Charles I:—

And while the armèd bands
Did clap their bloody hands,
He nothing common did, nor mean,
Upon that memorable scene;
Nor called the gods, in vulgar spite,
To vindicate his helpless might;
But, with his keener eye
The axe’s edge did try;
Then bowed his comely head
Down, as upon a bed.
Others narrate that the Queen ascended the steps of the scaffold
in great haste, and with apparent impatience, and turned her eyes
with much emotion towards the Palace of the Tuileries, the scene of
her former greatness, and that she made some slight resistance
before submitting to the executioner. My own impression is that she
was two-thirds dead—that the rigor mortis was upon her before she
reached the scaffold; that she was lifted out of the cart and half
carried to the guillotine, and that she did not give the headsman and
his assistants the slightest trouble.
It is, at all events, certain that at half past twelve her head was
severed from her body. One of the valets du bourreau, or
executioner’s men, lifted and showed the head streaming with blood,
from the four quarters of the scaffold, the mob meanwhile
screeching “Vive la République!” and it is asserted that a young man
who dipped his handkerchief in the blood, and pressed it with
veneration to his heart, was instantly apprehended. The corpse of
Marie Antoinette was immediately flung into a pit filled with
quicklime, in the graveyard of the Madeleine where the remains of
her husband had also been interred.
At the Restoration in 1814, diligent search was made for the ashes
of the King and Queen in the cemetery, on the site of which was
subsequently erected an Expiatory Chapel. Some half calcined bones

and a few scraps of cloth and linen were found; and these last
having been identified by experts as having been part of the apparel
of Louis XVI and Marie Antoinette, the relics with a considerable
quantity of the surrounding earth, were inhumed with much pomp
and solemnity, in the Royal Vault of the Cathedral of St. Denis.
Touching the executioner, it may be expedient to record that Marie
Antoinette was guillotined, not by Charles Henri Sanson, who
beheaded Louis XVI, but by his son, Henri, who died in Paris in
1840, aged seventy-three. The elder Sanson died only a few weeks
after he had executed Louis, and the Royalist historians maintain
that his death was hastened by remorse for the deed which he had
been constrained to commit, and that in his will he bequeathed a
considerable sum for the celebration of an annual Expiatory Mass.
But this is very doubtful. It has been shown, however, without the
possibility of doubt, that the Sanson family were of Florentine origin,
and that the ancestors of Charles Henri and of Henri Sanson came to
France in the train of Catherine de Medicis. For two hundred years,
without intermission, had members of this gloomy historic family
been executioners in ordinary to the city of Paris.
In addition to Marie Antoinette, the younger Sanson decapitated
the Queen’s sister-in-law, Madame Elisabeth, and the eloquent
advocate, Malesherbes, who undertook the defence of Louise XVI.
He likewise beheaded the Duke of Orléans (Philippe Égalité), and
last, but not least, Maximilien Robespierre. The so-called Memoirs of
the Sanson Family are more than half suspected to be mainly
apocryphal, and to have been written by one D’Olbreuse, a
bookseller’s hack; and, according to a writer in the Paris Temps, in
1875 the last of the Sansons was a remarkably mild, flaccid and
stupid old gentleman, who was certainly incapable of writing any
“Memoirs” whatever, since his own memory was hopelessly decayed,
and whose circumstances in his old age became so embarrassed
that he was arrested for debt, and confined in the prison of Clichy,
whence he only procured his enlargement by pawning the guillotine
itself for 4,000 francs!

Shortly after the conclusion of this singular transaction, a
murderer had to be executed, and the usual instructions were issued
by the Procureur General to Henri Sanson, to have his death dealing
apparatus ready on a certain morning in the Place de la Roquette. It
then became necessary to explain to the authorities that the fatal
machine was practically in the custody of My Uncle. Justice,
however, had to be satisfied, and the murderer’s head was duly cut
off on the appointed morning; but simultaneously with the signature
of the Minister of Justice of a draft for 4,000 francs to release the
hypothecated guillotine, there was issued an order dismissing
Sanson from his post.
And Marie Antoinette? I have drawn her picture as faithfully as I
could, not without much toil and more perplexity for the memoirs of
the period in which she lived and died absolutely bristle with
falsehoods, the inventions now of Royalist and now of Republican
writers. Comparatively few are the facts concerning her which have
been exactly ascertained and are altogether indisputable; whereas
the name of the unfounded assertions, the insinuations, the
hypotheses, and the downright lies, is legion. By some this most
unhappy woman has been represented as an angel of goodness and
purity, a faithful spouse, a fond parent, a kind mistress, and a most
pious and charitable princess. By others she has been depicted as a
crafty, unscrupulous and vindictive woman, as perfidious as Borgia
and profligate as Messalina.
This is no place in which to discuss at length a most intricate
question, all hedged about by obscurity, uncertainties and mysteries
which will, perhaps, never be solved. At all events, the story which I
have told of her trial and her last moments is true. For the rest, both
Royalists and Republicans agree that Marie Antoinette was born at
Vienna, in 1755, and was the daughter of Francis of Lorraine,
Emperor of Germany, and of Marie Theresa of Austria. In May, 1770,
she married the Dauphin Louis, who was grandson of Louis XV of
France, and who, in 1774, ascended the French throne as Louis XVI.
It would not seem that Marie Antoinette was absolutely beautiful, as
beautiful, say, as Queen Louisa of Prussia, or as the Empress

Eugene, still there is a tolerably unanimous consensus of opinion
that she was handsome, lively, amiable, and thoroughly kind-
hearted. It is possible that she may have been a little thoughtless in
her youth; and the ledgers of Madame Eloffe certainly show that, as
regards her toilet, Marie Antoinette was a most prodigal Queen. But
is it a mortal sin in a young, pretty and sprightly woman to spend a
good deal of money on dress? How many hundred dresses did our
chaste Queen Elizabeth leave behind her, in her wardrobe, at her
death?
It must be granted that when the dissensions of the Revolution
began, Marie Antoinette was on the Conservative side, and that she
tried her hardest to incline her husband to that side. Was it so very
unnatural that she should do so? Her brother, the Emperor Joseph,
used to say that “Royalty was his trade”; and poor Marie Antoinette
may have laboured under a similar persuasion. But the times were
very bad indeed for the “trade” of Royalty, and there arose a grim
conviction among the working millions that the best way of mending
matters was to dethrone, plunder, and murder their masters and
mistresses.
The influence of Marie Antoinette in the councils of Louis has
been, I should say, considerably exaggerated by her enemies. Her
husband, naturally disposed to concession, was by temper irresolute,
and he allowed himself to be led away by the course of events,
instead of striving to control and direct them. There can be little
doubt, either, that Marie Antoinette was one of the chief advisers of
the flight of the King and Royal Family to Varennes; and that
imprudent enterprise served, even more fiercely, to inflame the
public animosity against herself and her husband.
But again, I fail to see the criminality of this attempted escape.
The King and Queen knew well enough that the Revolutionists
intended to deprive them of their crowns, and, in all probability, of
their lives, they had no adequate armed force with which to resist
the mob. Were they not justified in running away? After the
deposition of Louis, all the elements of grandeur in the character of

Marie Antoinette began to manifest themselves. She showed the
greatest courage during the dastardly attacks made on the Royal
Family; and she appeared to be always more anxious for the safety
of her husband and children than for her own. She shared their
captivity with noble resignation, and her demeanour under the most
trying circumstances never lost an iota of its dignity. In the presence
of her judges her fortitude never forsook her; her burst of indignant
maternal feeling overawed even the butchers who were perverting
and burlesquing the law to bring her to the shambles; and her
behaviour in almost unparalleled misfortunes, has won for her not
only the pity and the sympathy, but the reverent admiration of
posterity.

CHAPTER XXXVIII
More sitters—Mr. John Burns walks and talks—We buy his only suit—Mr.
George Bernard Shaw has to work for his living—General Booth—
Four leading suffragettes—Christabel’s model “speaks”—The Channel
swimmer.
The most restless of all my sitters was the Right Honourable John
Burns, when he was plain John Burns.
I modelled him in the year 1889 or 1890, at the time of the great
Dock Strike. Mr. Burns was then throwing all his magnetic personality
into the cause of the workers, and he brought some of that
magnetic personality into my studio. Only in a technical sense did he
“sit” to me. He was walking and talking all the time.
These were very turbulent days, and Mr. Burns had figured in the
Trafalgar Square riots. Shipowners and shipbuilders—and everybody,
I imagine, having more than £500 a year—were the objects of his
implacable distrust. He was a younger and poorer man then.
Mr. Burns wore the blue reefer suit which had survived the
jostlings of many a crowd, but he did not bring to my studio the
famous straw hat of which so much was written in the Press at that
time. When I spoke to him about the hat he rather fenced the
question, and to this day I believe that hat to be somewhere in Mr.
Burns’s possession as a treasured souvenir of his stressful past. I
have never seen Mr. Burns wearing any other kind of clothes than
blue serge.
I struck a bargain with the dockers’ champion that he should let
me have the suit he was wearing with which to clothe his portrait in
the Exhibition, and so complete the realism of the model. Mr. Burns
demurred at first, and then it appeared he had an extremely good
reason for doing so. It was the only suit he possessed, and we

agreed that I should have it as soon as I provided him with a new
one to take its place on his own back.
Mr. Burns told the story of this transaction in reply to an
interrupter at a public meeting.
“Where did you get that suit?” asked the interrogator.
“I got it,” said Mr. Burns frankly, “from Madame Tussaud’s. When
my portrait was put in the Exhibition you may, or you may not, have
noticed that it was wearing my old suit. As I had no other clothes
the management gave me the suit I am wearing now, and I hope
you will agree that I made a pretty good bargain.”
The audience cheered the speaker and booed the heckler.
Mr. Burns’s portrait has been brought up to date since then, but it
still wears the old reefer suit, and the fact of this being out of the
fashion and rather skimpy only adds to the effectiveness of the
picture by recalling the working man the late Sir Henry Campbell-
Bannerman raised to Cabinet rank.
They tell me Mr. Burns is getting white, but when I modelled him
his hair was black and plentiful.
Judy commemorated the suit incident in the following verse,
depicting Burns making figure eights on the ice:
’Ave ye seen Johnny Burns
Strikin’ figgers on the hice?
’Ave ye seen his twists and turns?—
Sure, an’ can’t he do it nice!
In his Tussaud’s suit of navy blue
’N’ his famous old straw hat,
With his Hacmes ’n’ his knobstick too,
A reg’lar ’ristocrat!

A contrast to Mr. Burns, though possibly of similar socialistic
opinions, was Mr. George Bernard Shaw, whom I long wanted to sit
to me.
I had not made the acquaintance of the brilliant satirist, and
somehow hesitated about approaching him. Eventually I wrote to Mr.
Shaw making known my wish, and, without delay, I received from
him a good-humoured letter, in which he said that it would give him
much pleasure to “join the company of the Immortals.”
A little later he wrote making an appointment, and, in due course,
Mr. Shaw came to my studio and gave me a delightful hour of his
company.
He took up his position on the dais in the most natural manner,
and there was nothing more for me to do than proceed with my
modelling. I do not know who was the more amused, Mr. Shaw or
myself—I by his sayings, and he by the novelty of the situation.
He talked freely as I went on with my work, and one thing among
his many whimsical sayings I well remember:
“I took to writing with the object of obtaining a living without
having to work for it, but I have long since realised that I made a
great mistake.”
As we walked through the Exhibition he took a general interest in
all he saw, but it was the Napoleonic relics that detained him, as is
generally the case with distinguished people.
I thought I detected a certain shyness about Mr. Shaw in the
Chamber of Horrors. He was very reserved, and surveyed the faces
of degenerate men and women without offering any criticism. I
remember that the crafty, and yet not wholly repulsive, face of
Charles Peace engaged Mr. Shaw’s attention several minutes.
I have no knowledge whether Mr. Shaw ever called to see his
portrait. It is quite likely that he did, and it is no less likely that his
visit passed unobserved.

It was inevitable that so prominent a figure in the religious world
as the late General Booth should find a place in Madame Tussaud’s
Exhibition.
I went to see the General at the instance of some of his friends,
who thought that the portrait of him already included would be all
the better for being brought up to date. I recollect being impressed
by General Booth’s force of character as manifested alike in his
manner and in his appearance. He had a keen eye and classic
aquiline features.
Though he made no mention of the matter himself, it was pretty
plainly hinted to me that permission to include the General’s portrait
should be accompanied by some expression of gratitude on the part
of the Exhibition authorities “for the good of the cause.”
I also went to Exeter Hall to study the General’s demeanour while
addressing a large audience.
What I remember mostly about that visit was that a “converted”
sailor mounted the platform and made a rambling speech. So frank
were the confessions of the artless tar that General Booth found it
necessary to bundle him unceremoniously off the platform, to the
great amusement of the congregation.
I was much interested in modelling a quartette of leading
suffragettes, Mrs. Pankhurst, Mrs. Pethick Lawrence, Miss Christabel
Pankhurst, and Miss Annie Kenney.
The group is conspicuously shown in the Grand Hall to-day. The
ladies came separately, several mornings, and took as much interest
as I did in the production of their portraits, a process that was in no
sense tedious, as their conversation whiled away the time most
pleasantly.
I very soon became aware that the suffragette on the political
warpath is a very different woman from the suffragette in other
circumstances.

None of them in the least degree frightened me or hectored me;
in fact, political questions were discussed by them in the quietest,
most sensible, and most intelligent manner, giving me the
impression then that the extension of the vote to women would not
find such women unqualified to make reasonable use of the privilege
so long withheld from them.
After the figures were added to the Exhibition, two of the four
ladies very good-humouredly hinted to me that the portraits were
not very flattering. I remember the ladies in question coming to see
the group, and I promised I would make what alterations seemed
possible and desirable. As I have not heard from them since, I
gather that the likenesses have proved satisfactory.
Months later, after a batch of laughing damsels had left the
building, a paper disc, bearing the words “Votes for Women,” was
discovered fixed to a button on Mr. Asquith’s coat.
It was soon after the figures of the quartette had been placed in
the Exhibition that an incident occurred which comes to me through
the medium of a Fleet Street artist in black and white attached to a
well-known paper.
This gentleman had been instructed to attend a meeting some
distance away from town for the purpose of taking some sketches of
Miss Christabel Pankhurst, who was announced to speak. Having left
things till the last moment, he discovered, to his dismay, that he had
missed his train, and, not knowing what to do, he was bewailing his
misfortune to a fellow artist, when the latter slapped him on the
back and said:
“Never mind, old fellow, you just go to Tussaud’s Exhibition and
take as many pictures of the fair Christabel’s figure as you like. The
model is a speaking likeness, and you can take it from me that the
sketches will be all right; they will be quite as good as if drawn from
life.”
The advice was no sooner given than acted upon, and the result, I
am told, was most satisfactory.

Another sitter was Mr. T. W. Burgess, who came to my studio a
few days after he swam the Channel.
The burly Yorkshireman laughed as he entered and remarked:
“I am in pretty good training, but I would rather swim the Channel
again than sit still for you, Mr. Tussaud. However, I will do the best I
can.”
He sold the clothes he took off before he entered the water, and
these clothes are worn by his portrait, now in the Exhibition. He also
parted with the goggles and indiarubber cap he had worn during his
swim, and the cup from which he took nourishment. Unfortunately
one of Burgess’s too ardent “admirers” purloined his hero’s cup from
us.

T. W. BURGESS, THE CHANNEL SWIMMER
Modeled from life by John T. Tussaud. In
common with many of the models in Madame
Tussaud’s, this model is dressed in the
subject’s own clothing.

CHAPTER XXXIX
Bank Holiday queues—Cup-tie day—Gentlemen from the north—Bachelor
beanfeasts—The Member for Oldham—A scare.
The four regular Bank Holidays of the year are great occasions at
Madame Tussaud’s.
On each of them the precincts of Tussaud’s show signs of activity
long before the average Londoner is astir. The length of any of the
queues has never been actually measured, but it is no exaggeration
to say that the people have frequently waited four and five deep in a
line extending almost a quarter of a mile—from the doors of the
Exhibition to the gates of Regent’s Park.
The crowd at these times consists mainly of Londoners from all
the outlying districts of the Metropolis, for Madame Tussaud’s has
always been in great favour as a holiday resort for the multitude.
Parents also bring their children in great numbers, and the holiday
crowds continue to come for days after.
There is, however, at least one morning in the year when the
portals of the Exhibition are literally teeming with life while the
citizens are slumbering in bed.
On Easter Monday, Whit-Monday, the August Bank Holiday, and
even on Boxing Day, holiday-makers may be seen at an early hour
waiting in a queue, yet no comparison may be made between these
crowds and those of the Cup-tie mornings I have witnessed at the
Exhibition.
This day brings into London tens of thousands of men and boys
from the densely populated manufacturing towns and mining areas
of Lancashire, Yorkshire, Durham, and Northumberland. These
football enthusiasts arrive in the Metropolis as early in the morning

as two, three, and four o’clock on the day of the Crystal Palace
carnival.
It has always seemed to me that Madame Tussaud’s has received
the lion’s share of patronage during the long interval between the
arrival of the cheap excursion trains at the great railway stations and
the time when the Cup-tie is played in the afternoon. The
attendance at these hours is extraordinary, and the appearance of a
house of entertainment in full swing so early in the morning has an
indescribably weird and garish effect.
These north country patrons of ours take up position on the steps
of the entrance, and pass the time taking refreshments brought with
them from their homes. Though weary with their journey, they are
always cheery and well-behaved, and the way in which they banter
each other in the broad accents of Oldham, Manchester, Leeds,
Bradford, Sheffield, Halifax, Newcastle, etc., has many a time
afforded me a good deal of interest and diversion.
I have often stood on the broad open staircase and looked down
upon the swarming hundreds in the entrance-hall and the
refreshment rooms and it is a happy experience to dwell on that
there has never been occasion to rebuke any of them for roughness
or want of good behaviour. It is peculiarly true of the country cousin,
so far as my experience of him goes, that he never indulges in
horse-play when he comes to Madame Tussaud’s.
There is, however, one very striking contrast between the crowd
on a Bank Holiday and that on a Cup-tie day, and this is due to the
circumstances that the followers of football do not bring their
women-folk or children with them on the occasion of these
“bachelor” beanfeasts—a concession, I presume, made to their men
by the wives and sweethearts of the north.
Not by a long way do all these excursionists go to see the great
football finals at the Palace. Quite a large proportion, taking
advantage of the cheap fares, come to see London and its many
sights which the average Londoner proverbially overlooks.

It has more than once been remarked by the Exhibition attendants
that many Cup-tie visitors spend the greater part of the day at
Madame Tussaud’s, lingering for hours among the relics of Napoleon
and the figures and exhibits of the Chamber of Horrors, without
having the slightest intention of venturing so far as to see the
football contest played.
It is a mistake to imagine that the working classes of the north are
ignorant of English history, or not concerned with it; and if that
impression exists, I should like to correct it. I doubt whether any
class takes a keener interest in the Hall of Kings, or makes more use
of the information provided by the Catalogue.
The “trippers,” “country cousins,” or whatever one likes to call
them, seldom pester the Exhibition attendants with queries, for what
one does not know another does. The Catalogues are taken away for
further perusal, and one may often search the whole Exhibition in
vain the next morning for one that has been discarded.
All day long groups of Cup-tie trippers stand about the Sleeping
Beauty, not only for her sake, but also for the sake of Madame
Tussaud, whose figure stands at Madame St. Amaranthe’s head,
while at her feet sits William Cobbett, wearing his old beaver hat,
and holding in his hand the snuff-box which legend credits him with
passing to visitors on some weird occasions.
Men from Oldham naturally show special interest in Cobbett, who
was, in his day, Member of Parliament for that town.
Cobbett sits on a red upholstered ottoman, with room enough for
two other persons, and on a certain Cup-tie day two travel-stained,
tired men sat down by him, and, noticing that he moved his head
from side to side, took him to be alive. They addressed questions to
him, and jumped up very hurriedly as he jerked his head and looked
blankly at them through his horn spectacles.
The only two figures in the Exhibition that make any pretence of
life are William Cobbett and the Sleeping Beauty.

A wonderful self-made man was Cobbett, who began life as a
living scarecrow, armed with a shotgun, in the employment of a
farmer, and, after being, among other things, sergeant-major won a
great reputation as a writer of English prose and attained the
distinction of adding M.P. to his name in those days when
Parliamentary honours were less easily achieved than they are to-
day.
To be sure, the figures of statesmen have always interested Cup-
tie crowds, for the provincial is much more of a politician than the
Londoner.
So also literary men like Scott, Dickens, Tennyson, Burns, and
Kipling come in for much attention; more, perhaps, than portraits of
the clergy.
Sportsmen, too, such as W. G. Grace, Fred Archer, and “Tommy
Lipton”—the last-mentioned for his America Cup performances—
receive enough notice on Cup-tie days to maintain a good average of
appreciation for the year.
As on Bank Holidays, so on Cup-tie days, there are always many
more live than wax figures in the Chamber of Horrors from morning
till night. Indeed, I have seen the place so crowded that it was
difficult to distinguish the effigies from the awestricken observers.
Sometimes I have taken a walk round the Exhibition after it was
closed on the night of the Cup-tie to see that all was right. Once I
was called in haste to the Chamber of Horrors, where a stranger had
been found asleep in a dark corner. After he had been roused and
escorted outside, the scared fellow made off as if he had had the
hangman at his heels. A return ticket from Bolton was picked up
where he had lain. But the man from Bolton had bolted, and did not
return to claim the ticket.

CHAPTER XL
The mysterious Sun Yat Sen’s visit—His escape from the Chinese Legation
—The Dargai tableau—Sir William Treloar entertains his little friends.
Once in its long history Madame Tussaud’s Exhibition opened on a
Sunday—not, however, to the general public.
The occasion was special and, in a way, mysterious. It had to do
with one of the most dramatic personalities of the Chinese Empire
and Republic.
A message reached me late on a Saturday night that Dr. Sun Yat
Sen, the first President of the Chinese Republic, wished to visit the
Exhibition on the following Sunday morning. I was unable to receive
him in person, but arranged that an attendant should represent me.
The attendant knew nothing of the name of the visitor till he saw
him looking at his own portrait and calling the attention of General
Homer Lee—an American soldier holding high rank in the Chinese
Army—who accompanied him, to the dimple in the chin of the model
by placing his finger smilingly on the dimple in his own chin.
This was in the year 1911, and Sun Yat Sen was passing through
London on his way from America to take up his presidential duties.
His visit to the Exhibition had been planned by Dr. (now Sir James)
Cantlie, of Harley Street, to whom Sun Yat Sen owed—the greatest
of all debts of gratitude—his life.
For it was this same Sun Yat Sen who, eleven years before, was
liberated through the exertions of Dr. Cantlie from his prison in the
Chinese Legation at Portland Place, a few minutes’ walk from
Madame Tussaud’s.
What would have happened to him but for the fact that Dr.
Cantlie’s intervention resulted in Sun Yat Sen’s release through Lord

DR. SUN YAT SEN
From a photograph.
Salisbury’s representations to the Chinese
authorities can only be conjectured.
It was discovered at the time that a ship
had been chartered in the Thames for the
removal of Sun Yat Sen to China on a
charge of treason against the Emperor—
the same Emperor whose successor, under
a republican form of government, Sun Yat
Sen was destined to be.
Particulars were also disclosed regarding
the manner of his incarceration at the
Chinese Legation. He was inveigled into
the place by the lures of hospitality, and,
once inside, the officials relegated him to
an apartment which they kept locked for
many days.
It was only through Sun Yat Sen’s friendship with Dr. Cantlie,
whose suspicions were aroused by “inside” information, that the
British authorities learned of Sun Yat Sen’s fate and took steps to
have him set free.

DR. SUN YAT SEN
The wax model on view at Madame Tussaud’s
of the first President of the Chinese Republic.
When the hero of this adventure visited Madame Tussaud’s on the
Sunday morning in question to see his model, I wondered what his
reason could be, and asked myself whether it had anything to do
with the adapting of his disguise, while travelling from this country
to China, at a time when his life must have been in danger.
Perhaps, after all, it was nothing more than the natural curiosity
which attracts people whose portraits have been recently added to
come and see them. The Eastern mind may not differ from the
Western in this very human respect.

Touching and dramatic in the extreme was the incident which
accompanied the unveiling of the tableau representing the Gordon
Highlanders storming the Heights of Dargai. Lieutenant-Colonel
Mathias’s words were on all lips at the time:
“That position must be taken at any cost; the Gordon Highlanders
will take it.”
Mrs. Mathias was present with her son and daughter at the supper
we gave to celebrate the event, and a piper played “The Cock of the
North” to recall the deed of the wounded piper who fired his
comrades on to victory and was awarded the V.C. When his father’s
words were recited, young Mathias sprang to his feet and thrilled all
present by saluting in true military fashion.
One of the brightest of red-letter days in Madame Tussaud’s
romantic story was the 24th of January, 1907, when Sir William
Treloar, “the children’s Mayor,” accompanied by several local Mayors,
drove to the Exhibition in all the panoply of civic state to give éclat
to the visit of fifteen hundred boys and girls of the poorest of the
poor, whom we made our guests.
How richly the Right Honourable the Lord Mayor of London
enjoyed himself on that occasion, like the large-hearted man he is,
and how pre-eminently happy he was among the waifs and strays,
many of whom were cripples, whose lives he has done so much to
brighten! Sir John Kirk, of the Ragged School Union, was also there,
beaming with joy among his little beneficiaries. I remember Sir
William Treloar pointing to his civic headgear and calling out to the
children, “How do you like my Dick Turpin hat?”
Tea-tables were laid all among the figures, and the picture
produced in this way was both striking and amusing as the young
people laughed and chatted by the side of the approving mutes.
Perhaps the remark which seemed to create the greatest fun was
when the Lord Mayor said he would like to see his Sheriffs in the
Chamber of Horrors.

THE CHILDREN’S LORD MAYOR
Sir William Treloar entertains his little friends
at Madame Tussaud’s, 24th January, 1907.
It was very touching
to observe the boys
loyally and reverently
take off their caps in
front of the little alcove
in which Queen Victoria
sits, as someone has
said, “signing
despatches all day
long.” At the close of
the happy day the halls
and corridors of the
Exhibition rang with the
shrill treble of fifteen
hundred young voices
singing “For he’s a jolly
good fellow,” followed
by “Hip hip, hooray;
the donkey’s run away.”
A tragedy happened
that day not far away,
in Westbourne Grove,
which caused the
gentlemen of the Press
who attended the
function to leave the
Exhibition rather
hurriedly. News came of the murder of Mr. William Whiteley, the
Universal Provider.

CHAPTER XLI
A miscellany of humour—Our policeman—The mysterious lantern—The
danger of old Catalogues—Stories of children—Sir Ernest
Shackleton’s model.
Many of our visitors will remember the model of the policeman
which stands at the entrance to the main gallery in the Exhibition.
Hundreds—I might say thousands—of visitors have been “taken in”
by this lifelike officer, who is the embodiment of a genial bobby
prepared at any moment to show the way or tell the time.
The fame of this nameless policeman has extended to practically
all the grown-ups who bring their children to see the figures, and
many times in the day we see laughing parents watching the
nonplussed expression on the faces of their offspring whom they
have prevailed upon to go and ask where a certain model is to be
found.
Immediately opposite is the figure of the programme-seller in
somnolent mood, who is frequently offered sixpence for a Catalogue
she cannot sell. It is the would-be customer that is sold.
It is most amusing to observe how many adults are deceived who
seem to pride themselves on their discernment. For example, on
Bank Holidays it is customary to have a number of real live
constables on duty to regulate the crowd and give directions.
Bobby has a keen sense of humour, and some of them, entering
into the spirit of the situation, now and again stand stock-still in the
most natural attitude they can command. Not once, but frequently, a
visitor, in passing with his friends, has, with an air of superior
knowledge, pushed the ferrule of his stick or umbrella into the
supposed figure’s side, to be startled by the model’s ejaculating,
“Now then, young man, enough of that.”

Production Kubernetes: Building Successful Application Platforms 1st Edition Josh Rosso

More Related Content

Similar to Production Kubernetes: Building Successful Application Platforms 1st Edition Josh Rosso (20)

Recently uploaded (20)

Production Kubernetes: Building Successful Application Platforms 1st Edition Josh Rosso