![978-1-492-04653-0
[LSI]
Kubernetes: Up and Running
by Brendan Burns, Joe Beda, and Kelsey Hightower
Copyright © 2019 Brendan Burns, Joe Beda, and Kelsey Hightower. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.
O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are
also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional
sales department: 800-998-9938 or corporate@oreilly.com.
Acquisition Editor: John Devins
Development Editor: Virginia Wilson
Production Editor: Kristen Brown
Copyeditor: Kim Cofer
Proofreader: Rachel Head
Indexer: Ellen Troutman-Zaig
Interior Designer: David Futato
Cover Designer: Karen Montgomery
Illustrator: Rebecca Demarest
September 2017: First Edition
August 2019: Second Edition
Revision History for the Second Edition
2019-07-15: First Release
See http://oreilly.com/catalog/errata.csp?isbn=9781492046530 for release details.
The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Kubernetes: Up and Running, the cover
image, and related trade dress are trademarks of O’Reilly Media, Inc.
The views expressed in this work are those of the authors, and do not represent the publisher’s views.
While the publisher and the authors have used good faith efforts to ensure that the information and
instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility
for errors or omissions, including without limitation responsibility for damages resulting from the use of
or reliance on this work. Use of the information and instructions contained in this work is at your own
risk. If any code samples or other technology this work contains or describes is subject to open source
licenses or the intellectual property rights of others, it is your responsibility to ensure that your use
thereof complies with such licenses and/or rights.
This work is part of a collaboration between O’Reilly and Microsoft. See our statement of editorial inde‐
pendence.](https://image.slidesharecdn.com/5275662-250621163510-5e401873/85/Kubernetes-Up-And-Running-Dive-Into-The-Future-Of-Infrastructure-2nd-Brendan-Burns-8-320.jpg)
![# Copy package files and install dependencies
COPY package*.json ./
RUN npm install
# Copy all of the app files into the image
COPY . .
# The default command to run when starting the container
CMD [ "npm", "start" ]
Every Dockerfile builds on other container images. This line specifies that we are
starting from the node:10 image on the Docker Hub. This is a preconfigured
image with Node.js 10.
This line sets the work directory, in the container image, for all following
commands.
These two lines initialize the dependencies for Node.js. First we copy the package
files into the image. This will include package.json and package-lock.json. The RUN
command then runs the correct command in the container to install the neces‐
sary dependencies.
Now we copy the rest of the program files into the image. This will include every‐
thing except node_modules, as that is excluded via the .dockerignore file.
Finally, we specify the command that should be run when the container is run.
Run the following command to create the simple-node Docker image:
$ docker build -t simple-node .
When you want to run this image, you can do it with the following command. You
can navigate to http://localhost:3000 to access the program running in the container:
$ docker run --rm -p 3000:3000 simple-node
At this point our simple-node image lives in the local Docker registry where the
image was built and is only accessible to a single machine. The true power of Docker
comes from the ability to share images across thousands of machines and the broader
Docker community.
Optimizing Image Sizes
There are several gotchas that come when people begin to experiment with container
images that lead to overly large images. The first thing to remember is that files that
are removed by subsequent layers in the system are actually still present in the
images; they’re just inaccessible. Consider the following situation:
18 | Chapter 2: Creating and Running Containers](https://image.slidesharecdn.com/5275662-250621163510-5e401873/85/Kubernetes-Up-And-Running-Dive-Into-The-Future-Of-Infrastructure-2nd-Brendan-Burns-42-320.jpg)
![# Do the build. This script is part of incoming sources.
RUN build/build.sh
CMD [ "/go/bin/kuard" ]
This Dockerfile produces a container image containing a static executable, but it also
contains all of the Go development tools and the tools to build the React.js frontend
and the source code for the application, neither of which are needed by the final
application. The image, across all layers, adds up to over 500 MB.
To see how we would do this with multistage builds, examine this multistage Docker‐
file:
# STAGE 1: Build
FROM golang:1.11-alpine AS build
# Install Node and NPM
RUN apk update && apk upgrade && apk add --no-cache git nodejs bash npm
# Get dependencies for Go part of build
RUN go get -u github.com/jteeuwen/go-bindata/...
RUN go get github.com/tools/godep
WORKDIR /go/src/github.com/kubernetes-up-and-running/kuard
# Copy all sources in
COPY . .
# This is a set of variables that the build script expects
ENV VERBOSE=0
ENV PKG=github.com/kubernetes-up-and-running/kuard
ENV ARCH=amd64
ENV VERSION=test
# Do the build. Script is part of incoming sources.
RUN build/build.sh
# STAGE 2: Deployment
FROM alpine
USER nobody:nobody
COPY --from=build /go/bin/kuard /kuard
CMD [ "/kuard" ]
This Dockerfile produces two images. The first is the build image, which contains the
Go compiler, React.js toolchain, and source code for the program. The second is the
deployment image, which simply contains the compiled binary. Building a container
image using multistage builds can reduce your final container image size by hundreds
of megabytes and thus dramatically speed up your deployment times, since generally,
Multistage Image Builds | 21](https://image.slidesharecdn.com/5275662-250621163510-5e401873/85/Kubernetes-Up-And-Running-Dive-Into-The-Future-Of-Infrastructure-2nd-Brendan-Burns-45-320.jpg)
Kubernetes Up And Running Dive Into The Future Of Infrastructure 2nd Brendan Burns
- 1. Kubernetes Up And Running Dive Into The Future Of Infrastructure 2nd Brendan Burns download https://ebookbell.com/product/kubernetes-up-and-running-dive- into-the-future-of-infrastructure-2nd-brendan-burns-10551324 Explore and download more ebooks at ebookbell.com
- 2. Here are some recommended products that we believe you will be interested in. You can click the link to download. Kubernetes Up And Running Dive Into The Future Of Infrastructure 1st Edition Kelsey Hightower https://ebookbell.com/product/kubernetes-up-and-running-dive-into-the- future-of-infrastructure-1st-edition-kelsey-hightower-6778206 Kubernetes Up And Running Dive Into The Future Of Infrastructure 2nd Edition Brendan Burns https://ebookbell.com/product/kubernetes-up-and-running-dive-into-the- future-of-infrastructure-2nd-edition-brendan-burns-10790404 Kubernetes Up And Running 3rd Edition Second Early Release Brendan Burns Joe Beda Kelsey Hightower Lachlan Evenson https://ebookbell.com/product/kubernetes-up-and-running-3rd-edition- second-early-release-brendan-burns-joe-beda-kelsey-hightower-lachlan- evenson-36865060 Kubernetes Up And Running Third Edition 3rd Edition Brendan Burns https://ebookbell.com/product/kubernetes-up-and-running-third- edition-3rd-edition-brendan-burns-50629896
- 3. Kubernetes Up And Running Brendan Burns Joe Beda Kelsey Hightower https://ebookbell.com/product/kubernetes-up-and-running-brendan-burns- joe-beda-kelsey-hightower-232081722 Kubernetes Up And Running Kelsey Hightower Brendan Burns Joe Beda https://ebookbell.com/product/kubernetes-up-and-running-kelsey- hightower-brendan-burns-joe-beda-47879626 Kubernetes Up And Running Third Edition 3rd Edition Brendan Burns https://ebookbell.com/product/kubernetes-up-and-running-third- edition-3rd-edition-brendan-burns-50629890 Kubernetes Up And Running Brendan Burns https://ebookbell.com/product/kubernetes-up-and-running-brendan- burns-170495086 Linkerd Up And Running A Guide To Operationalizing A Kubernetesnative Service Mesh 1st Edition Jason Morgan https://ebookbell.com/product/linkerd-up-and-running-a-guide-to- operationalizing-a-kubernetesnative-service-mesh-1st-edition-jason- morgan-56773602
- 5. Brendan Burns, Joe Beda & Kelsey Hightower Kubernetes Up & Running Dive into the Future of Infrastructure S e c o n d E d i t i o n
- 6. Find out what you can do with a fully managed service for simplifying Kubernetes deployment, management and operations, including: • Build microservices applications. • Deploy a Kubernetes cluster. • Easily monitor and manage Kubernetes. Create a free account and get started with Kubernetes on Azure. Azure Kubernetes Service (AKS) is one of more than 25 products that are always free with your account. Start free > Then, try these labs to master the basic and advanced tasks required to deploy a multi- container application to Kubernetes on Azure Kubernetes Service (AKS). Try now > Get started Kubernetes on Azure
- 7. Brendan Burns, Joe Beda, and Kelsey Hightower Kubernetes: Up and Running Dive into the Future of Infrastructure SECOND EDITION Boston Farnham Sebastopol Tokyo Beijing Boston Farnham Sebastopol Tokyo Beijing
- 8. 978-1-492-04653-0 [LSI] Kubernetes: Up and Running by Brendan Burns, Joe Beda, and Kelsey Hightower Copyright © 2019 Brendan Burns, Joe Beda, and Kelsey Hightower. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or [email protected]. Acquisition Editor: John Devins Development Editor: Virginia Wilson Production Editor: Kristen Brown Copyeditor: Kim Cofer Proofreader: Rachel Head Indexer: Ellen Troutman-Zaig Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Rebecca Demarest September 2017: First Edition August 2019: Second Edition Revision History for the Second Edition 2019-07-15: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781492046530 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Kubernetes: Up and Running, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the authors, and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. This work is part of a collaboration between O’Reilly and Microsoft. See our statement of editorial inde‐ pendence.
- 9. For Robin, Julia, Ethan, and everyone who bought cookies to pay for that Commodore 64 in my third-grade class. —Brendan Burns For my Dad, who helped me fall in love with computers by bringing home punch cards and dot matrix banners. —Joe Beda For Klarissa and Kelis, who keep me sane. And for my Mom, who taught me a strong work ethic and how to rise above all odds. —Kelsey Hightower
- 11. Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii 1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Velocity 2 The Value of Immutability 3 Declarative Configuration 4 Self-Healing Systems 5 Scaling Your Service and Your Teams 5 Decoupling 6 Easy Scaling for Applications and Clusters 6 Scaling Development Teams with Microservices 7 Separation of Concerns for Consistency and Scaling 8 Abstracting Your Infrastructure 9 Efficiency 10 Summary 11 2. Creating and Running Containers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Container Images 14 The Docker Image Format 15 Building Application Images with Docker 16 Dockerfiles 16 Optimizing Image Sizes 18 Image Security 19 Multistage Image Builds 20 Storing Images in a Remote Registry 22 The Docker Container Runtime 23 Running Containers with Docker 23 Exploring the kuard Application 23 v
- 12. Limiting Resource Usage 24 Cleanup 24 Summary 25 3. Deploying a Kubernetes Cluster. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Installing Kubernetes on a Public Cloud Provider 28 Google Kubernetes Engine 28 Installing Kubernetes with Azure Kubernetes Service 28 Installing Kubernetes on Amazon Web Services 29 Installing Kubernetes Locally Using minikube 29 Running Kubernetes in Docker 30 Running Kubernetes on Raspberry Pi 31 The Kubernetes Client 31 Checking Cluster Status 31 Listing Kubernetes Worker Nodes 32 Cluster Components 34 Kubernetes Proxy 34 Kubernetes DNS 34 Kubernetes UI 35 Summary 36 4. Common kubectl Commands. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Namespaces 37 Contexts 37 Viewing Kubernetes API Objects 38 Creating, Updating, and Destroying Kubernetes Objects 39 Labeling and Annotating Objects 40 Debugging Commands 40 Command Autocompletion 42 Alternative Ways of Viewing Your Cluster 42 Summary 43 5. Pods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Pods in Kubernetes 46 Thinking with Pods 46 The Pod Manifest 47 Creating a Pod 48 Creating a Pod Manifest 48 Running Pods 49 Listing Pods 49 Pod Details 50 Deleting a Pod 51 vi | Table of Contents
- 13. Accessing Your Pod 52 Using Port Forwarding 52 Getting More Info with Logs 52 Running Commands in Your Container with exec 53 Copying Files to and from Containers 53 Health Checks 54 Liveness Probe 54 Readiness Probe 55 Types of Health Checks 56 Resource Management 56 Resource Requests: Minimum Required Resources 56 Capping Resource Usage with Limits 58 Persisting Data with Volumes 59 Using Volumes with Pods 59 Different Ways of Using Volumes with Pods 60 Persisting Data Using Remote Disks 61 Putting It All Together 61 Summary 63 6. Labels and Annotations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 Labels 65 Applying Labels 67 Modifying Labels 68 Label Selectors 68 Label Selectors in API Objects 70 Labels in the Kubernetes Architecture 71 Annotations 71 Defining Annotations 72 Cleanup 73 Summary 73 7. Service Discovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 What Is Service Discovery? 75 The Service Object 76 Service DNS 77 Readiness Checks 78 Looking Beyond the Cluster 79 Cloud Integration 81 Advanced Details 82 Endpoints 82 Manual Service Discovery 83 kube-proxy and Cluster IPs 84 Table of Contents | vii
- 14. Cluster IP Environment Variables 85 Connecting with Other Environments 86 Cleanup 86 Summary 86 8. HTTP Load Balancing with Ingress. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 Ingress Spec Versus Ingress Controllers 90 Installing Contour 91 Configuring DNS 92 Configuring a Local hosts File 92 Using Ingress 92 Simplest Usage 93 Using Hostnames 94 Using Paths 95 Cleaning Up 96 Advanced Ingress Topics and Gotchas 96 Running Multiple Ingress Controllers 97 Multiple Ingress Objects 97 Ingress and Namespaces 97 Path Rewriting 98 Serving TLS 98 Alternate Ingress Implementations 99 The Future of Ingress 100 Summary 101 9. ReplicaSets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 Reconciliation Loops 104 Relating Pods and ReplicaSets 104 Adopting Existing Containers 105 Quarantining Containers 105 Designing with ReplicaSets 105 ReplicaSet Spec 106 Pod Templates 106 Labels 107 Creating a ReplicaSet 107 Inspecting a ReplicaSet 108 Finding a ReplicaSet from a Pod 108 Finding a Set of Pods for a ReplicaSet 108 Scaling ReplicaSets 109 Imperative Scaling with kubectl scale 109 Declaratively Scaling with kubectl apply 109 Autoscaling a ReplicaSet 110 viii | Table of Contents
- 15. Deleting ReplicaSets 111 Summary 112 10. Deployments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 Your First Deployment 114 Deployment Internals 114 Creating Deployments 116 Managing Deployments 117 Updating Deployments 118 Scaling a Deployment 118 Updating a Container Image 119 Rollout History 120 Deployment Strategies 123 Recreate Strategy 123 RollingUpdate Strategy 123 Slowing Rollouts to Ensure Service Health 126 Deleting a Deployment 128 Monitoring a Deployment 128 Summary 129 11. DaemonSets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 DaemonSet Scheduler 132 Creating DaemonSets 132 Limiting DaemonSets to Specific Nodes 134 Adding Labels to Nodes 135 Node Selectors 135 Updating a DaemonSet 136 Rolling Update of a DaemonSet 136 Deleting a DaemonSet 137 Summary 138 12. Jobs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 The Job Object 139 Job Patterns 140 One Shot 140 Parallelism 144 Work Queues 146 CronJobs 150 Summary 151 13. ConfigMaps and Secrets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 ConfigMaps 153 Table of Contents | ix
- 16. Creating ConfigMaps 153 Using a ConfigMap 154 Secrets 157 Creating Secrets 158 Consuming Secrets 159 Private Docker Registries 160 Naming Constraints 161 Managing ConfigMaps and Secrets 162 Listing 162 Creating 163 Updating 163 Summary 165 14. Role-Based Access Control for Kubernetes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 Role-Based Access Control 168 Identity in Kubernetes 168 Understanding Roles and Role Bindings 169 Roles and Role Bindings in Kubernetes 169 Techniques for Managing RBAC 172 Testing Authorization with can-i 172 Managing RBAC in Source Control 172 Advanced Topics 172 Aggregating ClusterRoles 173 Using Groups for Bindings 173 Summary 175 15. Integrating Storage Solutions and Kubernetes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 Importing External Services 178 Services Without Selectors 179 Limitations of External Services: Health Checking 181 Running Reliable Singletons 181 Running a MySQL Singleton 181 Dynamic Volume Provisioning 185 Kubernetes-Native Storage with StatefulSets 186 Properties of StatefulSets 187 Manually Replicated MongoDB with StatefulSets 187 Automating MongoDB Cluster Creation 189 Persistent Volumes and StatefulSets 192 One Final Thing: Readiness Probes 193 Summary 194 x | Table of Contents
- 17. 16. Extending Kubernetes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 What It Means to Extend Kubernetes 195 Points of Extensibility 196 Patterns for Custom Resources 204 Just Data 204 Compilers 205 Operators 205 Getting Started 205 Summary 205 17. Deploying Real-World Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 Jupyter 207 Parse 209 Prerequisites 209 Building the parse-server 209 Deploying the parse-server 209 Testing Parse 210 Ghost 211 Configuring Ghost 211 Redis 214 Configuring Redis 215 Creating a Redis Service 216 Deploying Redis 217 Playing with Our Redis Cluster 218 Summary 219 18. Organizing Your Application. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221 Principles to Guide Us 221 Filesystems as the Source of Truth 222 The Role of Code Review 222 Feature Gates and Guards 223 Managing Your Application in Source Control 224 Filesystem Layout 224 Managing Periodic Versions 225 Structuring Your Application for Development, Testing, and Deployment 227 Goals 227 Progression of a Release 227 Parameterizing Your Application with Templates 229 Parameterizing with Helm and Templates 229 Filesystem Layout for Parameterization 230 Deploying Your Application Around the World 230 Architectures for Worldwide Deployment 230 Table of Contents | xi
- 18. Implementing Worldwide Deployment 232 Dashboards and Monitoring for Worldwide Deployments 233 Summary 233 A. Building a Raspberry Pi Kubernetes Cluster. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243 xii | Table of Contents
- 19. Preface Kubernetes: A Dedication Kubernetes would like to thank every sysadmin who has woken up at 3 a.m. to restart a process. Every developer who pushed code to production only to find that it didn’t run like it did on their laptop. Every systems architect who mistakenly pointed a load test at the production service because of a leftover hostname that they hadn’t updated. It was the pain, the weird hours, and the weird errors that inspired the development of Kubernetes. In a single sentence: Kubernetes intends to radically simplify the task of building, deploying, and maintaining distributed systems. It has been inspired by decades of real-world experience building reliable systems and it has been designed from the ground up to make that experience if not euphoric, at least pleasant. We hope you enjoy the book! Who Should Read This Book Whether you are new to distributed systems or have been deploying cloud-native sys‐ tems for years, containers and Kubernetes can help you achieve new levels of velocity, agility, reliability, and efficiency. This book describes the Kubernetes cluster orches‐ trator and how its tools and APIs can be used to improve the development, delivery, and maintenance of distributed applications. Though no previous experience with Kubernetes is assumed, to make maximal use of the book you should be comfortable building and deploying server-based applications. Familiarity with concepts like load balancers and network storage will be useful, though not required. Likewise, experi‐ ence with Linux, Linux containers, and Docker, though not essential, will help you make the most of this book. xiii
- 20. Why We Wrote This Book We have been involved with Kubernetes since its very beginnings. It has been truly remarkable to watch it transform from a curiosity largely used in experiments to a crucial production-grade infrastructure that powers large-scale production applica‐ tions in varied fields, from machine learning to online services. As this transition occurred, it became increasingly clear that a book that captured both how to use the core concepts in Kubernetes and the motivations behind the development of those concepts would be an important contribution to the state of cloud-native application development. We hope that in reading this book, you not only learn how to build reli‐ able, scalable applications on top of Kubernetes but also receive insight into the core challenges of distributed systems that led to its development. Why We Updated This Book In the few years that have passed since we wrote the first edition of this book, the Kubernetes ecosystem has blossomed and evolved. Kubernetes itself has had many releases, and many more tools and patterns for using Kubernetes have become de facto standards. In updating the book we added material on HTTP load balancing, role-based access control (RBAC), extending the Kubernetes API, how to organize your application in source control, and more. We also updated all of the existing chapters to reflect the changes and evolution in Kubernetes since the first edition. We fully expect to revise this book again in a few years (and look forward to doing so) as Kubernetes continues to evolve. A Word on Cloud-Native Applications Today From the first programming languages, to object-oriented programming, to the development of virtualization and cloud infrastructure, the history of computer sci‐ ence is a history of the development of abstractions that hide complexity and empower you to build ever more sophisticated applications. Despite this, the develop‐ ment of reliable, scalable applications is still dramatically more challenging than it ought to be. In recent years, containers and container orchestration APIs like Kuber‐ netes have proven to be an important abstraction that radically simplifies the devel‐ opment of reliable, scalable distributed systems. Though containers and orchestrators are still in the process of entering the mainstream, they are already enabling develop‐ ers to build and deploy applications with a speed, agility, and reliability that would have seemed like science fiction only a few years ago. xiv | Preface
- 21. Navigating This Book This book is organized as follows. Chapter 1 outlines the high-level benefits of Kuber‐ netes without diving too deeply into the details. If you are new to Kubernetes, this is a great place to start to understand why you should read the rest of the book. Chapter 2 provides a detailed introduction to containers and containerized applica‐ tion development. If you’ve never really played around with Docker before, this chap‐ ter will be a useful introduction. If you are already a Docker expert, it will likely be mostly review. Chapter 3 covers how to deploy Kubernetes. While most of this book focuses on how to use Kubernetes, you need to get a cluster up and running before you start using it. Although running a cluster for production is out of the scope of this book, this chap‐ ter presents a couple of easy ways to create a cluster so that you can understand how to use Kubernetes. Chapter 4 covers a selection of common commands used to inter‐ act with a Kubernetes cluster. Starting with Chapter 5, we dive into the details of deploying an application using Kubernetes. We cover Pods (Chapter 5), labels and annotations (Chapter 6), services (Chapter 7), Ingress (Chapter 8), and ReplicaSets (Chapter 9). These form the core basics of what you need to deploy your service in Kubernetes. We then cover deploy‐ ments (Chapter 10), which tie together the lifecycle of a complete application. After those chapters, we cover some more specialized objects in Kubernetes: Dae‐ monSets (Chapter 11), Jobs (Chapter 12), and ConfigMaps and secrets (Chapter 13). While these chapters are essential for many production applications, if you are just learning Kubernetes you can skip them and return to them later, after you gain more experience and expertise. Next we cover integrating storage into Kubernetes (Chapter 15). We discuss extend‐ ing Kubernetes in Chapter 16. Finally, we conclude with some examples of how to develop and deploy real-world applications in Kubernetes (Chapter 17) and a discus‐ sion of how to organize your applications in source control (Chapter 18). Online Resources You will want to install Docker. You likely will also want to familiarize yourself with the Docker documentation if you have not already done so. Likewise, you will want to install the kubectl command-line tool. You may also want to join the Kubernetes Slack channel, where you will find a large community of users who are willing to talk and answer questions at nearly any hour of the day. Finally, as you grow more advanced, you may want to engage with the open source Kubernetes repository on GitHub. Preface | xv
- 22. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program ele‐ ments such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user. Constant width italic Shows text that should be replaced with user-supplied values or by values deter‐ mined by context. This icon signifies a tip, suggestion, or general note. This icon indicates a warning or caution. Using Code Examples Supplemental material (code examples, exercises, etc.) is available for download at https://github.com/kubernetes-up-and-running/examples. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a signifi‐ cant amount of example code from this book into your product’s documentation does require permission. xvi | Preface
- 23. We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Kubernetes: Up and Running, 2nd edition, by Brendan Burns, Joe Beda, and Kelsey Hightower (O’Reilly). Copyright 2019 Brendan Burns, Joe Beda, and Kelsey Hightower, 978-1-492-04653-0.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at [email protected]. O’Reilly Online Learning For almost 40 years, O’Reilly Media has provided technology and business training, knowledge, and insight to help compa‐ nies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, conferences, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in- depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, please visit http://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at http://bit.ly/kubernetesUR_2e. To comment or ask technical questions about this book, send email to bookques‐ [email protected]. For more information about our books, courses, conferences, and news, see our web‐ site at http://www.oreilly.com. Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Preface | xvii
- 24. Watch us on YouTube: http://www.youtube.com/oreillymedia Acknowledgments We would like to acknowledge everyone who helped us develop this book. This includes our editor Virginia Wilson and all of the great folks at O’Reilly, as well as the technical reviewers who provided tremendous feedback that significantly improved the book. Finally, we would like to thank all of our first edition readers who took the time to report errata that were found and fixed in this second edition. Thank you all! We’re very grateful. xviii | Preface
- 25. 1 Brendan Burns et al., “Borg, Omega, and Kubernetes: Lessons Learned from Three Container-Management Systems over a Decade,” ACM Queue 14 (2016): 70–93, available at http://bit.ly/2vIrL4S. CHAPTER 1 Introduction Kubernetes is an open source orchestrator for deploying containerized applications. It was originally developed by Google, inspired by a decade of experience deploying scalable, reliable systems in containers via application-oriented APIs.1 Since its introduction in 2014, Kubernetes has grown to be one of the largest and most popular open source projects in the world. It has become the standard API for building cloud-native applications, present in nearly every public cloud. Kubernetes is a proven infrastructure for distributed systems that is suitable for cloud-native developers of all scales, from a cluster of Raspberry Pi computers to a warehouse full of the latest machines. It provides the software necessary to successfully build and deploy reliable, scalable distributed systems. You may be wondering what we mean when we say “reliable, scalable distributed sys‐ tems.” More and more services are delivered over the network via APIs. These APIs are often delivered by a distributed system, the various pieces that implement the API running on different machines, connected via the network and coordinating their actions via network communication. Because we rely on these APIs increasingly for all aspects of our daily lives (e.g., finding directions to the nearest hospital), these sys‐ tems must be highly reliable. They cannot fail, even if a part of the system crashes or otherwise stops working. Likewise, they must maintain availability even during soft‐ ware rollouts or other maintenance events. Finally, because more and more of the world is coming online and using such services, they must be highly scalable so that they can grow their capacity to keep up with ever-increasing usage without radical redesign of the distributed system that implements the services. 1
- 26. Depending on when and why you have come to hold this book in your hands, you may have varying degrees of experience with containers, distributed systems, and Kubernetes. You may be planning on building your application on top of public cloud infrastructure, in private data centers, or in some hybrid environment. Regardless of what your experience is, we believe this book will enable you to make the most of your use of Kubernetes. There are many reasons why people come to use containers and container APIs like Kubernetes, but we believe they can all be traced back to one of these benefits: • Velocity • Scaling (of both software and teams) • Abstracting your infrastructure • Efficiency In the following sections, we describe how Kubernetes can help provide each of these features. Velocity Velocity is the key component in nearly all software development today. The software industry has evolved from shipping products as boxed CDs or DVDs to software that is delivered over the network via web-based services that are updated hourly. This changing landscape means that the difference between you and your competitors is often the speed with which you can develop and deploy new components and fea‐ tures, or the speed with which you can respond to innovations developed by others. It is important to note, however, that velocity is not defined in terms of simply raw speed. While your users are always looking for iterative improvement, they are more interested in a highly reliable service. Once upon a time, it was OK for a service to be down for maintenance at midnight every night. But today, all users expect constant uptime, even if the software they are running is changing constantly. Consequently, velocity is measured not in terms of the raw number of features you can ship per hour or day, but rather in terms of the number of things you can ship while maintaining a highly available service. In this way, containers and Kubernetes can provide the tools that you need to move quickly, while staying available. The core concepts that enable this are: • Immutability • Declarative configuration • Online self-healing systems 2 | Chapter 1: Introduction
- 27. These ideas all interrelate to radically improve the speed with which you can reliably deploy software. The Value of Immutability Containers and Kubernetes encourage developers to build distributed systems that adhere to the principles of immutable infrastructure. With immutable infrastructure, once an artifact is created in the system it does not change via user modifications. Traditionally, computers and software systems have been treated as mutable infra‐ structure. With mutable infrastructure, changes are applied as incremental updates to an existing system. These updates can occur all at once, or spread out across a long period of time. A system upgrade via the apt-get update tool is a good example of an update to a mutable system. Running apt sequentially downloads any updated binaries, copies them on top of older binaries, and makes incremental updates to configuration files. With a mutable system, the current state of the infrastructure is not represented as a single artifact, but rather an accumulation of incremental updates and changes over time. On many systems these incremental updates come from not just system upgrades, but operator modifications as well. Furthermore, in any system run by a large team, it is highly likely that these changes will have been performed by many different people, and in many cases will not have been recorded anywhere. In contrast, in an immutable system, rather than a series of incremental updates and changes, an entirely new, complete image is built, where the update simply replaces the entire image with the newer image in a single operation. There are no incremental changes. As you can imagine, this is a significant shift from the more traditional world of configuration management. To make this more concrete in the world of containers, consider two different ways to upgrade your software: 1. You can log in to a container, run a command to download your new software, kill the old server, and start the new one. 2. You can build a new container image, push it to a container registry, kill the exist‐ ing container, and start a new one. At first blush, these two approaches might seem largely indistinguishable. So what is it about the act of building a new container that improves reliability? The key differentiation is the artifact that you create, and the record of how you cre‐ ated it. These records make it easy to understand exactly the differences in some new version and, if something goes wrong, to determine what has changed and how to fix it. Velocity | 3
- 28. Additionally, building a new image rather than modifying an existing one means the old image is still around, and can quickly be used for a rollback if an error occurs. In contrast, once you copy your new binary over an existing binary, such a rollback is nearly impossible. Immutable container images are at the core of everything that you will build in Kubernetes. It is possible to imperatively change running containers, but this is an anti-pattern to be used only in extreme cases where there are no other options (e.g., if it is the only way to temporarily repair a mission-critical production system). And even then, the changes must also be recorded through a declarative configuration update at some later time, after the fire is out. Declarative Configuration Immutability extends beyond containers running in your cluster to the way you describe your application to Kubernetes. Everything in Kubernetes is a declarative configuration object that represents the desired state of the system. It is the job of Kubernetes to ensure that the actual state of the world matches this desired state. Much like mutable versus immutable infrastructure, declarative configuration is an alternative to imperative configuration, where the state of the world is defined by the execution of a series of instructions rather than a declaration of the desired state of the world. While imperative commands define actions, declarative configurations define state. To understand these two approaches, consider the task of producing three replicas of a piece of software. With an imperative approach, the configuration would say “run A, run B, and run C.” The corresponding declarative configuration would be “replicas equals three.” Because it describes the state of the world, declarative configuration does not have to be executed to be understood. Its impact is concretely declared. Since the effects of declarative configuration can be understood before they are executed, declarative configuration is far less error-prone. Further, the traditional tools of software devel‐ opment, such as source control, code review, and unit testing, can be used in declara‐ tive configuration in ways that are impossible for imperative instructions. The idea of storing declarative configuration in source control is often referred to as “infrastruc‐ ture as code.” The combination of declarative state stored in a version control system and the ability of Kubernetes to make reality match this declarative state makes rollback of a change trivially easy. It is simply restating the previous declarative state of the system. This is usually impossible with imperative systems, because although the imperative instruc‐ tions describe how to get you from point A to point B, they rarely include the reverse instructions that can get you back. 4 | Chapter 1: Introduction
- 29. Self-Healing Systems Kubernetes is an online, self-healing system. When it receives a desired state configu‐ ration, it does not simply take a set of actions to make the current state match the desired state a single time. It continuously takes actions to ensure that the current state matches the desired state. This means that not only will Kubernetes initialize your system, but it will guard it against any failures or perturbations that might destabilize the system and affect reliability. A more traditional operator repair involves a manual series of mitigation steps, or human intervention performed in response to some sort of alert. Imperative repair like this is more expensive (since it generally requires an on-call operator to be avail‐ able to enact the repair). It is also generally slower, since a human must often wake up and log in to respond. Furthermore, it is less reliable because the imperative series of repair operations suffers from all of the problems of imperative management described in the previous section. Self-healing systems like Kubernetes both reduce the burden on operators and improve the overall reliability of the system by perform‐ ing reliable repairs more quickly. As a concrete example of this self-healing behavior, if you assert a desired state of three replicas to Kubernetes, it does not just create three replicas—it continuously ensures that there are exactly three replicas. If you manually create a fourth replica, Kubernetes will destroy one to bring the number back to three. If you manually destroy a replica, Kubernetes will create one to again return you to the desired state. Online self-healing systems improve developer velocity because the time and energy you might otherwise have spent on operations and maintenance can instead be spent on developing and testing new features. In a more advanced form of self-healing, there has been significant recent work in the operator paradigm for Kubernetes. With operators, more advanced logic needed to maintain, scale, and heal a specific piece of software (MySQL, for example) is enco‐ ded into an operator application that runs as a container in the cluster. The code in the operator is responsible for more targeted and advanced health detection and heal‐ ing than can be achieved via Kubernetes’s generic self-healing. Often this is packaged up as “operators,” which are discussed in a later section. Scaling Your Service and Your Teams As your product grows, it’s inevitable that you will need to scale both your software and the teams that develop it. Fortunately, Kubernetes can help with both of these goals. Kubernetes achieves scalability by favoring decoupled architectures. Scaling Your Service and Your Teams | 5
- 30. Decoupling In a decoupled architecture, each component is separated from other components by defined APIs and service load balancers. APIs and load balancers isolate each piece of the system from the others. APIs provide a buffer between implementer and con‐ sumer, and load balancers provide a buffer between running instances of each ser‐ vice. Decoupling components via load balancers makes it easy to scale the programs that make up your service, because increasing the size (and therefore the capacity) of the program can be done without adjusting or reconfiguring any of the other layers of your service. Decoupling servers via APIs makes it easier to scale the development teams because each team can focus on a single, smaller microservice with a comprehensible surface area. Crisp APIs between microservices limit the amount of cross-team communica‐ tion overhead required to build and deploy software. This communication overhead is often the major restricting factor when scaling teams. Easy Scaling for Applications and Clusters Concretely, when you need to scale your service, the immutable, declarative nature of Kubernetes makes this scaling trivial to implement. Because your containers are immutable, and the number of replicas is merely a number in a declarative config, scaling your service upward is simply a matter of changing a number in a configura‐ tion file, asserting this new declarative state to Kubernetes, and letting it take care of the rest. Alternatively, you can set up autoscaling and let Kubernetes take care of it for you. Of course, that sort of scaling assumes that there are resources available in your clus‐ ter to consume. Sometimes you actually need to scale up the cluster itself. Again, Kubernetes makes this task easier. Because many machines in a cluster are entirely identical to other machines in that set and the applications themselves are decoupled from the details of the machine by containers, adding additional resources to the cluster is simply a matter of imaging a new machine of the same class and joining it into the cluster. This can be accomplished via a few simple commands or via a pre‐ baked machine image. One of the challenges of scaling machine resources is predicting their use. If you are running on physical infrastructure, the time to obtain a new machine is measured in days or weeks. On both physical and cloud infrastructure, predicting future costs is difficult because it is hard to predict the growth and scaling needs of specific applica‐ tions. Kubernetes can simplify forecasting future compute costs. To understand why this is true, consider scaling up three teams, A, B, and C. Historically you have seen that 6 | Chapter 1: Introduction
- 31. each team’s growth is highly variable and thus hard to predict. If you are provisioning individual machines for each service, you have no choice but to forecast based on the maximum expected growth for each service, since machines dedicated to one team cannot be used for another team. If instead you use Kubernetes to decouple the teams from the specific machines they are using, you can forecast growth based on the aggregate growth of all three services. Combining three variable growth rates into a single growth rate reduces statistical noise and produces a more reliable forecast of expected growth. Furthermore, decoupling the teams from specific machines means that teams can share fractional parts of one another’s machines, reducing even further the overheads associated with forecasting growth of computing resources. Scaling Development Teams with Microservices As noted in a variety of research, the ideal team size is the “two-pizza team,” or roughly six to eight people. This group size often results in good knowledge sharing, fast decision making, and a common sense of purpose. Larger teams tend to suffer from issues of hierarchy, poor visibility, and infighting, which hinder agility and success. However, many projects require significantly more resources to be successful and achieve their goals. Consequently, there is a tension between the ideal team size for agility and the necessary team size for the product’s end goals. The common solution to this tension has been the development of decoupled, service-oriented teams that each build a single microservice. Each small team is responsible for the design and delivery of a service that is consumed by other small teams. The aggregation of all of these services ultimately provides the implementation of the overall product’s surface area. Kubernetes provides numerous abstractions and APIs that make it easier to build these decoupled microservice architectures: • Pods, or groups of containers, can group together container images developed by different teams into a single deployable unit. • Kubernetes services provide load balancing, naming, and discovery to isolate one microservice from another. • Namespaces provide isolation and access control, so that each microservice can control the degree to which other services interact with it. • Ingress objects provide an easy-to-use frontend that can combine multiple micro‐ services into a single externalized API surface area. Finally, decoupling the application container image and machine means that different microservices can colocate on the same machine without interfering with one another, reducing the overhead and cost of microservice architectures. The health- Scaling Your Service and Your Teams | 7
- 32. checking and rollout features of Kubernetes guarantee a consistent approach to appli‐ cation rollout and reliability that ensures that a proliferation of microservice teams does not also result in a proliferation of different approaches to service production lifecycle and operations. Separation of Concerns for Consistency and Scaling In addition to the consistency that Kubernetes brings to operations, the decoupling and separation of concerns produced by the Kubernetes stack lead to significantly greater consistency for the lower levels of your infrastructure. This enables you to scale infrastructure operations to manage many machines with a single small, focused team. We have talked at length about the decoupling of application container and machine/operating system (OS), but an important aspect of this decoupling is that the container orchestration API becomes a crisp contract that separates the responsi‐ bilities of the application operator from the cluster orchestration operator. We call this the “not my monkey, not my circus” line. The application developer relies on the service-level agreement (SLA) delivered by the container orchestration API, without worrying about the details of how this SLA is achieved. Likewise, the container orchestration API reliability engineer focuses on delivering the orchestration API’s SLA without worrying about the applications that are running on top of it. This decoupling of concerns means that a small team running a Kubernetes cluster can be responsible for supporting hundreds or even thousands of teams running applications within that cluster (Figure 1-1). Likewise, a small team can be responsi‐ ble for dozens (or more) of clusters running around the world. It’s important to note that the same decoupling of containers and OS enables the OS reliability engineers to focus on the SLA of the individual machine’s OS. This becomes another line of sepa‐ rate responsibility, with the Kubernetes operators relying on the OS SLA, and the OS operators worrying solely about delivering that SLA. Again, this enables you to scale a small team of OS experts to a fleet of thousands of machines. Of course, devoting even a small team to managing an OS is beyond the scale of many organizations. In these environments, a managed Kubernetes-as-a-Service (KaaS) provided by a public cloud provider is a great option. As Kubernetes has become increasingly ubiquitous, KaaS has become increasingly available as well, to the point where it is now offered on nearly every public cloud. Of course, using a KaaS has some limitations, since the operator makes decisions for you about how the Kubernetes clusters are built and configured. For example, many KaaS platforms dis‐ able alpha features because they can destabilize the managed cluster. 8 | Chapter 1: Introduction
- 33. Figure 1-1. An illustration of how different operations teams are decoupled using APIs In addition to a fully managed Kubernetes service, there is a thriving ecosystem of companies and projects that help to install and manage Kubernetes. There is a full spectrum of solutions between doing it “the hard way” and a fully managed service. Consequently, the decision of whether to use KaaS or manage it yourself (or some‐ thing in between) is one each user needs to make based on the skills and demands of their situation. Often for small organizations, KaaS provides an easy-to-use solution that enables them to focus their time and energy on building the software to support their work rather than managing a cluster. For a larger organization that can afford a dedicated team for managing its Kubernetes cluster, it may make sense to manage it yourself since it enables greater flexibility in terms of cluster capabilities and operations. Abstracting Your Infrastructure The goal of the public cloud is to provide easy-to-use, self-service infrastructure for developers to consume. However, too often cloud APIs are oriented around mirror‐ ing the infrastructure that IT expects, not the concepts (e.g., “virtual machines” instead of “applications”) that developers want to consume. Additionally, in many cases the cloud comes with particular details in implementation or services that are specific to the cloud provider. Consuming these APIs directly makes it difficult to run your application in multiple environments, or spread between cloud and physical environments. The move to application-oriented container APIs like Kubernetes has two concrete benefits. First, as we described previously, it separates developers from specific machines. This makes the machine-oriented IT role easier, since machines can simply Abstracting Your Infrastructure | 9
- 34. be added in aggregate to scale the cluster, and in the context of the cloud it also ena‐ bles a high degree of portability since developers are consuming a higher-level API that is implemented in terms of the specific cloud infrastructure APIs. When your developers build their applications in terms of container images and deploy them in terms of portable Kubernetes APIs, transferring your application between environments, or even running in hybrid environments, is simply a matter of sending the declarative config to a new cluster. Kubernetes has a number of plug-ins that can abstract you from a particular cloud. For example, Kubernetes services know how to create load balancers on all major public clouds as well as several different pri‐ vate and physical infrastructures. Likewise, Kubernetes PersistentVolumes and PersistentVolumeClaims can be used to abstract your applications away from spe‐ cific storage implementations. Of course, to achieve this portability you need to avoid cloud-managed services (e.g., Amazon’s DynamoDB, Azure’s CosmosDB, or Google’s Cloud Spanner), which means that you will be forced to deploy and manage open source storage solutions like Cassandra, MySQL, or MongoDB. Putting it all together, building on top of Kubernetes’s application-oriented abstrac‐ tions ensures that the effort you put into building, deploying, and managing your application is truly portable across a wide variety of environments. Efficiency In addition to the developer and IT management benefits that containers and Kuber‐ netes provide, there is also a concrete economic benefit to the abstraction. Because developers no longer think in terms of machines, their applications can be colocated on the same machines without impacting the applications themselves. This means that tasks from multiple users can be packed tightly onto fewer machines. Efficiency can be measured by the ratio of the useful work performed by a machine or process to the total amount of energy spent doing so. When it comes to deploying and managing applications, many of the available tools and processes (e.g., bash scripts, apt updates, or imperative configuration management) are somewhat ineffi‐ cient. When discussing efficiency it’s often helpful to think of both the cost of run‐ ning a server and the human cost required to manage it. Running a server incurs a cost based on power usage, cooling requirements, data- center space, and raw compute power. Once a server is racked and powered on (or clicked and spun up), the meter literally starts running. Any idle CPU time is money wasted. Thus, it becomes part of the system administrator’s responsibilities to keep utilization at acceptable levels, which requires ongoing management. This is where containers and the Kubernetes workflow come in. Kubernetes provides tools that automate the distribution of applications across a cluster of machines, ensuring higher levels of utilization than are possible with traditional tooling. 10 | Chapter 1: Introduction
- 35. A further increase in efficiency comes from the fact that a developer’s test environ‐ ment can be quickly and cheaply created as a set of containers running in a personal view of a shared Kubernetes cluster (using a feature called namespaces). In the past, turning up a test cluster for a developer might have meant turning up three machines. With Kubernetes it is simple to have all developers share a single test cluster, aggre‐ gating their usage onto a much smaller set of machines. Reducing the overall number of machines used in turn drives up the efficiency of each system: since more of the resources (CPU, RAM, etc.) on each individual machine are used, the overall cost of each container becomes much lower. Reducing the cost of development instances in your stack enables development prac‐ tices that might previously have been cost-prohibitive. For example, with your appli‐ cation deployed via Kubernetes it becomes conceivable to deploy and test every single commit contributed by every developer throughout your entire stack. When the cost of each deployment is measured in terms of a small number of con‐ tainers, rather than multiple complete virtual machines (VMs), the cost you incur for such testing is dramatically lower. Returning to the original value of Kubernetes, this increased testing also increases velocity, since you have strong signals as to the relia‐ bility of your code as well as the granularity of detail required to quickly identify where a problem may have been introduced. Summary Kubernetes was built to radically change the way that applications are built and deployed in the cloud. Fundamentally, it was designed to give developers more veloc‐ ity, efficiency, and agility. We hope the preceding sections have given you an idea of why you should deploy your applications using Kubernetes. Now that you are con‐ vinced of that, the following chapters will teach you how to deploy your application. Summary | 11
- 37. CHAPTER 2 Creating and Running Containers Kubernetes is a platform for creating, deploying, and managing distributed applica‐ tions. These applications come in many different shapes and sizes, but ultimately, they are all comprised of one or more programs that run on individual machines. These programs accept input, manipulate data, and then return the results. Before we can even consider building a distributed system, we must first consider how to build the application container images that contain these programs and make up the pieces of our distributed system. Application programs are typically comprised of a language runtime, libraries, and your source code. In many cases, your application relies on external shared libraries such as libc and libssl. These external libraries are generally shipped as shared components in the OS that you have installed on a particular machine. This dependency on shared libraries causes problems when an application developed on a programmer’s laptop has a dependency on a shared library that isn’t available when the program is rolled out to the production OS. Even when the development and production environments share the exact same version of the OS, problems can occur when developers forget to include dependent asset files inside a package that they deploy to production. The traditional methods of running multiple programs on a single machine require that all of these programs share the same versions of shared libraries on the system. If the different programs are developed by different teams or organizations, these shared dependencies add needless complexity and coupling between these teams. A program can only execute successfully if it can be reliably deployed onto the machine where it should run. Too often the state of the art for deployment involves running imperative scripts, which inevitably have twisty and byzantine failure cases. 13
- 38. This makes the task of rolling out a new version of all or parts of a distributed system a labor-intensive and difficult task. In Chapter 1, we argued strongly for the value of immutable images and infrastruc‐ ture. This immutability is exactly what the container image provides. As we will see, it easily solves all the problems of dependency management and encapsulation just described. When working with applications it’s often helpful to package them in a way that makes it easy to share them with others. Docker, the default container runtime engine, makes it easy to package an executable and push it to a remote registry where it can later be pulled by others. At the time of writing, container registries are avail‐ able in all of the major public clouds, and services to build images in the cloud are also available in many of them. You can also run your own registry using open source or commercial systems. These registries make it easy for users to manage and deploy private images, while image-builder services provide easy integration with continuous delivery systems. For this chapter, and the remainder of the book, we are going to work with a simple example application that we built to help show this workflow in action. You can find the application on GitHub. Container images bundle a program and its dependencies into a single artifact under a root filesystem. The most popular container image format is the Docker image for‐ mat, which has been standardized by the Open Container Initiative to the OCI image format. Kubernetes supports both Docker- and OCI-compatible images via Docker and other runtimes. Docker images also include additional metadata used by a con‐ tainer runtime to start a running application instance based on the contents of the container image. This chapter covers the following topics: • How to package an application using the Docker image format • How to start an application using the Docker container runtime Container Images For nearly everyone, their first interaction with any container technology is with a container image. A container image is a binary package that encapsulates all of the files necessary to run a program inside of an OS container. Depending on how you first experiment with containers, you will either build a container image from your local filesystem or download a preexisting image from a container registry. In either case, once the container image is present on your computer, you can run that image to produce a running application inside an OS container. 14 | Chapter 2: Creating and Running Containers
- 39. The Docker Image Format The most popular and widespread container image format is the Docker image for‐ mat, which was developed by the Docker open source project for packaging, distrib‐ uting, and running containers using the docker command. Subsequently, work has begun by Docker, Inc., and others to standardize the container image format via the Open Container Initiative (OCI) project. While the OCI standard achieved a 1.0 release milestone in mid-2017, adoption of these standards is proceeding slowly. The Docker image format continues to be the de facto standard, and is made up of a series of filesystem layers. Each layer adds, removes, or modifies files from the preceding layer in the filesystem. This is an example of an overlay filesystem. The overlay system is used both when packaging up the image and when the image is actually being used. During runtime, there are a variety of different concrete implementations of such file‐ systems, including aufs, overlay, and overlay2. Container Layering The phrases “Docker image format” and “container images” may be a bit confusing. The image isn’t a single file but rather a specification for a manifest file that points to other files. The manifest and associated files are often treated by users as a unit. The level of indirection allows for more efficient storage and transmittal. Associated with this format is an API for uploading and downloading images to an image registry. Container images are constructed with a series of filesystem layers, where each layer inherits and modifies the layers that came before it. To help explain this in detail, let’s build some containers. Note that for correctness the ordering of the layers should be bottom up, but for ease of understanding we take the opposite approach: . └── container A: a base operating system only, such as Debian └── container B: build upon #A, by adding Ruby v2.1.10 └── container C: build upon #A, by adding Golang v1.6 At this point we have three containers: A, B, and C. B and C are forked from A and share nothing besides the base container’s files. Taking it further, we can build on top of B by adding Rails (version 4.2.6). We may also want to support a legacy application that requires an older version of Rails (e.g., version 3.2.x). We can build a container image to support that application based on B also, planning to someday migrate the app to version 4: . (continuing from above) └── container B: build upon #A, by adding Ruby v2.1.10 └── container D: build upon #B, by adding Rails v4.2.6 └── container E: build upon #B, by adding Rails v3.2.x Container Images | 15
- 40. Conceptually, each container image layer builds upon a previous one. Each parent reference is a pointer. While the example here is a simple set of containers, other real- world containers can be part of a larger extensive directed acyclic graph. Container images are typically combined with a container configuration file, which provides instructions on how to set up the container environment and execute an application entry point. The container configuration often includes information on how to set up networking, namespace isolation, resource constraints (cgroups), and what syscall restrictions should be placed on a running container instance. The container root filesystem and configuration file are typically bundled using the Docker image format. Containers fall into two main categories: • System containers • Application containers System containers seek to mimic virtual machines and often run a full boot process. They often include a set of system services typically found in a VM, such as ssh, cron, and syslog. When Docker was new, these types of containers were much more com‐ mon. Over time, they have come to be seen as poor practice and application contain‐ ers have gained favor. Application containers differ from system containers in that they commonly run a single program. While running a single program per container might seem like an unnecessary constraint, it provides the perfect level of granularity for composing scal‐ able applications and is a design philosophy that is leveraged heavily by Pods. We will examine how Pods work in detail in Chapter 5. Building Application Images with Docker In general, container orchestration systems like Kubernetes are focused on building and deploying distributed systems made up of application containers. Consequently, we will focus on application containers for the remainder of this chapter. Dockerfiles A Dockerfile can be used to automate the creation of a Docker container image. Let’s start by building an application image for a simple Node.js program. This exam‐ ple would be very similar for many other dynamic languages, like Python or Ruby. 16 | Chapter 2: Creating and Running Containers
- 41. The simplest of npm/Node/Express apps has two files: package.json (Example 2-1) and server.js (Example 2-2). Put these in a directory and then run npm install express --save to establish a dependency on Express and install it. Example 2-1. package.json { "name": "simple-node", "version": "1.0.0", "description": "A sample simple application for Kubernetes Up & Running", "main": "server.js", "scripts": { "start": "node server.js" }, "author": "" } Example 2-2. server.js var express = require('express'); var app = express(); app.get('/', function (req, res) { res.send('Hello World!'); }); app.listen(3000, function () { console.log('Listening on port 3000!'); console.log(' http://localhost:3000'); }); To package this up as a Docker image we need to create two additional files: .docker‐ ignore (Example 2-3) and the Dockerfile (Example 2-4). The Dockerfile is a recipe for how to build the container image, while .dockerignore defines the set of files that should be ignored when copying files into the image. A full description of the syntax of the Dockerfile is available on the Docker website. Example 2-3. .dockerignore node_modules Example 2-4. Dockerfile # Start from a Node.js 10 (LTS) image FROM node:10 # Specify the directory inside the image in which all commands will run WORKDIR /usr/src/app Building Application Images with Docker | 17
- 42. # Copy package files and install dependencies COPY package*.json ./ RUN npm install # Copy all of the app files into the image COPY . . # The default command to run when starting the container CMD [ "npm", "start" ] Every Dockerfile builds on other container images. This line specifies that we are starting from the node:10 image on the Docker Hub. This is a preconfigured image with Node.js 10. This line sets the work directory, in the container image, for all following commands. These two lines initialize the dependencies for Node.js. First we copy the package files into the image. This will include package.json and package-lock.json. The RUN command then runs the correct command in the container to install the neces‐ sary dependencies. Now we copy the rest of the program files into the image. This will include every‐ thing except node_modules, as that is excluded via the .dockerignore file. Finally, we specify the command that should be run when the container is run. Run the following command to create the simple-node Docker image: $ docker build -t simple-node . When you want to run this image, you can do it with the following command. You can navigate to http://localhost:3000 to access the program running in the container: $ docker run --rm -p 3000:3000 simple-node At this point our simple-node image lives in the local Docker registry where the image was built and is only accessible to a single machine. The true power of Docker comes from the ability to share images across thousands of machines and the broader Docker community. Optimizing Image Sizes There are several gotchas that come when people begin to experiment with container images that lead to overly large images. The first thing to remember is that files that are removed by subsequent layers in the system are actually still present in the images; they’re just inaccessible. Consider the following situation: 18 | Chapter 2: Creating and Running Containers
- 43. . └── layer A: contains a large file named 'BigFile' └── layer B: removes 'BigFile' └── layer C: builds on B by adding a static binary You might think that BigFile is no longer present in this image. After all, when you run the image, it is no longer accessible. But in fact it is still present in layer A, which means that whenever you push or pull the image, BigFile is still transmitted through the network, even if you can no longer access it. Another pitfall that people fall into revolves around image caching and building. Remember that each layer is an independent delta from the layer below it. Every time you change a layer, it changes every layer that comes after it. Changing the preceding layers means that they need to be rebuilt, repushed, and repulled to deploy your image to development. To understand this more fully, consider two images: . └── layer A: contains a base OS └── layer B: adds source code server.js └── layer C: installs the 'node' package versus: . └── layer A: contains a base OS └── layer B: installs the 'node' package └── layer C: adds source code server.js It seems obvious that both of these images will behave identically, and indeed the first time they are pulled they do. However, consider what happens when server.js changes. In one case, it is only the change that needs to be pulled or pushed, but in the other case, both server.js and the layer providing the node package need to be pulled and pushed, since the node layer is dependent on the server.js layer. In general, you want to order your layers from least likely to change to most likely to change in order to optimize the image size for pushing and pulling. This is why, in Example 2-4, we copy the package*.json files and install dependencies before copying the rest of the pro‐ gram files. A developer is going to update and change the program files much more often than the dependencies. Image Security When it comes to security, there are no shortcuts. When building images that will ultimately run in a production Kubernetes cluster, be sure to follow best practices for packaging and distributing applications. For example, don’t build containers with passwords baked in—and this includes not just in the final layer, but any layers in the image. One of the counterintuitive problems introduced by container layers is that deleting a file in one layer doesn’t delete that file from preceding layers. It still takes Building Application Images with Docker | 19
- 44. up space, and it can be accessed by anyone with the right tools—an enterprising attacker can simply create an image that only consists of the layers that contain the password. Secrets and images should never be mixed. If you do so, you will be hacked, and you will bring shame to your entire company or department. We all want to be on TV someday, but there are better ways to go about that. Multistage Image Builds One of the most common ways to accidentally build large images is to do the actual program compilation as part of the construction of the application container image. Compiling code as part of the image build feels natural, and it is the easiest way to build a container image from your program. The trouble with doing this is that it leaves all of the unnecessary development tools, which are usually quite large, lying around inside of your image and slowing down your deployments. To resolve this problem, Docker introduced multistage builds. With multistage builds, rather than producing a single image, a Docker file can actually produce multiple images. Each image is considered a stage. Artifacts can be copied from preceding stages to the current stage. To illustrate this concretely, we will look at how to build our example application, kuard. This is a somewhat complicated application that involves a React.js frontend (with its own build process) that then gets embedded into a Go program. The Go program runs a backend API server that the React.js frontend interacts with. A simple Dockerfile might look like this: FROM golang:1.11-alpine # Install Node and NPM RUN apk update && apk upgrade && apk add --no-cache git nodejs bash npm # Get dependencies for Go part of build RUN go get -u github.com/jteeuwen/go-bindata/... RUN go get github.com/tools/godep WORKDIR /go/src/github.com/kubernetes-up-and-running/kuard # Copy all sources in COPY . . # This is a set of variables that the build script expects ENV VERBOSE=0 ENV PKG=github.com/kubernetes-up-and-running/kuard ENV ARCH=amd64 ENV VERSION=test 20 | Chapter 2: Creating and Running Containers
- 45. # Do the build. This script is part of incoming sources. RUN build/build.sh CMD [ "/go/bin/kuard" ] This Dockerfile produces a container image containing a static executable, but it also contains all of the Go development tools and the tools to build the React.js frontend and the source code for the application, neither of which are needed by the final application. The image, across all layers, adds up to over 500 MB. To see how we would do this with multistage builds, examine this multistage Docker‐ file: # STAGE 1: Build FROM golang:1.11-alpine AS build # Install Node and NPM RUN apk update && apk upgrade && apk add --no-cache git nodejs bash npm # Get dependencies for Go part of build RUN go get -u github.com/jteeuwen/go-bindata/... RUN go get github.com/tools/godep WORKDIR /go/src/github.com/kubernetes-up-and-running/kuard # Copy all sources in COPY . . # This is a set of variables that the build script expects ENV VERBOSE=0 ENV PKG=github.com/kubernetes-up-and-running/kuard ENV ARCH=amd64 ENV VERSION=test # Do the build. Script is part of incoming sources. RUN build/build.sh # STAGE 2: Deployment FROM alpine USER nobody:nobody COPY --from=build /go/bin/kuard /kuard CMD [ "/kuard" ] This Dockerfile produces two images. The first is the build image, which contains the Go compiler, React.js toolchain, and source code for the program. The second is the deployment image, which simply contains the compiled binary. Building a container image using multistage builds can reduce your final container image size by hundreds of megabytes and thus dramatically speed up your deployment times, since generally, Multistage Image Builds | 21
- 46. deployment latency is gated on network performance. The final image produced from this Dockerfile is somewhere around 20 MB. You can build and run this image with the following commands: $ docker build -t kuard . $ docker run --rm -p 8080:8080 kuard Storing Images in a Remote Registry What good is a container image if it’s only available on a single machine? Kubernetes relies on the fact that images described in a Pod manifest are available across every machine in the cluster. One option for getting this image to all machines in the cluster would be to export the kuard image and import it on each of them. We can’t think of anything more tedious than managing Docker images this way. The process of manually importing and exporting Docker images has human error writ‐ ten all over it. Just say no! The standard within the Docker community is to store Docker images in a remote registry. There are tons of options when it comes to Docker registries, and what you choose will be largely based on your needs in terms of security and collaboration features. Generally speaking, the first choice you need to make regarding a registry is whether to use a private or a public registry. Public registries allow anyone to download images stored in the registry, while private registries require authentication to down‐ load images. In choosing public versus private, it’s helpful to consider your use case. Public registries are great for sharing images with the world, because they allow for easy, unauthenticated use of the container images. You can easily distribute your soft‐ ware as a container image and have confidence that users everywhere will have the exact same experience. In contrast, a private registry is best for storing applications that are private to your service and that you don’t want the world to use. Regardless, to push an image, you need to authenticate to the registry. You can gener‐ ally do this with the docker login command, though there are some differences for certain registries. In the examples here we are pushing to the Google Cloud Platform registry, called the Google Container Registry (GCR); other clouds, including Azure and Amazon Web Services (AWS), also have hosted container registries. For new users hosting publicly readable images, the Docker Hub is a great place to start. Once you are logged in, you can tag the kuard image by prepending the target Docker registry. You can also append another identifier that is usually used for the version or variant of that image, separated by a colon (:): 22 | Chapter 2: Creating and Running Containers
- 47. $ docker tag kuard gcr.io/kuar-demo/kuard-amd64:blue Then you can push the kuard image: $ docker push gcr.io/kuar-demo/kuard-amd64:blue Now that the kuard image is available on a remote registry, it’s time to deploy it using Docker. Because we pushed it to the public Docker registry, it will be available every‐ where without authentication. The Docker Container Runtime Kubernetes provides an API for describing an application deployment, but relies on a container runtime to set up an application container using the container-specific APIs native to the target OS. On a Linux system that means configuring cgroups and namespaces. The interface to this container runtime is defined by the Container Run‐ time Interface (CRI) standard. The CRI API is implemented by a number of different programs, including the containerd-cri built by Docker and the cri-o implementa‐ tion contributed by Red Hat. Running Containers with Docker Though generally in Kubernetes containers are launched by a daemon on each node called the kubelet, it’s easier to get started with containers using the Docker command-line tool. The Docker CLI tool can be used to deploy containers. To deploy a container from the gcr.io/kuar-demo/kuard-amd64:blue image, run the following command: $ docker run -d --name kuard --publish 8080:8080 gcr.io/kuar-demo/kuard-amd64:blue This command starts the kuard container and maps ports 8080 on your local machine to 8080 in the container. The --publish option can be shortened to -p. This forwarding is necessary because each container gets its own IP address, so listening on localhost inside the container doesn’t cause you to listen on your machine. Without the port forwarding, connections will be inaccessible to your machine. The -d option specifies that this should run in the background (daemon), while --name kuard gives the container a friendly name. Exploring the kuard Application kuard exposes a simple web interface, which you can load by pointing your browser at http://localhost:8080 or via the command line: $ curl http://localhost:8080 The Docker Container Runtime | 23
- 48. kuard also exposes a number of interesting functions that we will explore later on in this book. Limiting Resource Usage Docker provides the ability to limit the amount of resources used by applications by exposing the underlying cgroup technology provided by the Linux kernel. These capabilities are likewise used by Kubernetes to limit the resources used by each Pod. Limiting memory resources One of the key benefits to running applications within a container is the ability to restrict resource utilization. This allows multiple applications to coexist on the same hardware and ensures fair usage. To limit kuard to 200 MB of memory and 1 GB of swap space, use the --memory and --memory-swap flags with the docker run command. Stop and remove the current kuard container: $ docker stop kuard $ docker rm kuard Then start another kuard container using the appropriate flags to limit memory usage: $ docker run -d --name kuard --publish 8080:8080 --memory 200m --memory-swap 1G gcr.io/kuar-demo/kuard-amd64:blue If the program in the container uses too much memory, it will be terminated. Limiting CPU resources Another critical resource on a machine is the CPU. Restrict CPU utilization using the --cpu-shares flag with the docker run command: $ docker run -d --name kuard --publish 8080:8080 --memory 200m --memory-swap 1G --cpu-shares 1024 gcr.io/kuar-demo/kuard-amd64:blue Cleanup Once you are done building an image, you can delete it with the docker rmi command: 24 | Chapter 2: Creating and Running Containers
- 49. docker rmi <tag-name> or: docker rmi <image-id> Images can either be deleted via their tag name (e.g., gcr.io/kuar-demo/kuard- amd64:blue) or via their image ID. As with all ID values in the docker tool, the image ID can be shortened as long as it remains unique. Generally only three or four char‐ acters of the ID are necessary. It’s important to note that unless you explicitly delete an image it will live on your system forever, even if you build a new image with an identical name. Building this new image simply moves the tag to the new image; it doesn’t delete or replace the old image. Consequently, as you iterate while you are creating a new image, you will often create many, many different images that end up taking up unnecessary space on your computer. To see the images currently on your machine, you can use the docker images com‐ mand. You can then delete tags you are no longer using. Docker provides a tool called docker system prune for doing general cleanup. This will remove all stopped containers, all untagged images, and all unused image layers cached as part of the build process. Use it carefully. A slightly more sophisticated approach is to set up a cron job to run an image garbage collector. For example, the docker-gc tool is a commonly used image garbage collector that can easily run as a recurring cron job, once per day or once per hour, depending on how many images you are creating. Summary Application containers provide a clean abstraction for applications, and when pack‐ aged in the Docker image format, applications become easy to build, deploy, and dis‐ tribute. Containers also provide isolation between applications running on the same machine, which helps avoid dependency conflicts. In future chapters we’ll see how the ability to mount external directories means we can run not only stateless applications in a container, but also applications like mysql and others that generate lots of data. Summary | 25
- 51. CHAPTER 3 Deploying a Kubernetes Cluster Now that you have successfully built an application container, the next step is to learn how to transform it into a complete, reliable, scalable distributed system. To do that, you need a working Kubernetes cluster. At this point, there are cloud-based Kuber‐ netes services in most public clouds that make it easy to create a cluster with a few command-line instructions. We highly recommend this approach if you are just get‐ ting started with Kubernetes. Even if you are ultimately planning on running Kuber‐ netes on bare metal, it’s a good way to quickly get started with Kubernetes, learn about Kubernetes itself, and then learn how to install it on physical machines. Fur‐ thermore, managing a Kubernetes cluster is a complicated task in itself, and for most people it makes sense to defer this management to the cloud—especially when in most clouds the management service is free. Of course, using a cloud-based solution requires paying for those cloud-based resour‐ ces as well as having an active network connection to the cloud. For these reasons, local development can be more attractive, and in that case the minikube tool provides an easy-to-use way to get a local Kubernetes cluster up running in a VM on your local laptop or desktop. Though this is a nice option, minikube only creates a single-node cluster, which doesn’t quite demonstrate all of the aspects of a complete Kubernetes cluster. For that reason, we recommend people start with a cloud-based solution, unless it really doesn’t work for their situation. A more recent alternative is to run a Docker-in-Docker cluster, which can spin up a multi-node cluster on a single machine. This project is still in beta, though, so keep in mind that you may encounter unexpected issues. If you truly insist on starting on bare metal, Appendix A at the end of this book gives instructions for building a cluster from a collection of Raspberry Pi single-board computers. These instructions use the kubeadm tool and can be adapted to other machines beyond Raspberry Pis. 27
- 52. Installing Kubernetes on a Public Cloud Provider This chapter covers installing Kubernetes on the three major cloud providers: Ama‐ zon Web Services, Microsoft Azure, and the Google Cloud Platform. If you choose to use a cloud provider to manage Kubernetes, you only need to install one of these options; once you have a cluster configured and ready to go you can skip to “The Kubernetes Client” on page 31, unless you would prefer to install Kubernetes elsewhere. Google Kubernetes Engine The Google Cloud Platform offers a hosted Kubernetes-as-a-Service called Google Kubernetes Engine (GKE). To get started with GKE, you need a Google Cloud Plat‐ form account with billing enabled and the gcloud tool installed. Once you have gcloud installed, first set a default zone: $ gcloud config set compute/zone us-west1-a Then you can create a cluster: $ gcloud container clusters create kuar-cluster This will take a few minutes. When the cluster is ready you can get credentials for the cluster using: $ gcloud auth application-default login If you run into trouble, you can find the complete instructions for creating a GKE cluster in the Google Cloud Platform documentation. Installing Kubernetes with Azure Kubernetes Service Microsoft Azure offers a hosted Kubernetes-as-a-Service as part of the Azure Con‐ tainer Service. The easiest way to get started with Azure Container Service is to use the built-in Azure Cloud Shell in the Azure portal. You can activate the shell by click‐ ing the shell icon in the upper-right toolbar: The shell has the az tool automatically installed and configured to work with your Azure environment. Alternatively, you can install the az command-line interface (CLI) on your local machine. When you have the shell up and working, you can run: 28 | Chapter 3: Deploying a Kubernetes Cluster
- 53. $ az group create --name=kuar --location=westus Once the resource group is created, you can create a cluster using: $ az aks create --resource-group=kuar --name=kuar-cluster This will take a few minutes. Once the cluster is created, you can get credentials for the cluster with: $ az aks get-credentials --resource-group=kuar --name=kuar-cluster If you don’t already have the kubectl tool installed, you can install it using: $ az aks install-cli You can find complete instructions for installing Kubernetes on Azure in the Azure documentation. Installing Kubernetes on Amazon Web Services Amazon offers a managed Kubernetes service called Elastic Kubernetes Service (EKS). The easiest way to create an EKS cluster is via the open source eksctl command-line tool.. Once you have eksctl installed and in your path, you can run the following com‐ mand to create a cluster: $ eksctl create cluster --name kuar-cluster ... For more details on installation options (such as node size and more), view the help using this command: $ eksctl create cluster --help The cluster installation includes the right configuration for the kubectl command- line tool. If you don’t already have kubectl installed, you can follow the instructions in the documentation. Installing Kubernetes Locally Using minikube If you need a local development experience, or you don’t want to pay for cloud resources, you can install a simple single-node cluster using minikube. Alternatively, if you have already installed Docker Desktop, it comes bundled with a single-machine installation of Kubernetes. While minikube (or Docker Desktop) is a good simulation of a Kubernetes cluster, it’s really intended for local development, learning, and experimentation. Because it only runs in a VM on a single node, it doesn’t provide the reliability of a distributed Kubernetes cluster. Installing Kubernetes Locally Using minikube | 29
- 54. In addition, certain features described in this book require integration with a cloud provider. These features are either not available or work in a limited way with minikube. You need to have a hypervisor installed on your machine to use minikube. For Linux and macOS, this is generally virtualbox. On Windows, the Hyper-V hypervisor is the default option. Make sure you install the hypervisor before using minikube. You can find the minikube tool on GitHub. There are binaries for Linux, macOS, and Windows that you can download. Once you have the minikube tool installed, you can create a local cluster using: $ minikube start This will create a local VM, provision Kubernetes, and create a local kubectl configu‐ ration that points to that cluster. When you are done with your cluster, you can stop the VM with: $ minikube stop If you want to remove the cluster, you can run: $ minikube delete Running Kubernetes in Docker A different approach to running a Kubernetes cluster has been developed more recently, which uses Docker containers to simulate multiple Kubernetes nodes instead of running everything in a virtual machine. The kind project provides a great experi‐ ence for launching and managing test clusters in Docker. (kind stands for Kubernetes IN Docker.) kind is still a work in progress (pre 1.0), but is widely used by those building Kubernetes for fast and easy testing. Installation instructions for your platform can be found at the kind site. Once you get it installed, creating a cluster is as easy as: $ kind create cluster --wait 5m $ export KUBECONFIG="$(kind get kubeconfig-path)" $ kubectl cluster-info $ kind delete cluster 30 | Chapter 3: Deploying a Kubernetes Cluster
- 55. Other documents randomly have different content
- 56. "I did not intend it as a comparison," answered Robert. "With you it can never be one, and with me such ideas would be very ungrateful, applied to my oldest friend. I wish to heaven, no thought against him would ever enter my head again." "Conquer them—never breathe them even to yourself!" said Florence, with sudden impetuosity. "They have killed me—those weary, base suspicions—not mine! not mine! Oh, I am so thankful that they were not formed in my heart? —they were whispered to me—forced on me. I would not believe them—but the evil thing is here. I have no strength to cast it out alone, and he never comes to help me." "Perhaps he does not know how deeply you feel for him," said Robert, anxious to console her. Florence shook her head, and leaning forward, shrouded her eyes with one hand. After a while, she turned her gaze upon Robert, and addressed him more quietly. "You must not think ill of him," she said, with a dim smile. "See what suspicion and pining thoughts can do, when they have crept into the heart." The poor girl drew up the muslin sleeve from her arm, and Robert was startled to see how greatly the delicate limb was attenuated. Tears came into his eyes, and bending down he touched the snowy wrist with his lips. "I must tell him that you are ill—that you suffer—surely he cannot dream of this!" "Not yet—we must not importune him; besides, I am becoming used to this desolate feeling. You will come oftener now. It is something to know that he has been near you—touched your clothes—held your hand—the atmosphere of his presence hangs about your very garments, and does me good. This seems childish, does it not? but it is true. Sometime, when you have given up your being to another, this will appear less strange. Oh, how I sometimes envy you!" "I might have loved, young as you think me, even as you love this man," said Robert, annoyed, spite of his sympathy, by the words which she had unconsciously applied to his youth; "but that which has wounded you, saved me. You do not know, Miss Craft, all that I have felt since the evening when Mr. Leicester brought me here. What I saw that night awoke me from the first sweet dream of passion I ever knew. I could have loved you then, even as you loved Mr. Leicester."
- 57. "Me!" said Florence, and a momentary smile lighted her eyes—as if the very thought of his young love amused her, sad as she was; "how strange! to me you seemed so young and embarrassed—a mere boy—now——" "Now I am changed, you would say—now I am a different person—older, firmer, more self-possessed; yet it is only a few months ago. I may seem older and less timid—for in this little time I have thought and suffered—but then, I was more worthy of your love, for I had not learned to distrust my oldest friend. Like you, I have struggled against suspicion—and like you, I have failed to cast it forth. It has withered your gentle nature—mine it has embittered." "Ah! but you had not my temptation. It was not his own mother who poisoned your mind against him." "His mother? I did not know that either of his parents were living." "That quiet, cold lady; the woman whom you have seen here! Did he never tell you that she was his mother?" "He never even hinted it!" said Robert, greatly surprised. "She told me so with her own lips: she warned me against him—she, his mother." "Indeed!" said Robert, thoughtfully. "Yet with what coldness she received him!" "It is not her nature," answered Florence, and her eyes filled with grateful tears. "To me, her kindness has been unvaried; there is something almost holy in her calm, sweet affection: but for this I had not been so unhappy. Had I detected prejudice, temper, anything selfish mingled with her words, they would never have reached my heart; but now, I cannot turn from her. With all her stately coldness she had something of his power—I dare not doubt her. But I will not believe the warning she gave me." Robert walked up and down the room. New and stern thoughts were making their way in his mind. Gratitude is a powerful feeling, but it possesses none of the infatuation and blindness which characterizes the grand passion. Suspicions that had haunted his conscience like crimes, were beginning to shape themselves into stubborn facts. Still he would not yield to them. Like the gentle girl, drooping before his eyes, he dared not believe anything against William Leicester. Humiliation, nay, almost ruin, lay in the thought.
- 59. CHAPTER XIV. A WEDDING FORESHADOWED. When her heart was all dreary and burdened with fears, Hope came like a seraph and touched it with light, Like sunshine or rain-drops it kindled her tears Till they trembled like stars 'mid her soul's quick delight. Florence had taken up her pencil again, but still remained inactive, gazing wistfully through the lace curtains, at the little fountain flinging up a storm of spray amid flowers gorgeous with autumn tints and the crisp brown that had settled on the little grass-plat. Notwithstanding the dahlias were in a glow of rich tints, and the chrysanthemums sheeted with white, rosy, and golden blossoms, there was a tinge of decay upon the leaves, very beautiful, but always productive of mournful feelings. Florence had felt this influence more than usual that morning, and now to her excited nerves there was something in the glow of those flowers, and the soft rush of water-drops, that made her heart sink. If the autumn and summer had been so dreary, with all the warmth and brightness of sunshine and blossoms, what had the winter of promise to her? Spite of herself she looked down to the thin, white hand that lay so listlessly on the paper, and gazed on it till tears swelled once more against those half- closed eye-lids. "How desolate to be buried in the winter, and away from all ——" These were the thoughts that arose in that young heart. The objects that gave rise to them were flowers, autumn flowers, the richest and most beautiful things on earth. Thus it often happens in life, that lovely things awake our most painful and bitter feelings, either by a mocking contrast with the sorrow that is within us, or because they are associated with the memory of wasted happiness. As Florence sat gazing upon the half veiled splendor of the garden flowers, she saw a man open the little gate, and move with a slow, heavy step toward the door. The face was unfamiliar, and the fact of any strange person seeking that dwelling was rare enough to excite some nervous trepidation in a young and fragile creature like Florence.
- 60. "There is some one coming," she said, addressing Robert, who was thoughtfully pacing the room, with a tone and look of alarm quite disproportioned to the occasion. "Will you go to the door, I believe every one is out except us?" Robert shook off the train of thought that had made him unconscious of the heavy footsteps now plainly heard in the veranda, and went to the door. Jacob Strong did not seem in the least embarrassed, though nothing could be supposed further from his thoughts than an encounter with the young man in that place. Perhaps he lost something of the abruptness unconsciously maintained during his walk, for his mien instantly assumed a loose, almost slouching carelessness, such as had always characterized it in the presence of Leicester or his protégé. "Well, how do you do, Mr. Otis? I didn't just expect to find you here! Hain't got much to do down at the store, I reckon?" "Never mind that, Mr. Strong," answered the youth, good-humoredly, "but tell me what brought you here. Some message from Mr. Leicester, ha!" "Well, now, you do beat all at guessing," answered Jacob, drawing forth the billet-doux with which he was charged. "Ain't there a young gal a-living here, Miss Flo—Florence Craft? If that ain't the name, I can't cipher it out any how!" "Yes, that is the name—Miss Craft does live here," said Robert. "Let me have the note—I will deliver it." "Not as you know on, Mr. Otis," replied Jacob, with a look of shrewd determination. "Mr. Leicester told me to give this ere little concern into the gal's own hand, and I always obey orders though I break owners. Jest be kind enough to show me where the young critter is, and I'll do my errand and back again in less than no time." "Very well, come this way; Miss Craft will receive the note herself." Florence was standing near the window, her bright, eager eyes were turned upon the door, she had overheard Leicester's name, and it thrilled through every nerve of her body. Jacob entered with his usual heavy indifference. He looked a moment at the young girl, and then held out the note. Robert fancied that a shade of feeling swept over that usually composed face, but the lace curtains were waving softly to a current of air let in through the open doors, and it might be the
- 61. transient shadows thus flung upon his face. Still there was something keen and intelligent in the glance with which Jacob regarded the young girl while she bent over the note. Suddenly he bent those keen, grey eyes, now full of meaning, and almost stern in their searching power, upon the youth himself. Robert grew restless beneath that strict scrutiny, the color mounted to his forehead, and as a relief he turned toward Florence. She was busy reading the note, apparently unconscious of the person, but oh, how wildly beautiful her face had become! Her eyes absolutely sparkled through the drooping lashes; her small mouth was parted in a glowing smile— you could see the pearly edges of her teeth behind the bright red of lips that seemed just bathed in wine. She trembled from head to foot, not violently, but a blissful shiver, like that which stirs a leaf at noonday, in the calm summer time, wandered over her delicate frame. Twice—three times, she read the note, and then her soft eyes were uplifted and turned upon Robert, in all their glorious joy. "See!" she said, and her voice was one burst of melody—"Oh, what ingrates we have been to doubt him!" In her bright triumph, she held forth the note, but as Robert advanced to receive it, she drew back. "I had forgotten," she said, "I alone was to know it; but you can guess—you can see how happy it has made me." Robert Otis turned away, somewhat annoyed by this half confidence. Florence, without heeding this, sat down by the table, and, with the open note before her, prepared to answer it, but her excitement was too eager—her hand too unsteady. After several vain efforts, she took the note and ran up stairs. Thus Jacob and Robert were left alone together. The youth, possessed by his own thoughts, seemed quite unconscious of the companionship forced upon him. He sat down on the couch which Florence had occupied, and, leaning upon the table, supported his forehead with one hand. Jacob stood in his old place, regarding the varied expressions that came and went on that young face. His own rude features were greatly disturbed, and at this moment bore a look that approached to anguish. Twice he moved, as if to approach Robert —and then fell back irresolute; but at last, he strode forward, and before the youth was aware of the movement, a hand lay heavily upon his shoulder. "So you love her, my boy?"
- 62. Robert started. The drawling tone, the rude Down East enunciation was gone. The man who stood before him seemed to have changed his identity. Rude and uncouth he certainly was—but even in this, there was something imposing. Robert looked at him with parted lips and wondering eyes—there was something even of awe in his astonishment. "Tell me, boy," continued Jacob, and his voice was full of tenderness—"tell me, is it love for this girl, that makes you thoughtful? Are you jealous of William Leicester?" Robert lost all presence of mind—he did not answer—but sat motionless, with his eyes turned upon the changed face bending close to his. "Will you not speak to me, Robert Otis? You may—you should, for I am an honest man." "I believe you are!" said Robert, starting up and reaching forth his hand—"I know that you are, for my heart leaps toward you. What was the question? I will answer it now. Did you ask if I loved Florence Craft?" "Yes, that was it—I would know; otherwise events may shape themselves unluckily. I trust, Robert, that in this you have escaped the snare." "I do not understand you, but can answer your question a great deal better than I could have done three days ago. I do love Miss Craft as it has always seemed to me that I should love a sister, had one been made an orphan with me: I would do any thing for her, sacrifice anything for her. Once I thought this love, but now I know better. There was another question—am I jealous of William Leicester? I do not know; my heart sinks when I see them together—I cannot force myself to wish her his wife, and yet this repugnance is unaccountable to myself. He is my friend—she something even dearer than a sister; but my very soul revolts at the thought of their union. It was this that made me thoughtful: I do not love Florence in your meaning of the word; I am not jealous of Mr. Leicester; but God forgive me! there is something in my heart that rises up against him! There, sir, you have my answer. I may be imprudent—I may be wrong; but it cannot be helped now." "You have been neither imprudent nor wrong," answered Jacob, laying his hand on the bent head of the youth. "I am a plain man, but you will find in me a safer counsellor than you imagine—a wiser one—though not more sincere—than your good aunt." "Then you know my aunt?" cried Robert, profoundly astonished.
- 63. "It would have been well had you confided even in her, on Thanksgiving night, when you were so near confessing the difficulties that seem so terrible to you. A few words then, might have relieved all your troubles." "Then Mr. Leicester has told—has betrayed me to—to his servant, I would not have believed it!" Robert grew pale as he spoke; there was shame and terror in his face; deep bitterness in his tone; he was suffering the keen pangs which a first proof of treachery brings to youth. "No, you wrong Mr. Leicester there—he has not betrayed you, never will, probably, nor do I know the exact nature of your anxieties." "But who are you then? An hour ago I could have answered this question, or thought so. Now, you bewilder me; I can scarcely recognize any look or tone about you—which is the artificial? which the real?" "Both are real; I was what you have hitherto seen me, years ago. I am what you see now; but I can at will throw off the present and identify myself with the past. You see, Robert Otis, I give confidence when I ask it—a breath of what you have seen or heard to-day, repeated to Mr. Leicester, would send me from his service. But I do not fear to trust you!" "There is no cause of fear—I never betrayed anything in my life—only convince me that you mean no evil to him." "I only mean to prevent evil! and I will!" "All this perplexes me," said Robert, raising one hand to his forehead—"I seem to have known you many years; my heart warms toward you as it never did to any one but my aunt." "That is right; an honest heart seldom betrays itself. But hush! the young lady is coming; God help her, she loves that man." "It is worship—idolatry—not love; that seems but a feeble word; it gives one the heart-ache to witness its ravages on her sweet person." "And does she feel so much?" said Jacob, with emotion. Before Robert could answer, the light step of Florence was heard on the stairs; when she entered the room, Jacob stood near the window, holding his hat awkwardly between both hands, and with his eyes bent upon the floor. "You will give this to Mr. Leicester," she said, still radiant and beautiful with happiness, placing a note in Jacob's hand—"here is something for yourself, I
- 64. only wish it could make you as happy as—as—that it may be of use, I mean." Blushing and hesitating thus in her speech, she placed a small gold coin upon the note. Poor girl, it was a pocket-piece given by her father, but in her wild gratitude she would have cast thousands upon the man whose coming had brought so much happiness. Jacob received the coin, looked at her earnestly for a moment, half extended his hand, and then thrust it into his pocket. "Thank you, ma'am, a thousand times—I will do the errand right off!" and putting on his hat, Jacob strode from the house, muttering, as he cast a hurried glance around the little garden, "It seems like shooting a robin on her nest—I must think it all over again." Robert would have followed Jacob Strong, for his mind was in tumult, and he panted for some more perfect elucidation of the mystery that surrounded this singular man. But Florence laid her hand gently on his arm, and drew him into the window recess: her face was bright with smiles and bathed in blushes. "You were ready to go without wishing me joy," she said; "and yet you must have guessed what was in that precious, precious note!" Robert felt a strange thrill creep through his frame. He turned his eyes from the soft orbs looking into his, for their brilliancy pained him. "No," he said, almost bitterly, "I cannot guess—perhaps I do not care to guess!" "Oh, Robert! you do not know what happiness is; no human being ever was so happy before. How cold—how calm you are! You could feel for me when I was miserable, but now—now it is wrong: he charged me to keep it secret, but my heart is so full, Robert; stoop and let me whisper it—tell nobody, he would be very angry—but this week we are to be married!" "Now," said Robert, drawing a deep breath, and speaking in a voice so calm that it seemed like prophecy—"now I feel for you more than ever." The little, eager hand fell from his arm, and in a voice that thrilled with disappointment, Florence said, "Then you will not wish me joy!" Robert took her hand, grasped it a moment in his, and flinging aside the cloud of lace that had fallen over them, left the room. Florence followed him with her eyes, and while he was in sight a shade of sadness hung upon her sweet
- 65. face—but her happiness was too perfect even for this little shadow to visit it more than a moment. She sunk upon an ottoman in the recess, and, with her eyes fixed upon the autumn flowers without, subsided into a reverie, the sweetest, the brightest that ever fell upon a youthful heart.
- 66. CHAPTER XV. THE MOTHER'S APPEAL. Wrong to one's self but wrongs the world; God linketh soul so close to soul, That germs of evil, once unfurled, Spread through the life and mock control. Pen, ink, and paper lay upon the table. The curtains were flung back, admitting the broad sunshine that revealed more clearly than the usual soft twilight with which Leicester was in the habit of enveloping himself, the lines which time and passion sometimes allowed to run wild, sometimes curbed with an iron will, had left on his handsome features. Papers were on the table, not letters, but scraps that bore a business aspect, some half printed, others without signature, but still in legal form, as notes of hand or checks are given. Leicester took one of these checks—a printed blank—and gazed on it some moments with a fixed and thoughtful scrutiny. He laid it gently down, took up a pen, and held the drop of ink on its point up to the light, as if even the color were an object of interest. He wrote a word or two, merely filling up the blank before him, but simple as the act seemed, that hand, usually firm as marble, quivered on the paper, imperceptibly, it is true, but enough to render the words unsteady. His face, too, was fiercely pale, if I may use the term, for there was something in the expression of those features that sent a sort of hard glow through their whiteness. It was the glow of a desperate will mastering fear. With a quick and scornful quiver of the lip, he tore the half-filled check in twain, and cast the fragments into the fire. "Am I growing old?" he said aloud, "or is this pure cowardice? Fear!—what have I to fear?" he continued, hushing his voice. "It cannot be brought back to me. A chain that has grown, link by link, for years, will not break with any common wrench. Still, if it could be avoided, the boy loves me!—well, and have not others loved me? Of what use is affection, if it adds nothing to one's enjoyments? If the old planter had left my pretty Florence the property at once, why then—but till she is of age—that is almost two years—till she is of age we must live."
- 67. Half in thought, half in words, these ideas passed through the brain and upon the lip of William Leicester. When his mind was once made up to the performance of an act, it seldom paused even to excuse a sin to his own soul, but this was not exactly a question of right and wrong: that had been too often decided with his conscience to admit of the least hesitation. There was peril in the act he meditated—peril to himself—this made his brow pale and his hand unsteady. During a whole life of fraud and evil-doing, he had never once placed himself within the grasp of the law. His instruments, less guilty, and far less treacherous than himself, had often suffered for crimes that his keen intellect had suggested. For years he had luxuriated upon the fruit of iniquities prompted by himself, but with which his personal connection could never be proved. But for once his subtle forethought in selecting and training an agent who should bear the responsibility of crime while he reaped the benefit, had failed. The time had arrived when Robert Otis was, if ever, to become useful to his teacher. But evil fruit in that warm, generous nature had been slow in ripening. With all his subtle craft, Leicester dared not propose the fraud which was to supply him with means for two years' residence in Europe. There was something in the boy too clear-sighted and prompt even for his wily influence, and now, after years of training worthy of Lucifer himself, Leicester, for the first time, was afraid to trust his chosen instrument. Robert might be deluded into wrong—might innocently become his victim, but Leicester despaired of making him, with his bright intellect and honorable impulses, the principal or accomplice of an act such as he meditated. A decanter of brandy stood upon the table—Leicester filled a goblet and half drained it. This in no way disturbed the pallor of his countenance, but his hand grew firm, and he filled up several of the printed checks with a rapidity that betrayed the misgivings that still beset him. He examined the papers attentively after they were written, and, selecting one, laid it in an embroidered letter-case which he took from his bosom; the others he placed in an old copy-book that had been lying open before him all the time; it was the same book that Robert Otis had taken from his aunt's stand-drawer on Thanksgiving night. When these arrangements were finished, Leicester drew out his watch, and seemed to be waiting for some one that he expected. Again he opened the copy-book and compared the checks with other papers it contained. The scrutiny seemed to satisfy him, for a smile gleamed in his eyes
- 68. as he closed the book. Just then, Robert Otis came in. His step had become quiet, and the rosy buoyancy of look and manner that had been so interesting a few months before, was entirely gone. There was restraint—nay, something amounting almost to dislike in his air as he drew a seat to the table. "You are looking pale, Robert; has anything gone amiss at the counting- house?" said Leicester, regarding his visitor with interest. "Nothing!" "Are you ill then?" "No, I am well—quite well!" "But something distresses you; those shadows under the eye, the rigid lines about the mouth—there is trouble beneath them. Tell me what it is—am I not your friend?" Robert smiled a meaning, bitter smile, that seemed strangely unnatural on those fresh lips. Leicester read the meaning of that silent reproach, and it warned him to be careful. "Surely," he said, "you have not been at F—— street, without your friend?— you have not indulged in high play, and no prudent person to guide you?" "No!" said Robert, with bitter energy—"that night I did play—how, why, it is impossible for me to remember. Those few hours of wild sin were enough— they have stained my soul—they have plunged me into debt—they have made me ashamed to look a good man in the face." "But I warned, I cautioned you!" Robert did not answer, but by the gleam of his eyes and the quiver of his lips, you could see that words of fire were smothered in his heart. "You would have plunged into the game deeper and deeper, but for me." "Perhaps I should—it was a wild dream—I was mad—the very memory almost makes me insane. I, so young, so cherished, in debt—and how—to what amount?" "Enough—I am afraid," said Leicester, gently—"enough to cover that pretty farm, and all the bank stock your nice old aunt has scraped together. But what
- 69. of that?—she is in no way responsible, and gambling debts are only debts of honor—no law reaches them?" "I will not make sin the shelter of meanness," answered the youth, with a wild flash of feeling; "these men may be villains, but they did not force themselves upon me. I sought them of my own free choice; no—I cannot say that either, for heaven knows I never wished to enter that den!" "It was I that invited, nay, urged you!" "Else I had never been there!" "But I intended it as a warning—I cautioned you, pleaded with you." "Yes, I remember—you said I was ignorant, awkward, a novice—Mr. Leicester; your advice was like a jeer—your caution a taunt; your words and manner were at variance; I played that night, but not of my own free will. I say to you, it was not of my free will!" "Is it me, upon whom your words reflect?" said Leicester, with every appearance of wounded feeling. Robert was silent. "Do you know," continued Leicester, in that deep, musical tone, that was sure to make the heart thrill—"do you know, Robert Otis, why it is that you have not been openly exposed?—why this debt has not been demanded long ago?" "Because the note which I gave is not yet due!" "The note—a minor's note—what man in his senses would receive a thing so worthless? No, Robert—it was my endorsement that made the paper valuable. It is from me, your old friend, Robert, that the money must come to meet the paper at its maturity." Tears gushed into the young man's eyes—he held out his hand across the table—Leicester took the hand and pressed it very gently. "You know," he said, "this note becomes due almost immediately." "I know—I know. It seems to me that every day has left a mark on my heart; oh, Mr. Leicester, how I have suffered!" "I will not say that suffering is the inevitable consequence of a wrong act, because that just now would be unkind," said Leicester, with a soft smile, "but hereafter you must try and remember that it is so."
- 70. Robert looked upon his friend; his large eyes dilated, and his lips began to tremble; you could see that his heart was smitten to the core. How he had wrought that man! Tears of generous compunction rushed to his eyes. "It will be rather difficult, but I have kept this thing in my mind," said Leicester. "To-morrow I shall draw a large sum; a portion must redeem your debt, but on condition that you never play again!" Robert shuddered. "Play again!" he said, and tears gushed through the fingers which he had pressed to his eyes. "Do you fear that a man who has been racked would of his own free will seek the wheel again? But how am I to repay you?" "Confide in me; trust me. Robert, the suspicions that were in your heart but an hour since—they will return." Robert shook his head, and swept the tears from his eyes. "No, no! even then I hated myself for them: how good, how forgiving, how generous you are! I am young, strong, have energy. In time this shameful debt can be paid—but kindness like this—how can I ever return that?" "Oh! opportunities for gratitude are never wanting: the bird we tend gives back music in return for care, yet what can be more feeble? Give me love, Robert, that is the music of a young heart—do not distrust me again!" "I never will!" Leicester wrung the youth's hand. They both arose. "If you are going to the counting-room, I will accompany you," he said, "my business must be negotiated with your firm." "I was first going to my room," said Robert. "No matter, I will walk slowly—by the way, here is your old copy-book; I have just been examining it. Those were pleasant evenings, my boy, when I taught you how to use the pen." "Yes," said Robert, receiving the book, "my dear aunt claims the old copies as a sort of heir-loom. I remembered your wish to see it, and so took it quietly away. I really think she would not have given it up, even to you." "Then she did not know when you took it?"
- 71. "No, I had forgotten it, and so stole down in the night. She was sound asleep, and I came away very early in the morning." "Dear old lady," said Leicester, smiling; "you must return her treasure before it is missed. Stay; fold your cloak over it. I shall see you again directly." Leicester's bed-chamber communicated with another small room, which was used as a dressing-closet. From some caprice he had draped the entrance with silken curtains such as clouded the windows. Scarcely had he left the room when this drapery was flung aside, revealing the door which had evidently stood open during his interview with Robert Otis. Jacob Strong closed the door very softly, but in evident haste; dropped the curtains over it, and taking a key from his pocket, let himself out of the bed- chamber. He overtook Robert Otis, a few paces from the hotel, and touched him upon the shoulder. "Mr. Otis, that copy-book—my master wishes to see it again—will you send it back?" "Certainly," answered Robert, producing the book. "But what on earth can he want it for?" "Come back with me, and I will tell you!" "I will," said Robert; "but remember, friend, no more hints against Mr. Leicester, I cannot listen to them." "I don't intend to hint anything against him now!" said Jacob, dryly, and they entered the hotel together. Jacob took the young man to his own little room, and the two were locked in together more than an hour. When the door opened, Jacob appeared composed and awkward as ever, but a powerful change had fallen upon the youth. His face was not only pale, but a look of wild horror disturbed his countenance. "Yet I will not believe it," he said, "it is too fiendish. In what have I ever harmed him?" "I do not ask you to believe, but to know. Keep out of the way a single week, it can do harm to no one." "But in less than a week this miserable debt must be paid!"
- 72. "Then pay it!" Robert smiled bitterly. "How? by ruining my aunt? Shall I ask her to sell the old homestead?" "She would do it—she would give up the last penny rather than see you disgraced, Robert Otis!" "How can you know this?" "I do know it, but this is not the question. Here is money to pay your debt, I have kept it in my pocket for weeks." Robert did not reach forth his hand to receive the roll of bank-notes held toward him, for surprise held him motionless. "Take the money, it is the exact sum," said Jacob, in a voice that carried authority with it. "I ask no promise that you never enter another gambling hall —you never will!" "Never!" said Robert, receiving the money; "but how—why have you done this?" "Ask me no questions now; by-and-bye you will know all about it; the money is mine. I have earned it honestly; as much more is all that I have in the world. No thanks! I never could bear them, besides it will be repaid in time!" "If I live," said Robert, with tears in his eyes. "This week, remember—this week you must be absent. A visit to the old homestead, anything that will take you out of town." "I will go," said Robert, "it can certainly do no harm." And they parted. Ada Leicester fled from the keen disappointment which almost crushed her for a time, and sought to drown all thought in the whirl of fashionable life. Her reception evenings were splendid. Beauty, talent, wit, everything that could charm or dazzle gathered beneath her roof. She gave herself no time for grief. Occasionally a thought of her husband would sting her into fresh bursts of excitement—sometimes the memory of her parents and her child passed over her heart, leaving a swell behind like that which followed the angels when they went down to trouble the still waters. Her wit grew more sparkling, her graceful sarcasm keener than ever it had been. She was the rage that season,
- 73. and exhausted her rich talent in efforts to win excitement. She did not hope for happiness from the homage and splendor that her beauty and wealth had secured; excitement was all she asked. When all other devices for amusement failed to keep up the fever of her artificial life, she bethought her of a new project. Her talent, her wealth must achieve something more brilliant than had yet been dreamed of, she would give a fancy ball, something far more picturesque than had ever been known in Saratoga or Newport. At first Ada thought of this ball only as a something that should pass like a rocket through the upper ten thousand; but as the project grew upon her, she resolved to make it an epoch in her own inner life. The man whom she had loved, the husband who had so coldly trampled her to the earth in her seeming poverty—he should witness this grand gala—he should see her in the fall blaze of her splendid career. There was something of proud retaliation in this; she fancied that it was resentful hate that prompted this desire to see and triumph over the man who had scorned her. Alas! poor woman, was there no lurking hope?—no feeling that she dared not call by its right name in all that wild excitement? She sent for Jacob, and besought him to devise some means by which Leicester should be won to attend the ball, without suspecting her identity. "Let it be superb—let it surpass everything hitherto known in elegance," she said—"he shall be here—he shall see the poor governess, the scorned wife in a new phase." There was triumph in her eyes as she spoke. "You love this man, even now, in spite of all that he has done?" said Jacob Strong, who stood before her while she spoke. "No," she answered—"no, I hate—oh! how I do hate him!" Jacob regarded her with a steady, fixed glance of the eye; he was afraid to believe her. He would not have believed her but for the powerful wish that gave an unnatural impulse to his faith. "He may be dazzled by all this splendor; the knowledge of so much wealth will make him humble—he will be your slave again!" Ada glanced around the sumptuous array of her boudoir. Her eyes sparkled; her lip quivered with haughty triumph.
- 74. "And I would spurn him even as he spurned me in that humble room over- head—that room filled with its wealth of old memories." Jacob turned away to hide the joy that burned in his eyes. "Oh! my mistress, say it again. In earnest truth, you hate this man; do not deceive yourself. Have you unwound the adder from your heart? Did that night do its work?" Ada Leicester paused; she was ashamed to own, even before that devoted servant, how closely the adder still folded himself in her bosom. She turned pale, but still answered with unfaltering voice, "Jacob, I hate him!" "Not yet—not as you ought to hate him," answered Jacob, regarding her pallid face so searchingly that his own cheek whitened, "but when you see him in all his villany, as I have seen him; when you know all!" "And do I not know all? What is it you keep from me? What is there to learn more vile—more terrible than the past?" "What if I tell you that within a month, William Leicester, your husband, will be married to another woman?" "Married! married to another!—Leicester—my——" she broke off, for her white lips refused to utter another syllable. After a momentary struggle she started up—"does he think that I am dead?—does he hope that night has killed me?" "He knows that you are living; but thinks you have returned to England." "But this is crime—punishable crime." "I know that it is." A faint, incredulous smile stole over her lips, and she waved her hand. "He will not violate the law; never was a bad man more prudent." "He will be married to-morrow night." "And to that girl? Does he love her so much? Is her beauty so overpowering? What has she to tempt Leicester into this crime?" "Her father is dead. By his will a large property falls to this poor girl. The letter came under cover to Leicester; he opened it. After the marriage they will sail for the north of Europe—there the letter will follow them, telling the poor orphan of her father's death. How can she guess that her husband has seen it before!"
- 75. "But I—I am not dead!" "You love him, he knows that better than you do. Death is no stronger safeguard than that knowledge. In your love or in your death he is equally safe." "God help me; but I will not be a slave to this abject love forever. If this last treachery be true, my soul will loathe him as he deserves." "It is true." "But my ball is to-morrow night. He accepted the invitation. You are certain that he will come?" "He accepted the invitation eagerly enough," said Jacob, dryly; "but what then?" "Why, to-morrow night—this cannot happen before to-morrow night—then I shall see him; after that—no, no, he dare not. You see, Jacob, it is in order to save him from deeper crime; we must not sit still and allow this poor girl to be sacrificed; that would be terrible. It must be prevented." "Nothing easier. Let him know that the brilliant, the wealthy Mrs. Gordon, is his wife; say that she has millions at her disposal; this poor girl has only one or two hundred thousand, the choice would be soon made." "Do you believe it? can you think it was belief in my poverty, and not—not a deeper feeling that made him so cruel that night? would he have accepted me for this wealth?" A painful red hovered in Ada's cheek, as she asked this question; it was shaping a humiliating doubt into words. It was exposing the scorpion that stung most keenly at her heart. Jacob drew closer to his mistress; he clasped her two hands between his, and his heavy frame bent over her, not awkwardly, for deep feeling is never awkward. "Oh, my mistress, say to me that you will give up this man—utterly give him up; even now you cannot guess how wicked he is; do not, by your wealth, help him to make new victims; do not see him and thus give him a right over yourself and your property—a right he will not fail to use; give up this ball; leave the city—this is no way to find that poor old man, that child——"
- 76. Welcome to our website – the perfect destination for book lovers and knowledge seekers. We believe that every book holds a new world, offering opportunities for learning, discovery, and personal growth. That’s why we are dedicated to bringing you a diverse collection of books, ranging from classic literature and specialized publications to self-development guides and children's books. More than just a book-buying platform, we strive to be a bridge connecting you with timeless cultural and intellectual values. With an elegant, user-friendly interface and a smart search system, you can quickly find the books that best suit your interests. Additionally, our special promotions and home delivery services help you save time and fully enjoy the joy of reading. Join us on a journey of knowledge exploration, passion nurturing, and personal growth every day! ebookbell.com