Docker in development

Editor's Notes

• #2 Hello and welcome! Let’s talk about Docker in Development.
• #3 First up, we should get the obligatory docker introduction out of the way. What is Docker? Briefly, Docker is a way for you to wrap up your application and its dependencies so they can be passed around, reused, and layered on top of each other.
• #4 Before we get too far, let’s look at a small sampling of companies that use Docker. This is just a notable selection who have publicly said they are using it in their infrastructure. Many more large companies are using it privately and even more small and medium-sized companies have jumped into the fray…
• #5 But why have they done this? Docker allows you, the developer or operations person, to take your application and it’s dependencies and wrap them up into a standardized format that can be used for development and deployment in a reusable, composable and sandboxed way. The last two parts, composable and sandboxed, are the interesting ones.
• #6 By default, a running instance of a docker image is like a standalone Linux machine that can make requests outwards, but is walled off from receiving incoming connections. Additionally, it is wrapped up in such a way that it thinks it is the entire machine. It sees it’s own network, it’s own processes and it’s own filesystem. Nothing else.
• #7 In order to get outside connections into the container, you tell Docker what ports to expose (80, 443, 5432, etc). You can also map these ports between containers directly, without them ever getting exposed to the parent Linux machine. This means the two containers can *only* talk to each other through that explicit connection. Now, why would you build your images to be so spread out? Docker believes in the concept of a single responsibility for each image. Ultimately, it’s up to you to decide what the scope of responsibility is, but I agree with many others that a single “service”, launched from a single root executable is that responsibility. It can launch workers underneath it but they should all be that same service.
• #8 Docker is able to accomplish these feats using a series of underlying technologies: cgroups, UnionFS and libcontainer, to name a few. Each one could be a presentation all it’s own as they all work independently of Docker. Docker is simply orchestrating each of them along with a set of standardized ways of working with the filesystem on the host. The result is something that works *like* a virtual machine but without the overhead of actually emulating the hardware and booting up a separate operating system for each application.
• #9 Moving on to how this helps us with our day-to-day development I would like to give a bit of background on how local development environments have been constructed in the past. If you’ve ever had to work on more than one application, especially when different tech stacks are involved, you may have run into an issue developing on your local system due to some dependency conflict. Different versions of databases, languages, etc.
• #10 The easiest way to solve this was to create a virtual machine, which can fully encapsulate everything your application needs, and it can even be configured to be similar to your staging and production environments. VirtualBox became a favorite of developers to fulfill this requirement because it was free and open source, worked generally as well as VMware, and could be scripted.
• #11 Using these scripting endpoints, the Vagrant tool came along and allowed you to programmatically define your virtual machine. It could also download pre-built machine images which already had things like Puppet and Chef installed, easing installation and setup. You could create your own starter images and even re-use some of your deployment recipes to get dependencies installed.
• #12 When Docker came along, it was quickly adopted by developers for testing out their deployments. It also came to be used for local development to spin up resources that could be dedicated to a project, without worrying about collisions with other projects. But there was a problem: It didn’t (and still doesn’t) run on Mac or Windows machines. Docker requires a Linux kernel under the hood to work. Boot2Docker solved this problem by controlling VirtualBox, much like Vagrant, managing a tiny virtual machine that comes pre-installed with Docker, and automatically linking it to the local host.
• #13 Once developers had a nice streamlined way of working with Docker on their local machines they started to run into the issue of how to orchestrate all the services they were running in conjunction with each other. Docker is powerful but it is also verbose and all command-line driven. Start this, then that, link those two together, now access the app… a pain every time you need to swap something out. Enter Fig, a simple orchestration tool built around a single YAML file, which let’s you define all your images and connections in one place, then use one command to bring them all up at once. It even combines all the service’s logs into a single terminal.
• #14 Fig turned out to be such a good solution that Docker, Inc. purchased the company that built it, in order to make it a key piece of their overall orchestration strategy. Renamed Docker Compose, it can continue to be used exactly the same way, or used with Docker’s two other new tools: Docker Machine, a kind of Vagrant for configuring machines running docker in production; and Docker Swarm, to help coordinate lots of Docker instances across different machines and networks. Docker Compose is now being developed to work for both local development and production uses.
• #15 Awesome, now that we have an idea of the tools available to us, let’s get concrete and talk about a specific application I have been working on. It’s a toy CRM I started to teach myself Docker and a few other technologies. Originally I was teaching myself how to do production deployment but development just fell out of it. Let’s look at what goes into this app.
• #16 I am using PostgreSQL as the primary data store. I prefer Postgres over MySQL for a number of reasons, some of which are listed here. PostgreSQL is also the “primary source of truth” for this application — other data stores will in some way be slaved to this for any long-lasting data.
• #17 I also have a second database, Redis, which is used for storing tasks the asynchronous workers will consume. It is fast, lightweight and requires no additional dependencies. Great for development and production.
• #18 Moving up to the application layer, Sidekiq is sitting on top of Redis to queue and dequeue worker tasks, as well as run the workers themselves.
• #19 And finally, the application itself is Rails. The key point here is that these are the tools and services I am using for *my* application. You can use whatever you want. MySQL, Mongo, Python, Go, something that uses the JVM, whatever. If it runs on Linux then you can probably containerize it.
• #20 Ok, let’s put this all together. Where to start? Let’s go ahead and install the dependencies we need. I’m going to assume you’re on a Mac with Homebrew installed. Docker, of course, runs natively on Linux and Boot2Docker supports Mac OS X and Windows alike. Links up here will have instructions to get you started on your preferred OS.
• #21 First up, we need VirtualBox installed. I used to install this manually using their instructions. These days `brew cask` works just as well. And since VirtualBox *still* doesn’t have an auto-update feature, I use it to upgrade my existing installation as well.
• #22 Next up is Boot2Docker. I use regular homebrew here and it includes Docker as a dependency, so that installation is taken care of for you.
• #23 Once it’s installed, let’s boot it up. Just a quick `boot2docker up` and it will download the VM image and create a new VirtualBox VM. I already have the image downloaded here but it’s only about 20 MB, so it goes by quickly. Once it’s booted, it will alter your environment with a few docker-specific environment variables to link things up…
• #24 Great, let’s make sure it worked. Running `docker version` should give you a version for the local command line client, as well as the “docker server”, which is located on the VM. When you talk to production Docker instances, it will work exactly the same way. It’s worth noting that currently the version of Docker must be the same on both sides or you will get an error. This is because the communication API has changed a lot over the past year and a misunderstood command could really mess up a deployment.
• #25 Great, now that we’re set up we need to create a Dockerfile for our application. A Dockerfile is basically just a recipe for how to build and run your application. Every line of the file must start with a command and commands can not wrap across lines. Each command is cached so if you change something that would require the image to be rebuilt, it can use the caches all the way up to where the change happened, then start from there. «describe the parts» «point out timestamps and not file hash is used to tell difference»
• #26 The rest of the parts of our application are pulled from existing images built by the Docker community. For common things like PostgreSQL, Redis, NginX and others, there are a set of “authorized” packages built by or for the organization that owns the service and Docker hosts them for everyone to use. Here is a portion of the docker-compose.yml file. I have defined two related things: a data-only container (which we can talk about in Q&A) and a normal Redis container, which will run the Redis service. «describe the parts»
• #27 Alright… so how does this all come together? docker-compose has a few commands you can run, one of them is “run” which takes the name of a defined container and executes the command given within the scope of that container. Here we will use this to run an app rake script which will create (or recreate) our database for the given environment, run migrations, and then seed it with initial data. Behind the scenes docker-compose has also spun up the postgres and redis instances, and made them available, simply because they are listed as dependencies of the app.
• #28 To recap: we installed VirtualBox and Boot2Docker, linked our local Docker install to the one within Boot2Docker so we can call it from the command line, created a Dockerfile to define our application, and created a docker-compose.yml file that will compile our app using that Dockerfile, as well as use several other images from the community. Then, to ensure our app will run, we bootstrapped it to run migrations and load initial data. Now we get to the best part. From now on, to bring up the app we just type one command: `docker-compose up` — this will boot up the whole app, dependencies first, and collate all the logs to the terminal. If we need to bundle new gems, we just `control-c`, run `docker-compose build` and then relaunch.
• #29 Let me show it to you live! «show the full Dockerfile and docker-compose file, docker-compose up and load the app in the browser»
• #30 Any questions?