Linux Containers and
Dockers
Quando, vantaggi e svantaggi
Ciao
ciao
Vai a fare
ciao ciao
Dr. Fabio Fumarola
Contents
• The Evolution of IT
• The Solutions: Virtual Machines vs Vagrant vs Docker
• Differences
• Examples
– Vagrant
– Boot2Docker
– Docker
– Docker Hub
• CoreOS
2
From 1995 to 2015
3
Client-Server
App
Well-defined stack:
- O/S
- Runtime
- Middleware
Monolithic
Physical
Infrastructure
Thin app on mobile,
tablet
Assembled by
developers using
best available
services
Running on any
available set of
physical resources
(public/private/
virtualized)
Static website
Web frontend
User DB
Queue Analytics DB
Background workers
API endpoint
nginx 1.5 + modsecurity + openssl + bootstrap
2
postgresql + pgv8 + v8
hadoop + hive + thrift + OpenJDK
Ruby + Rails + sass + Unicorn
Redis + redis-sentinel
Python 3.0 + celery + pyredis + libcurl + ffmpeg + libopencv
+ nodejs + phantomjs
Python 2.7 + Flask + pyredis + celery + psycopg + postgresql-
client
Development VM
QA server
Public Cloud
Disaster recovery
Contributor’s laptop
Production Servers
2015 in Detail
Production Cluster
Customer Data Center
4
Challenges
• How to ensure that services interact consistently?
• How to avoid to setup N different configurations and
dependencies for each service?
• How to migrate and scale quickly ensuring
compatibility?
• How to replicate my VM and services quickly?
5
How to deal with different confs?
6
Static website
Web frontend
Background workers
User DB
Analytics DB
Queue
Development
VM
QA Server
Single Prod
Server
Onsite Cluster Public Cloud
Contributor’s
laptop
Customer
Servers
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
1. Virtual Machines
7
Virtual Machines
• Run on top of an Hypervisor
Pros
– fully virtualized OS
– Totally isolated
Cons
– Needs to take a snapshot of
the entire VM to replicate
– Uses a lot of space
– Slow to move around
8
App
A
Hypervisor
Host OS
Server
Guest
OS
Bins/
Libs
App
A’
Guest
OS
Bins/
Libs
App
B
Guest
OS
Bins/
Libs
Guest
OS
Guest
OS
VM
Hypervisors Trend
2011
– XEN: Default choice given Rackspace and Amazon use
– KVM: Bleeding edge users
2012
– KVM: Emerges as the lead
– XEN: Loses momentum
9
Hipervisors Trend
2013
– KVM: Maintains lead (around 90%+ for Mirantis)
– Vmware: Emerges as a surprising second choice
– Containers (LXC, Parallels, Docker): Web Hosting and SAS
focused
– Xen and HyperV: Infrequent requests (XenServer.org)
2014 – 2015
– ???
10
2. Vagrant
11
Vagrant
• Open source VM manager released in 2010
• It allows you to script and package VMs config and
the provisioning setup via a VagrantFile
• It is designed to run on top of almost any VM tool:
VirtualBox, VMVare, AWS, OpenStack1
• It can be used together with provisioning tools such
as shell scripts, Chef and Puppet.
12
1. https://github.com/cloudbau/vagrant-openstack-plugin
Vagrant: idea
Use a VagrantFile to install
1.an operating system
2.Required libraries and
software
and finally run programs and
processes of your final
application
13
Vagrant: Feature
• Command-Line Interface
• Vagrant Share
• VagrantFile
• Boxes
• Provisioning
• Networking
• Synced Folders
• Multi-Machine
• Providers
• Plugins
14
https://www.vagrantup.com/downloads
Vagrant: Demo
• It allows us to interact with Vagrant
• It offers the following commands: box, connect,
destroy, halt, init, login, package a vm, rdp, …
https://docs.vagrantup.com/v2/cli/index.html
15
Vagrant Example
1. Download and install VirtualBox and Vagrant
1. This will place a VagrantFile in the directory
2. Install a Box
3. Using a Box -> https://vagrantcloud.com/
16
$ mkdir vagrant_first_vm && cd vagrant_first_vm
$ vagrant init
$ vagrant box add ubuntu/trusty64
Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/trusty64"
end
Vagran: Start
1. Start the box
2. Login into the vm
3. You can destroy the vm by
17
$ vagrant up
$ vagrant ssh
$ vagrant destroy
Vagrant: Synced Folders
• By default, it shares your project directory to the /vagrant
directory on the guest machine.
• If you create a file on your gues os the file will be on the
vagrant vm.
18
$ vagrant up
$ vagrant ssh
$ ls /vagrant
--Vagrantfile
$ touch pippo.txt
$vagrant ssh
$ls /vagrant/
Vagrant: Provisioning
• Let’s install Apache via a boostrap.sh file
• If you create a file on your gues os the file will be on the
vagrant vm. (vagrant reload --provision)
19
#!/usr/bin/env bash
apt-get update
apt-get install -y apache2
rm -rf /var/www
ln -fs /vagrant /var/www
Vagrant.configure("2") do |config|
config.vm.box = "hashicorp/precise32"
config.vm.provision :shell, path: "bootstrap.sh"
end
Vagrant: Networking
• Port Forwarding: llows you to specify ports on the guest
machine to share via a port on the host machine
• By running vagrant reload or vagrant up we can see on
http://127.0.0.1:4567 our apache
• It supports also bridge configurations and other
configurations (https://docs.vagrantup.com/v2/networking/)
20
Vagrant.configure("2") do |config|
config.vm.box = "hashicorp/precise32"
config.vm.provision :shell, path: "bootstrap.sh"
config.vm.network :forwarded_port, host: 4567, guest: 80
end
Vagrant: Share and Provider
• It is possible to share Vagrant box via vagrant cloud (but?)
Providers
• By default Vagrant is configured with VirtualBox but you can
change the provider
• How?
21
$ vagrant up --provider=vmware_fusion
$ vagrant up --provider=aws
$ vagrant plugin install vagrant-aws
Vagrant: AWS Vagrantfile
22
Vagrant.configure("2") do |config|
# config.vm.box = "sean"
config.vm.provider :aws do |aws, override|
aws.access_key_id = "AAAAIIIIYYYY4444AAAA”
aws.secret_access_key =
"c344441LooLLU322223526IabcdeQL12E34At3mm”
aws.keypair_name = "iheavy"
aws.ami = "ami-7747d01e"
override.ssh.username = "ubuntu"
override.ssh.private_key_path = "/var/root/iheavy_aws/pk-
XHHHHHMMMAABPEDEFGHOAOJH1QBH5324.pem"
end
end
3. Docker
23
Quick Survey
• How many people have heard of Docker before this
Seminar?
• How many people have tried Docker ?
• How many people are using Docker in production ?
24
What is Docker?
"With Docker, developers can build any app in any
language using any toolchain. “Dockerized” apps are
completely portable and can run anywhere -
colleagues’ OS X and Windows laptops, QA servers
running Ubuntu in the cloud, and production data
center VMs running Red Hat.”
Docker.io
25
Docker in simple words
• It is a technology that allow you running applications
inside containers (not VM)
• This assures that libraries and package needed by the
application you run are always the same.
• This means you can make a container for Memcache
and another for Redis and they will work the same in
any OS (also in Vagrant).
26
Why Docker?
• Fast delivery of your applications
• Deploy and scale more easily
• Get higher density and run more workload
• Faster deployment makes for easier management
27
How does docker work?
• LinuX Containers (LXC)
• Control Groups & Namespaces (CGroups)
• AUFS
• Client – Server with an HTTP API
28
LXC- Linux Containers
• It is a user-space interface for the Linux kernel containment
features
• Through a powerful API and simple tools, it lets Linux users easily
create and manage system or application containers.
• Currently LXC can apply the following kernel features to contain
processes:
– Kernel namespaces (ipc, uts, mount, pid, network and user)
– Apparmor and SELinux profiles
– Seccomp policies
– Chroots (using pivot_root)
– Kernel capabilities & Control groups (cgroups)
29
Cgroups
• Control groups is a Linux kernel feature to limit, account and
isolate resource usage (CPU, memory, disk I/O, etc) of process
groups.
• Features:
– Resource limitation: limit CPU, memory…
– Prioritization: assign more CPU etc to some groups.
– Accounting: to measure the resource usage.
– Control: freezing groups or check-pointing and restarting.
30
LCX based Containers
• It allows us to run a Linux system within another Linux system.
• A container is a group of processes on a Linux box, put together
is an isolated environment.
31
AppA’
Docker Engine
Host OS
Server
Bins/Libs
AppA
Bins/Libs
AppB
AppB’
AppB’
AppB’
AppB’
Container
• From the inside it looks like a VM
• From the outside, it looks like normal
processes
Docker Features
• VE (Virtual Environments) based on LXC
• Portable deployment across machines
• Versioning: docker include git-like capabilities for tracking
versions of a container
• Component reuse: it allows building or stacking already
created packages. You can create ‘base images’ and then
running more machine based on the image.
• Shared libraries: there is a public repository with several
images (https://registry.hub.docker.com/)
32
Why are Docker Containers lightweight?
33
Bins
/
Libs
App
A
Original App
(No OS to take
up space, resources,
or require restart)
AppΔ
Bins/
App
A
Bins/
Libs
App
A’
Gues
t
OS
Bins/
Libs
Modified App
Union file system allows
us to only save the diffs
Between container A
and container A’
VMs
App
A
Gues
t
OS
Bins/
Libs
Copy of
App
No OS. Can
Share bins/libs
App
A
Gues
t
OS
Gues
t
OS
Containers
Prerequisites
• I use Oh My Zsh1
with the Docker plugin2
for autocompletion
of docker commands
• Linux at least with kernel 3.8 but 3.10.x is recommended
– $ uname –r
• MacOS or Windows via Boot2Docker3
or via Vagrant
34
1. https://github.com/robbyrussell/oh-my-zsh
2. https://github.com/robbyrussell/oh-my-zsh/wiki/Plugins#docker
3. http://boot2docker.io/
Docker Installation Ubuntu
• AUFS support
$ sudo apt-get update
$ sudo apt-get intall linux-image-extra-`uname –r`
• Add docker repo
$ sudo sh –c “curl https://get.docker.io/gpg | apt-key add -”
$ sudo sh –c “echo deb http://get.docker.io/ubuntu docker 
main > /etc/apt/sources.list.d/docker.list”
• Install
$ sudo apt-get update
$ sudo apt-get install lxc-docker
35
Docker install Vagrant
• Create the folders
$ mkdir ~/boot2docker
$ cd ~/boot2docker
• Init the vagrant box
$ vagrant init yungsang/boot2docker
$ vagrant up; export DOCKER_HOST=tcp://localhost:2375
• Check docker
$ docker version
* NOTE: the YungSang boot2docker opens up port forwarding
to the network, so is not safe on public wifi.
36
Docker Installation Vagrant
• Clone the docker repository
$ git clone https://github.com/dotcloud/docker.git
• Startup the vagrant image
$ vagrant up
• SSH into the image
$ vagrant ssh
• Docker client works normally
37
Docker install boot2docker
http://boot2docker.io/
38
Base Commands
39
Docker: hello world
• Get one base image from https://registry.hub.docker.com
$ sudo docker pull centos
• List images on your system
$ sudo docker images
• Check the images
–$ sudo docker images
• Run your first container
–$ sudo docker run centos:latest echo “hello world”
40
An Interactive Container
• Run bash in your container
– $ sudo docker run -t -i centos /bin/bash
• The -t flag assigns a pseudo-tty or terminal inside our new
container
• The -i flag allows us to make an interactive connection by
grabbing the standard in (STDIN) of the container
• We also specified a command for the container
41
A Daemonized Hello world
• Run a sh script
– sudo docker run -d centos:6 /bin/sh –c ‘while true; do echo hello
world; sleep 1; done’
• The -d flag tells Docker to run the container and put it in the
background, to daemonize it.
• To list the docker containers running
– $ docker ps
• To get the logs of the container
– $ sudo docker logs container_id
• To stop the container:
– $ sudo docker stop container_id
42
A web container with docker
• To run a Python Flask application
– $ sudo docker run -d -P training/webapp python app.py
• The -P flag is new and tells Docker to map any required
network ports inside our container to our host.
• To view our application with the port mapping
– $ sudo docker ps –l
• We can see that the default flask port 5000 is exposed to
49155
– $ sudo docker run -d -p 5000:5000 training/webapp python app.py
• Check the url to continue the guide
– https://docs.docker.com/userguide/usingdocker/
43
Working with docker images
• To find images go to
– https://hub.docker.com/
• To pull an image
– $ sudo docker pull training/sinatra
• Updating and committing an image
– $ sudo docker run -t -i training/sinatra /bin/bash
– # gem install json
– $ sudo docker commit -m="Added json gem" -a="Kate Smith" 
0b2616b0e5a8 ouruser/sinatra:v2
- $ sudo docker images
44
Create an image from a Dockerfile
FROM library/centos:centos6
MAINTAINER fabio fumarola fabiofumarola@gmail.com
RUN yum install -y curl which tar sudo openssh-server openssh-clients rsync
# passwordless ssh
RUN ssh-keygen -q -N "" -t dsa -f /etc/ssh/ssh_host_dsa_key
RUN ssh-keygen -q -N "" -t rsa -f /etc/ssh/ssh_host_rsa_key
RUN ssh-keygen -q -N "" -t rsa -f /root/.ssh/id_rsa
RUN cp /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys
EXPOSE 22
CMD ["/usr/sbin/sshd", "-D"]
45
Build and run an image
• $docker build –t fabio/centos:ssh .
• $docker run –i –t fabio/centos:ssh /bin/bash
• Or
• $docker run –d fabio/centos:ssh /bin/bash
• Check the following commands:
– $ docker top
– $ docker logs
– $ docker inspect
46
Other Commands
• Docker cp: copy a file from container to host
• Docker diff: print container changes
• Docker top: display running processes in a container
• Docker rm /rmi: delete container/image
• Docker wait: wait until container stop and print exit code
More on: http://docs.docker.io/en/latest/commandline/cli
47
Big insight with Docker
48
Steps
1. Build the docker image via the docker file
– $ docker build –t ouruser/biginsight:v ./
1. Run the container
– docker run -d –p --name biginsight –v
/abs/install/path:/opt/ibm ouruser/biginsight:v /bin/sh
path/to/install.sh
This will mount the host directory, /abs/install/path,
into the container at /opt/ibm
49
Install Hadoop Spark Hbase
50
Steps
• There are several dockerfile that we can use
– http://blog.sequenceiq.com/blog/2014/12/02/hadoop-2-
6-0-docker/
– https://github.com/sequenceiq?page=2&query=docker
– https://registry.hub.docker.com/u/oddpoet/hbase-cdh5/
51
Docker vs Vagrant?
• Less memory for Dockers w.r.t VMs
• With a VM you get more isolation, but is much heavier.
Indeed you can run 1000 of Dockers in a machine but not
thousand of VMs with Xen.
• A VM requires minutes to start a Docker seconds
There are pros and cons for each type.
• If you want full isolation with guaranteed resources a full VM
is the way to go.
• If you want hundred of isolate processes into a reasonably
sized host then Docker might be the best solution
52
Core OS
53
CoreOS
• A minimal operating system
• Painless updating: utilizes active/passive scheme to update
the OS as single unit instead of package by package.
• Docker container
• Clustered by default
• Distributed System tools: etcd key-value store
• Service discovery: easily locate where service are running in
the cluster
• High availability and automatic fail-over
54
CoreOS
55
Clustered by default
High availability and a
utomatic fail-over
Docker with CoreOS
Features
•Automatically runs on each CoreOS
machine
•Updated with regular automatic OS
updates
•Integrates with etcd
•Networking automatically configured
Example Akka cluster + Docker + CoreOS
https://github.com/dennybritz/akka-
cluster-deploy
56
References
• http://www.iheavy.com/2014/01/16/how-to-deploy-on-amazon-ec2-
with-vagrant/
• https://docs.vagrantup.com/v2/
• Vagrant: Up and Running Paperback – June 15, 2013
• https://github.com/patrickdlee/vagrant-examples
• https://linuxcontainers.org/ LXC
• https://www.kernel.org/doc/Documentation/cgroups/
• http://lamejournal.com/2014/09/19/vagrant-vs-docker-osx-tales-front/
• https://medium.com/@_marcos_otero/docker-vs-vagrant-582135beb623
• https://coreos.com/using-coreos/docker/
57

Develop with linux containers and docker

  • 1.
    Linux Containers and Dockers Quando,vantaggi e svantaggi Ciao ciao Vai a fare ciao ciao Dr. Fabio Fumarola
  • 2.
    Contents • The Evolutionof IT • The Solutions: Virtual Machines vs Vagrant vs Docker • Differences • Examples – Vagrant – Boot2Docker – Docker – Docker Hub • CoreOS 2
  • 3.
    From 1995 to2015 3 Client-Server App Well-defined stack: - O/S - Runtime - Middleware Monolithic Physical Infrastructure Thin app on mobile, tablet Assembled by developers using best available services Running on any available set of physical resources (public/private/ virtualized)
  • 4.
    Static website Web frontend UserDB Queue Analytics DB Background workers API endpoint nginx 1.5 + modsecurity + openssl + bootstrap 2 postgresql + pgv8 + v8 hadoop + hive + thrift + OpenJDK Ruby + Rails + sass + Unicorn Redis + redis-sentinel Python 3.0 + celery + pyredis + libcurl + ffmpeg + libopencv + nodejs + phantomjs Python 2.7 + Flask + pyredis + celery + psycopg + postgresql- client Development VM QA server Public Cloud Disaster recovery Contributor’s laptop Production Servers 2015 in Detail Production Cluster Customer Data Center 4
  • 5.
    Challenges • How toensure that services interact consistently? • How to avoid to setup N different configurations and dependencies for each service? • How to migrate and scale quickly ensuring compatibility? • How to replicate my VM and services quickly? 5
  • 6.
    How to dealwith different confs? 6 Static website Web frontend Background workers User DB Analytics DB Queue Development VM QA Server Single Prod Server Onsite Cluster Public Cloud Contributor’s laptop Customer Servers ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
  • 7.
  • 8.
    Virtual Machines • Runon top of an Hypervisor Pros – fully virtualized OS – Totally isolated Cons – Needs to take a snapshot of the entire VM to replicate – Uses a lot of space – Slow to move around 8 App A Hypervisor Host OS Server Guest OS Bins/ Libs App A’ Guest OS Bins/ Libs App B Guest OS Bins/ Libs Guest OS Guest OS VM
  • 9.
    Hypervisors Trend 2011 – XEN:Default choice given Rackspace and Amazon use – KVM: Bleeding edge users 2012 – KVM: Emerges as the lead – XEN: Loses momentum 9
  • 10.
    Hipervisors Trend 2013 – KVM:Maintains lead (around 90%+ for Mirantis) – Vmware: Emerges as a surprising second choice – Containers (LXC, Parallels, Docker): Web Hosting and SAS focused – Xen and HyperV: Infrequent requests (XenServer.org) 2014 – 2015 – ??? 10
  • 11.
  • 12.
    Vagrant • Open sourceVM manager released in 2010 • It allows you to script and package VMs config and the provisioning setup via a VagrantFile • It is designed to run on top of almost any VM tool: VirtualBox, VMVare, AWS, OpenStack1 • It can be used together with provisioning tools such as shell scripts, Chef and Puppet. 12 1. https://github.com/cloudbau/vagrant-openstack-plugin
  • 13.
    Vagrant: idea Use aVagrantFile to install 1.an operating system 2.Required libraries and software and finally run programs and processes of your final application 13
  • 14.
    Vagrant: Feature • Command-LineInterface • Vagrant Share • VagrantFile • Boxes • Provisioning • Networking • Synced Folders • Multi-Machine • Providers • Plugins 14 https://www.vagrantup.com/downloads
  • 15.
    Vagrant: Demo • Itallows us to interact with Vagrant • It offers the following commands: box, connect, destroy, halt, init, login, package a vm, rdp, … https://docs.vagrantup.com/v2/cli/index.html 15
  • 16.
    Vagrant Example 1. Downloadand install VirtualBox and Vagrant 1. This will place a VagrantFile in the directory 2. Install a Box 3. Using a Box -> https://vagrantcloud.com/ 16 $ mkdir vagrant_first_vm && cd vagrant_first_vm $ vagrant init $ vagrant box add ubuntu/trusty64 Vagrant.configure("2") do |config| config.vm.box = "ubuntu/trusty64" end
  • 17.
    Vagran: Start 1. Startthe box 2. Login into the vm 3. You can destroy the vm by 17 $ vagrant up $ vagrant ssh $ vagrant destroy
  • 18.
    Vagrant: Synced Folders •By default, it shares your project directory to the /vagrant directory on the guest machine. • If you create a file on your gues os the file will be on the vagrant vm. 18 $ vagrant up $ vagrant ssh $ ls /vagrant --Vagrantfile $ touch pippo.txt $vagrant ssh $ls /vagrant/
  • 19.
    Vagrant: Provisioning • Let’sinstall Apache via a boostrap.sh file • If you create a file on your gues os the file will be on the vagrant vm. (vagrant reload --provision) 19 #!/usr/bin/env bash apt-get update apt-get install -y apache2 rm -rf /var/www ln -fs /vagrant /var/www Vagrant.configure("2") do |config| config.vm.box = "hashicorp/precise32" config.vm.provision :shell, path: "bootstrap.sh" end
  • 20.
    Vagrant: Networking • PortForwarding: llows you to specify ports on the guest machine to share via a port on the host machine • By running vagrant reload or vagrant up we can see on http://127.0.0.1:4567 our apache • It supports also bridge configurations and other configurations (https://docs.vagrantup.com/v2/networking/) 20 Vagrant.configure("2") do |config| config.vm.box = "hashicorp/precise32" config.vm.provision :shell, path: "bootstrap.sh" config.vm.network :forwarded_port, host: 4567, guest: 80 end
  • 21.
    Vagrant: Share andProvider • It is possible to share Vagrant box via vagrant cloud (but?) Providers • By default Vagrant is configured with VirtualBox but you can change the provider • How? 21 $ vagrant up --provider=vmware_fusion $ vagrant up --provider=aws $ vagrant plugin install vagrant-aws
  • 22.
    Vagrant: AWS Vagrantfile 22 Vagrant.configure("2")do |config| # config.vm.box = "sean" config.vm.provider :aws do |aws, override| aws.access_key_id = "AAAAIIIIYYYY4444AAAA” aws.secret_access_key = "c344441LooLLU322223526IabcdeQL12E34At3mm” aws.keypair_name = "iheavy" aws.ami = "ami-7747d01e" override.ssh.username = "ubuntu" override.ssh.private_key_path = "/var/root/iheavy_aws/pk- XHHHHHMMMAABPEDEFGHOAOJH1QBH5324.pem" end end
  • 23.
  • 24.
    Quick Survey • Howmany people have heard of Docker before this Seminar? • How many people have tried Docker ? • How many people are using Docker in production ? 24
  • 25.
    What is Docker? "WithDocker, developers can build any app in any language using any toolchain. “Dockerized” apps are completely portable and can run anywhere - colleagues’ OS X and Windows laptops, QA servers running Ubuntu in the cloud, and production data center VMs running Red Hat.” Docker.io 25
  • 26.
    Docker in simplewords • It is a technology that allow you running applications inside containers (not VM) • This assures that libraries and package needed by the application you run are always the same. • This means you can make a container for Memcache and another for Redis and they will work the same in any OS (also in Vagrant). 26
  • 27.
    Why Docker? • Fastdelivery of your applications • Deploy and scale more easily • Get higher density and run more workload • Faster deployment makes for easier management 27
  • 28.
    How does dockerwork? • LinuX Containers (LXC) • Control Groups & Namespaces (CGroups) • AUFS • Client – Server with an HTTP API 28
  • 29.
    LXC- Linux Containers •It is a user-space interface for the Linux kernel containment features • Through a powerful API and simple tools, it lets Linux users easily create and manage system or application containers. • Currently LXC can apply the following kernel features to contain processes: – Kernel namespaces (ipc, uts, mount, pid, network and user) – Apparmor and SELinux profiles – Seccomp policies – Chroots (using pivot_root) – Kernel capabilities & Control groups (cgroups) 29
  • 30.
    Cgroups • Control groupsis a Linux kernel feature to limit, account and isolate resource usage (CPU, memory, disk I/O, etc) of process groups. • Features: – Resource limitation: limit CPU, memory… – Prioritization: assign more CPU etc to some groups. – Accounting: to measure the resource usage. – Control: freezing groups or check-pointing and restarting. 30
  • 31.
    LCX based Containers •It allows us to run a Linux system within another Linux system. • A container is a group of processes on a Linux box, put together is an isolated environment. 31 AppA’ Docker Engine Host OS Server Bins/Libs AppA Bins/Libs AppB AppB’ AppB’ AppB’ AppB’ Container • From the inside it looks like a VM • From the outside, it looks like normal processes
  • 32.
    Docker Features • VE(Virtual Environments) based on LXC • Portable deployment across machines • Versioning: docker include git-like capabilities for tracking versions of a container • Component reuse: it allows building or stacking already created packages. You can create ‘base images’ and then running more machine based on the image. • Shared libraries: there is a public repository with several images (https://registry.hub.docker.com/) 32
  • 33.
    Why are DockerContainers lightweight? 33 Bins / Libs App A Original App (No OS to take up space, resources, or require restart) AppΔ Bins/ App A Bins/ Libs App A’ Gues t OS Bins/ Libs Modified App Union file system allows us to only save the diffs Between container A and container A’ VMs App A Gues t OS Bins/ Libs Copy of App No OS. Can Share bins/libs App A Gues t OS Gues t OS Containers
  • 34.
    Prerequisites • I useOh My Zsh1 with the Docker plugin2 for autocompletion of docker commands • Linux at least with kernel 3.8 but 3.10.x is recommended – $ uname –r • MacOS or Windows via Boot2Docker3 or via Vagrant 34 1. https://github.com/robbyrussell/oh-my-zsh 2. https://github.com/robbyrussell/oh-my-zsh/wiki/Plugins#docker 3. http://boot2docker.io/
  • 35.
    Docker Installation Ubuntu •AUFS support $ sudo apt-get update $ sudo apt-get intall linux-image-extra-`uname –r` • Add docker repo $ sudo sh –c “curl https://get.docker.io/gpg | apt-key add -” $ sudo sh –c “echo deb http://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list” • Install $ sudo apt-get update $ sudo apt-get install lxc-docker 35
  • 36.
    Docker install Vagrant •Create the folders $ mkdir ~/boot2docker $ cd ~/boot2docker • Init the vagrant box $ vagrant init yungsang/boot2docker $ vagrant up; export DOCKER_HOST=tcp://localhost:2375 • Check docker $ docker version * NOTE: the YungSang boot2docker opens up port forwarding to the network, so is not safe on public wifi. 36
  • 37.
    Docker Installation Vagrant •Clone the docker repository $ git clone https://github.com/dotcloud/docker.git • Startup the vagrant image $ vagrant up • SSH into the image $ vagrant ssh • Docker client works normally 37
  • 38.
  • 39.
  • 40.
    Docker: hello world •Get one base image from https://registry.hub.docker.com $ sudo docker pull centos • List images on your system $ sudo docker images • Check the images –$ sudo docker images • Run your first container –$ sudo docker run centos:latest echo “hello world” 40
  • 41.
    An Interactive Container •Run bash in your container – $ sudo docker run -t -i centos /bin/bash • The -t flag assigns a pseudo-tty or terminal inside our new container • The -i flag allows us to make an interactive connection by grabbing the standard in (STDIN) of the container • We also specified a command for the container 41
  • 42.
    A Daemonized Helloworld • Run a sh script – sudo docker run -d centos:6 /bin/sh –c ‘while true; do echo hello world; sleep 1; done’ • The -d flag tells Docker to run the container and put it in the background, to daemonize it. • To list the docker containers running – $ docker ps • To get the logs of the container – $ sudo docker logs container_id • To stop the container: – $ sudo docker stop container_id 42
  • 43.
    A web containerwith docker • To run a Python Flask application – $ sudo docker run -d -P training/webapp python app.py • The -P flag is new and tells Docker to map any required network ports inside our container to our host. • To view our application with the port mapping – $ sudo docker ps –l • We can see that the default flask port 5000 is exposed to 49155 – $ sudo docker run -d -p 5000:5000 training/webapp python app.py • Check the url to continue the guide – https://docs.docker.com/userguide/usingdocker/ 43
  • 44.
    Working with dockerimages • To find images go to – https://hub.docker.com/ • To pull an image – $ sudo docker pull training/sinatra • Updating and committing an image – $ sudo docker run -t -i training/sinatra /bin/bash – # gem install json – $ sudo docker commit -m="Added json gem" -a="Kate Smith" 0b2616b0e5a8 ouruser/sinatra:v2 - $ sudo docker images 44
  • 45.
    Create an imagefrom a Dockerfile FROM library/centos:centos6 MAINTAINER fabio fumarola [email protected] RUN yum install -y curl which tar sudo openssh-server openssh-clients rsync # passwordless ssh RUN ssh-keygen -q -N "" -t dsa -f /etc/ssh/ssh_host_dsa_key RUN ssh-keygen -q -N "" -t rsa -f /etc/ssh/ssh_host_rsa_key RUN ssh-keygen -q -N "" -t rsa -f /root/.ssh/id_rsa RUN cp /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys EXPOSE 22 CMD ["/usr/sbin/sshd", "-D"] 45
  • 46.
    Build and runan image • $docker build –t fabio/centos:ssh . • $docker run –i –t fabio/centos:ssh /bin/bash • Or • $docker run –d fabio/centos:ssh /bin/bash • Check the following commands: – $ docker top – $ docker logs – $ docker inspect 46
  • 47.
    Other Commands • Dockercp: copy a file from container to host • Docker diff: print container changes • Docker top: display running processes in a container • Docker rm /rmi: delete container/image • Docker wait: wait until container stop and print exit code More on: http://docs.docker.io/en/latest/commandline/cli 47
  • 48.
  • 49.
    Steps 1. Build thedocker image via the docker file – $ docker build –t ouruser/biginsight:v ./ 1. Run the container – docker run -d –p --name biginsight –v /abs/install/path:/opt/ibm ouruser/biginsight:v /bin/sh path/to/install.sh This will mount the host directory, /abs/install/path, into the container at /opt/ibm 49
  • 50.
  • 51.
    Steps • There areseveral dockerfile that we can use – http://blog.sequenceiq.com/blog/2014/12/02/hadoop-2- 6-0-docker/ – https://github.com/sequenceiq?page=2&query=docker – https://registry.hub.docker.com/u/oddpoet/hbase-cdh5/ 51
  • 52.
    Docker vs Vagrant? •Less memory for Dockers w.r.t VMs • With a VM you get more isolation, but is much heavier. Indeed you can run 1000 of Dockers in a machine but not thousand of VMs with Xen. • A VM requires minutes to start a Docker seconds There are pros and cons for each type. • If you want full isolation with guaranteed resources a full VM is the way to go. • If you want hundred of isolate processes into a reasonably sized host then Docker might be the best solution 52
  • 53.
  • 54.
    CoreOS • A minimaloperating system • Painless updating: utilizes active/passive scheme to update the OS as single unit instead of package by package. • Docker container • Clustered by default • Distributed System tools: etcd key-value store • Service discovery: easily locate where service are running in the cluster • High availability and automatic fail-over 54
  • 55.
    CoreOS 55 Clustered by default Highavailability and a utomatic fail-over
  • 56.
    Docker with CoreOS Features •Automaticallyruns on each CoreOS machine •Updated with regular automatic OS updates •Integrates with etcd •Networking automatically configured Example Akka cluster + Docker + CoreOS https://github.com/dennybritz/akka- cluster-deploy 56
  • 57.
    References • http://www.iheavy.com/2014/01/16/how-to-deploy-on-amazon-ec2- with-vagrant/ • https://docs.vagrantup.com/v2/ •Vagrant: Up and Running Paperback – June 15, 2013 • https://github.com/patrickdlee/vagrant-examples • https://linuxcontainers.org/ LXC • https://www.kernel.org/doc/Documentation/cgroups/ • http://lamejournal.com/2014/09/19/vagrant-vs-docker-osx-tales-front/ • https://medium.com/@_marcos_otero/docker-vs-vagrant-582135beb623 • https://coreos.com/using-coreos/docker/ 57

Editor's Notes