Scaling Jenkins with Docker: Swarm, Kubernetes or Mesos?
7 likes4,571 views
The document discusses various methods for scaling Jenkins with Docker Swarm, Kubernetes, or Mesos, highlighting the pros and cons of each approach. It explains the importance of managing build agents, memory and CPU limits, persistent storage, and networking for effective Jenkins operations. Furthermore, it outlines the different plugins available for Docker and Jenkins integration, as well as pipeline implementations for dynamic agents using these technologies.
![CAVEATS
Only a limited number of EBS volumes can be mounted
Docs say /dev/sd[f-p], but /dev/sd[q-z]seem to
work too
NFS users must be centralized and match in cluster and NFS
server](https://image.slidesharecdn.com/scalingjenkinswithdockerswarmkubernetesormesos-160917004606/85/Scaling-Jenkins-with-Docker-Swarm-Kubernetes-or-Mesos-34-320.jpg)
![JENKINS KUBERNETES PIPELINE
podTemplate(label: 'mypod', containers: [
[name: 'jnlp', image: 'jenkinsci/jnlp-slave:alpine', args: '${compute
[name: 'maven', image: 'maven:3-jdk-8', ttyEnabled: true, command:
[name: 'golang', image: 'golang:1.6', ttyEnabled: true, command:
]) {
node ('mypod') {
stage 'Get a Maven project'
git 'https://github.com/jenkinsci/kubernetes-plugin.git'
container('maven') {
stage 'Build a Maven project'
sh 'mvn clean install'
}
stage 'Get a Golang project'
git url: 'https://github.com/hashicorp/terraform.git'
container('golang') {
stage 'Build a Go project'
sh """
mkdir -p /go/src/github.com/hashicorp
ln -s `pwd` /go/src/github.com/hashicorp/terraform
cd /go/src/github.com/hashicorp/terraform && make core-dev
"""
}
}](https://image.slidesharecdn.com/scalingjenkinswithdockerswarmkubernetesormesos-160917004606/85/Scaling-Jenkins-with-Docker-Swarm-Kubernetes-or-Mesos-56-320.jpg)
Scaling Jenkins with Docker: Swarm, Kubernetes or Mesos?
- 1. SCALING JENKINS WITH DOCKER SWARM, KUBERNETES OR MESOS? / /Carlos Sanchez csanchez.org @csanchez Watch online at carlossg.github.io/presentations
- 2. ABOUT ME Engineer @ CloudBees, Private SaaS Edition Team Contributor to Jenkins Mesos plugin Jenkins and Maven official Docker images Author of Jenkins Kubernetes plugin Long time OSS contributor at Apache, Eclipse, Puppet,…
- 5. BUT IT IS NOT TRIVIAL
- 6. CLUSTER SCHEDULING Running in public cloud, private cloud, VMs or bare metal HA and fault tolerant With Docker support of course
- 8. A distributed systems kernel
- 9. APACHE MESOS Started before 2011 Runs tasks, any binary or Docker, rkt, appcimages Frameworks run on top of Mesos Mesosphere Marathon: long running services Apache Aurora: long running services Chronos: distributed cron-like system Used in Twitter, Airbnb, eBay, Apple, Verizon, Yelp,...
- 10. DOCKER SWARM
- 11. DOCKER SWARM By Docker Inc. Uses the same Docker API No need to modify existing tooling
- 12. DOCKER ENGINE SWARM MODE New in Docker 1.12 No need to install extra so ware, each daemon can run as a Swarm member New serviceobject to describe distributed containers Existing tooling needs to be updated Swarm mode
- 14. KUBERNETES Based on Google Borg Run in local machine, virtual, cloud Google provides Google Container Engine (GKE) Other services run by stackpoint.io, CoreOS Tectonic, Azure,... Minikube for local testing
- 15. SCALING JENKINS Two options: More build agents per master More masters
- 16. SCALING JENKINS: MORE BUILD AGENTS Pros Multiple plugins to add more agents, even dynamically Cons The master is still a SPOF Handling multiple configurations, plugin versions,... There is a limit on how many build agents can be attached
- 17. SCALING JENKINS: MORE MASTERS Pros Different sub-organizations can self service and operate independently Cons Single Sign-On Centralized configuration and operation Covered by CloudBees Jenkins Operations Center and CloudBees Jenkins Platform Private SaaS Edition
- 19. If you haven't automatically destroyed something by mistake, you are not automating enough
- 22. CLUSTER SCHEDULING Isolated build agents and jobs Using Docker Capabilities can be dropped
- 23. GROUPING CONTAINERS Example: Jenkins agent Maven build Selenium testing in Firefox Chrome Safari 5 containers
- 24. GROUPING CONTAINERS Mesos In progress Swarm Supports grouping through Docker Compose Can force execution in the same host Kubernetes Supports the concept of Pods natively All running in the same host MESOS-2449
- 25. MEMORY LIMITS Scheduler needs to account for container memory requirements and host available memory Prevent containers for using more memory than allowed Mesos required Swarm optional Kubernetes optional (plus namespaces) Memory constrains translate to Docker --memory
- 26. WHAT DO YOU THINK HAPPENS WHEN? Your container goes over memory quota?
- 28. WHAT ABOUT THE JVM? WHAT ABOUT THE CHILD PROCESSES?
- 29. CPU LIMITS Scheduler needs to account for container CPU requirements and host available CPUs Mesos required Swarm optional Kubernetes optional (plus namespaces) CPU translates into Docker --cpu-shares
- 30. WHAT DO YOU THINK HAPPENS WHEN? Your container tries to access more than one CPU Your container goes over CPU limits
- 31. Totally different from memory
- 32. STORAGE Handling distributed storage Jenkins masters need persistent storage, agents (typically) don't Mesos in 1.0+ Swarm Docker volume plugins: RexRay, Convoy, Flocker,... Kubernetes Docker volume support Persistent volumes
- 33. PERMISSIONS Containers should not run as root Container user id != host user id i.e. jenkinsuser in container is always 1000 but matches ubuntuuser in host
- 34. CAVEATS Only a limited number of EBS volumes can be mounted Docs say /dev/sd[f-p], but /dev/sd[q-z]seem to work too NFS users must be centralized and match in cluster and NFS server
- 35. NETWORKING Jenkins masters open several ports HTTP JNLP Build agent SSH server (Jenkins CLI type operations) Jenkins agents connect to master: inbound (SSH) outbound (JNLP)
- 36. Allows getting one IP per container Mesos : Calico, Weave Swarm , and others from plugins Kubernetes Multiple : GCE, Weave, Calico,... Network Isolator Modules Docker overlay networking options
- 37. JENKINS PLUGINS
- 38. JENKINS DOCKER PLUGINS Dynamic Jenkins agents with Docker plugin or Yet Another Docker Plugin No support yet for Docker 1.12 Swarm mode Agent image needs to include Java, downloads slave jar from Jenkins master Multiple plugins for different tasks Docker build and publish Docker build step plugin CloudBees Docker Hub/Registry Notification CloudBees Docker Traceability Great pipeline support
- 43. JENKINS DOCKER PIPELINE def maven = docker.image('maven:3.3.9-jdk-8'); stage 'Mirror' maven.pull() docker.withRegistry('https://secure-registry/', 'docker-registry-login' stage 'Build' maven.inside { sh "mvn -B clean package" } stage 'Bake Docker image' def pcImg = docker.build("examplecorp/spring-petclinic:${env.BUILD_TAG}" pcImg.push(); }
- 44. JENKINS DOCKER SLAVES PLUGIN Use any Docker image, no need for Java Definition in pipeline Can have side containers Just released!
- 46. Building Maven dockerNode("maven:3.3.3-jdk-8") { sh "mvn -version" }
- 47. JENKINS MESOS PLUGIN Dynamic Jenkins agents, both Docker and isolated processes Agent image needs to include Java, grabs slave jar from Mesos sandbox Can run Docker commands on the host, outside of Mesos
- 53. JENKINS MESOS PLUGIN Can use Docker pipelines with some tricks Need Docker client installed Shared docker.sock from host Mount the workspace in the host, visible under same dir
- 54. MESOS PLUGIN AND PIPELINE node('docker') { docker.image('golang:1.6').inside { stage 'Get sources' git url: 'https://github.com/hashicorp/terraform.git', tag: "v0.6.15" stage 'Build' sh """#!/bin/bash -e mkdir -p /go/src/github.com/hashicorp ln -s `pwd` /go/src/github.com/hashicorp/terraform pushd /go/src/github.com/hashicorp/terraform make core-dev plugin-dev PLUGIN=provider-aws popd cp /go/bin/terraform-provider-aws . """ stage 'Archive' archive "terraform-provider-aws" } }
- 55. JENKINS KUBERNETES PLUGIN Dynamic Jenkins agents, running as Pods Multiple container support One jnlp image, others custom Pipeline support for both agent Pod definition and execution will be in next version
- 56. JENKINS KUBERNETES PIPELINE podTemplate(label: 'mypod', containers: [ [name: 'jnlp', image: 'jenkinsci/jnlp-slave:alpine', args: '${compute [name: 'maven', image: 'maven:3-jdk-8', ttyEnabled: true, command: [name: 'golang', image: 'golang:1.6', ttyEnabled: true, command: ]) { node ('mypod') { stage 'Get a Maven project' git 'https://github.com/jenkinsci/kubernetes-plugin.git' container('maven') { stage 'Build a Maven project' sh 'mvn clean install' } stage 'Get a Golang project' git url: 'https://github.com/hashicorp/terraform.git' container('golang') { stage 'Build a Go project' sh """ mkdir -p /go/src/github.com/hashicorp ln -s `pwd` /go/src/github.com/hashicorp/terraform cd /go/src/github.com/hashicorp/terraform && make core-dev """ } }
- 57. JENKINS PLUGINS RECAP Dynamic Jenkins agent creation Using JNLP slave jar In complex environments need to use the tunnel option to connect internally Using the Cloud API Not ideal for containerized workload Agents take > 1 min to start provision and are kept around Agents can provide more than one executor
- 58. JENKINS ONE SHOT EXECUTOR Improved API to handle one off agents Optimized for containerized agents Plugins need to support it