From Monolith to Docker Distributed Applications
1 like1,026 views
The document discusses the transition from monolithic to Docker-based distributed applications, focusing on scaling Jenkins with a large cluster of Jenkins masters and utilizing Apache Mesos for task distribution. It details the architecture, resource management, and the use of Terraform for infrastructure as code, while exploring container scheduling, networking, and dynamic Jenkins agents. The document also highlights various plugins and pipeline support for managing Docker images and integrating with Jenkins and Kubernetes.
![TERRAFORM
resource "aws_instance" "worker" {
count = 1
instance_type = "m3.large"
ami = "ami-xxxxxx"
key_name = "tiger-csanchez"
security_groups = ["sg-61bc8c18"]
subnet_id = "subnet-xxxxxx"
associate_public_ip_address = true
tags {
Name = "tiger-csanchez-worker-1"
"cloudbees:pse:cluster" = "tiger-csanchez"
"cloudbees:pse:type" = "worker"
}
root_block_device {
volume_size = 50
}
}](https://image.slidesharecdn.com/frommonolithtodockerdistributedapplications-161121115244/85/From-Monolith-to-Docker-Distributed-Applications-21-320.jpg)
![JENKINS KUBERNETES PIPELINE
podTemplate(label: 'mypod', containers: [
[name: 'jnlp', image: 'jenkinsci/jnlp-slave:alpine', args: '${compute
[name: 'maven', image: 'maven:3-jdk-8', ttyEnabled: true, command:
[name: 'golang', image: 'golang:1.6', ttyEnabled: true, command:
]) {
node ('mypod') {
stage 'Get a Maven project'
git 'https://github.com/jenkinsci/kubernetes-plugin.git'
container('maven') {
stage 'Build a Maven project'
sh 'mvn clean install'
}
stage 'Get a Golang project'
git url: 'https://github.com/hashicorp/terraform.git'
container('golang') {
stage 'Build a Go project'
sh """
mkdir -p /go/src/github.com/hashicorp
ln -s `pwd` /go/src/github.com/hashicorp/terraform
cd /go/src/github.com/hashicorp/terraform && make core-dev
"""
}
}](https://image.slidesharecdn.com/frommonolithtodockerdistributedapplications-161121115244/85/From-Monolith-to-Docker-Distributed-Applications-58-320.jpg)
From Monolith to Docker Distributed Applications
- 1. FROM MONOLITH TO DOCKER DISTRIBUTED APPLICATIONS Carlos Sanchez @csanchez csanchez.org
- 2. ABOUT ME Senior So ware Engineer @ CloudBees Author of Jenkins Kubernetes plugin Long time OSS contributor at Apache Maven, Eclipse, Puppet,…
- 5. OUR USE CASE Scaling Jenkins Your mileage may vary
- 10. A 2000 JENKINS MASTERS CLUSTER 3 Mesos masters (m3.xlarge: 4 vCPU, 15GB, 2x40 SSD) 317 Mesos slaves (c3.2xlarge, m3.xlarge, m4.4xlarge) 7 Mesos slaves dedicated to ElasticSearch: (c3.8xlarge: 32 vCPU, 60GB) 12.5 TB - 3748 CPU Running 2000 masters and ~8000 concurrent jobs
- 11. ARCHITECTURE
- 13. Isolated Jenkins masters Isolated build agents and jobs Memory and CPU limits
- 16. CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, private cloud, VMs or bare metal HA and fault tolerant With Docker support of course
- 17. APACHE MESOS A distributed systems kernel
- 18. ALTERNATIVES Docker Swarm / Kubernetes
- 20. TERRAFORM
- 21. TERRAFORM resource "aws_instance" "worker" { count = 1 instance_type = "m3.large" ami = "ami-xxxxxx" key_name = "tiger-csanchez" security_groups = ["sg-61bc8c18"] subnet_id = "subnet-xxxxxx" associate_public_ip_address = true tags { Name = "tiger-csanchez-worker-1" "cloudbees:pse:cluster" = "tiger-csanchez" "cloudbees:pse:type" = "worker" } root_block_device { volume_size = 50 } }
- 22. TERRAFORM State is managed Runs are idempotent terraform apply Sometimes it is too automatic Changing image id will restart all instances
- 24. IF YOU HAVEN'T AUTOMATICALLY DESTROYED SOMETHING BY MISTAKE, YOU ARE NOT AUTOMATING ENOUGH
- 25. STORAGE Handling distributed storage Servers can start in any host of the cluster And they can move when they are restarted
- 28. PERMISSIONS Containers should not run as root Container user id != host user id i.e. jenkins user in container is always 1000 but matches ubuntu user in host
- 29. MEMORY Scheduler needs to account for container memory requirements and host available memory Prevent containers for using more memory than allowed Memory constrains translate to Docker --memory
- 30. WHAT DO YOU THINK HAPPENS WHEN? Your container goes over memory quota?
- 32. WHAT ABOUT THE JVM? WHAT ABOUT THE CHILD PROCESSES?
- 33. CPU Scheduler needs to account for container CPU requirements and host available CPUs
- 34. WHAT DO YOU THINK HAPPENS WHEN? Your container tries to access more than one CPU Your container goes over CPU limits
- 35. Totally different from memory Mesos/Kubernetes CPU translates into Docker --cpu- shares
- 36. NETWORKING Multiple services running in the same ports Must redirect from random ports in the host Services running in one host need to access services in other hosts
- 37. NETWORKING: SOFTWARE DEFINED NETWORKS Create new custom networks on top of physical networks Allow grouping containers in subnets
- 38. NETWORKING: SOFTWARE DEFINED NETWORKS Battlefield: Calico, Flannel, Weave and Docker Overlay Network http://chunqi.li/2015/11/15/Battlefield-Calico-Flannel- Weave-and-Docker-Overlay-Network/
- 39. SCALING New and interesting problems
- 40. AWS Resource limits: VPCs, S3 snapshots, some instance sizes Rate limits: affect the whole account Retrying is your friend, but with exponential backoff
- 41. EMBRACE FAILURE!
- 42. JENKINS PLUGINS
- 43. JENKINS DOCKER PLUGINS Dynamic Jenkins agents with Docker plugin or Yet Another Docker Plugin No support yet for Docker 1.12 Swarm mode Agent image needs to include Java, downloads slave jar from Jenkins master Multiple plugins for different tasks Docker build and publish Docker build step plugin CloudBees Docker Hub/Registry Notification CloudBees Docker Traceability Great pipeline support
- 48. JENKINS DOCKER PIPELINE def maven = docker.image('maven:3.3.9-jdk-8'); stage 'Mirror' maven.pull() docker.withRegistry('https://secure-registry/', 'docker-registry-login' stage 'Build' maven.inside { sh "mvn -B clean package" } stage 'Bake Docker image' def pcImg = docker.build("examplecorp/spring-petclinic:${env.BUILD_TAG}" pcImg.push(); }
- 49. JENKINS MESOS PLUGIN Dynamic Jenkins agents, both Docker and isolated processes Agent image needs to include Java, grabs slave jar from Mesos sandbox Can run Docker commands on the host, outside of Mesos
- 55. JENKINS MESOS PLUGIN Can use Docker pipelines with some tricks Need Docker client installed Shared docker.sock from host Mount the workspace in the host, visible under same dir
- 56. MESOS PLUGIN AND PIPELINE node('docker') { docker.image('golang:1.6').inside { stage 'Get sources' git url: 'https://github.com/hashicorp/terraform.git', tag: "v0.6.15" stage 'Build' sh """#!/bin/bash -e mkdir -p /go/src/github.com/hashicorp ln -s `pwd` /go/src/github.com/hashicorp/terraform pushd /go/src/github.com/hashicorp/terraform make core-dev plugin-dev PLUGIN=provider-aws popd cp /go/bin/terraform-provider-aws . """ stage 'Archive' archive "terraform-provider-aws" } }
- 57. JENKINS KUBERNETES PLUGIN Dynamic Jenkins agents, running as Pods Multiple container support One jnlp image, others custom Pipeline support for both agent Pod definition and execution will be in next version
- 58. JENKINS KUBERNETES PIPELINE podTemplate(label: 'mypod', containers: [ [name: 'jnlp', image: 'jenkinsci/jnlp-slave:alpine', args: '${compute [name: 'maven', image: 'maven:3-jdk-8', ttyEnabled: true, command: [name: 'golang', image: 'golang:1.6', ttyEnabled: true, command: ]) { node ('mypod') { stage 'Get a Maven project' git 'https://github.com/jenkinsci/kubernetes-plugin.git' container('maven') { stage 'Build a Maven project' sh 'mvn clean install' } stage 'Get a Golang project' git url: 'https://github.com/hashicorp/terraform.git' container('golang') { stage 'Build a Go project' sh """ mkdir -p /go/src/github.com/hashicorp ln -s `pwd` /go/src/github.com/hashicorp/terraform cd /go/src/github.com/hashicorp/terraform && make core-dev """ } }
- 59. JENKINS PLUGINS RECAP Dynamic Jenkins agent creation Using JNLP slave jar In complex environments need to use the tunnel option to connect internally Using the Cloud API Not ideal for containerized workload Agents take > 1 min to start provision and are kept around Agents can provide more than one executor