Testing Distributed Micro Services. Agile Testing Days 2017
0 likes598 views
The document outlines the implementation of distributed microservices using Jenkins, Docker, and Apache Mesos. It describes infrastructure setup for a scaled Jenkins environment, container management for building and testing applications, and the use of Terraform for infrastructure automation in AWS and OpenStack. Emphasis is placed on best practices for testing, monitoring, and deployment strategies in a cloud-based architecture.
![DOCKERFILE
FROM openjdk:8-jdk
RUN apt-get update && apt-get install -y git curl && rm -rf /var/lib/apt/list
ARG JENKINS_VERSION
ENV JENKINS_VERSION ${JENKINS_VERSION:-2.19.3}
ARG JENKINS_SHA=e97670636394092af40cc626f8e07b092105c07b
ARG JENKINS_URL=https://repo.jenkins-ci.org/public/org/jenkins-ci/main/jenkin
RUN curl -fsSL ${JENKINS_URL} -o /usr/share/jenkins/jenkins.war
&& echo "${JENKINS_SHA} /usr/share/jenkins/jenkins.war" | sha1sum -c -
COPY jenkins-support /usr/local/bin/jenkins-support
COPY jenkins.sh /usr/local/bin/jenkins.sh
ENTRYPOINT ["/usr/local/bin/jenkins.sh"]](https://image.slidesharecdn.com/testingdistributedmicro-services-170112101410/85/Testing-Distributed-Micro-Services-Agile-Testing-Days-2017-21-320.jpg)
![JENKINS PIPELINE WITH MULTIPLE CONTAINERS
pcImg = docker.image("examplecorp/spring-petclinic:dev")
stage 'Test'
docker.withRegistry('https://dev.docker.example.com/',
'docker-registry-login') {
pcImg.withRun {petclinic ->
testImg.inside("--link=${petclinic.id}:petclinic") {
wrap([$class: 'Xvnc',
takeScreenshot: true,
useXauthority: true]) {
sh "mvn -B clean test"
}
}
}
}
stage 'Promote'
docker.withRegistry('https://staging.docker.example.com/',
'docker-registry-login') {
pcImg.push()
}](https://image.slidesharecdn.com/testingdistributedmicro-services-170112101410/85/Testing-Distributed-Micro-Services-Agile-Testing-Days-2017-36-320.jpg)
![KUBERNETES PLUGIN PIPELINES
Ex. run maven tests against webapp with Selenium, using
pre-made Docker images
podTemplate(label: 'petclinic', containers: [
containerTemplate(
name: 'maven', image: 'maven:3.3.9-jdk-8-alpine',
ttyEnabled: true, command: 'cat'),
containerTemplate(
name: 'petclinic', image: 'csanchez/petclinic'),
containerTemplate(
name: 'selenium', image: 'selenium/standalone-firefox')
]) {
node ('petclinic') {
stage 'Get a Maven project'
git 'https://github.com/jenkinsci/kubernetes-plugin.git'
container('maven') {
stage 'Build a Maven project'
sh 'cd demo/test && mvn -B clean test'
}
}
}](https://image.slidesharecdn.com/testingdistributedmicro-services-170112101410/85/Testing-Distributed-Micro-Services-Agile-Testing-Days-2017-41-320.jpg)
![TERRAFORM
resource "aws_instance" "worker" {
count = 1
instance_type = "m3.large"
ami = "ami-xxxxxx"
key_name = "tiger-csanchez"
security_groups = ["sg-61bc8c18"]
subnet_id = "subnet-xxxxxx"
associate_public_ip_address = true
tags {
Name = "tiger-csanchez-worker-1"
"cloudbees:pse:cluster" = "tiger-csanchez"
"cloudbees:pse:type" = "worker"
}
root_block_device {
volume_size = 50
}
}](https://image.slidesharecdn.com/testingdistributedmicro-services-170112101410/85/Testing-Distributed-Micro-Services-Agile-Testing-Days-2017-45-320.jpg)
Testing Distributed Micro Services. Agile Testing Days 2017
- 1. TESTING DISTRIBUTED MICRO-SERVICES Carlos Sanchez @csanchez csanchez.org Watch online at carlossg.github.io/presentations
- 2. ABOUT ME Senior So ware Engineer @ CloudBees Author of Jenkins Kubernetes plugin Long time OSS contributor at Apache Maven, Eclipse, Puppet,… Google Cloud Platform "Expert"
- 5. OUR USE CASE Scaling Jenkins Your mileage may vary
- 9. A 2000 JENKINS MASTERS CLUSTER 3 Mesos masters (m3.xlarge: 4 vCPU, 15GB, 2x40 SSD) 317 Mesos slaves (c3.2xlarge, m3.xlarge, m4.4xlarge) 7 Mesos slaves dedicated to ElasticSearch: (c3.8xlarge: 32 vCPU, 60GB) 12.5 TB - 3748 CPU Running 2000 masters and ~8000 concurrent jobs
- 10. ARCHITECTURE
- 12. Isolated Jenkins masters Isolated build agents and jobs Memory and CPU limits
- 15. CLUSTER SCHEDULING Distribute tasks across a cluster of hosts HA and fault tolerant With Docker support of course
- 16. INFRASTRUCTURE Running in public cloud, private cloud, VMs or bare metal
- 17. APACHE MESOS & MESOSPHERE MARATHON A distributed systems kernel
- 18. ALTERNATIVES Docker Swarm / Kubernetes
- 20. THE DOCKERFILE A lot like a shell script RUN comands COPY files ...
- 21. DOCKERFILE FROM openjdk:8-jdk RUN apt-get update && apt-get install -y git curl && rm -rf /var/lib/apt/list ARG JENKINS_VERSION ENV JENKINS_VERSION ${JENKINS_VERSION:-2.19.3} ARG JENKINS_SHA=e97670636394092af40cc626f8e07b092105c07b ARG JENKINS_URL=https://repo.jenkins-ci.org/public/org/jenkins-ci/main/jenkin RUN curl -fsSL ${JENKINS_URL} -o /usr/share/jenkins/jenkins.war && echo "${JENKINS_SHA} /usr/share/jenkins/jenkins.war" | sha1sum -c - COPY jenkins-support /usr/local/bin/jenkins-support COPY jenkins.sh /usr/local/bin/jenkins.sh ENTRYPOINT ["/usr/local/bin/jenkins.sh"]
- 22. Mocking and stubbing are your friends
- 23. BUILDING WITH JENKINS DOCKER PIPELINE def maven = docker.image('maven:3.3.9-jdk-8'); stage 'Mirror' maven.pull() docker.withRegistry('https://secure-registry/', 'docker-registry-login') { stage 'Build' maven.inside { sh "mvn -B clean package" } stage 'Bake Docker image' def pcImg = docker.build( "examplecorp/spring-petclinic:${env.BUILD_TAG}", 'app') pcImg.push(); }
- 24. TESTING WITH JENKINS DOCKER PIPELINE Build + test + promotion Promotion = different Docker registries for different environments dev staging production
- 25. TESTING WITH JENKINS DOCKER PIPELINE pcImg = docker.image("examplecorp/spring-petclinic:dev") stage 'Test' docker.withRegistry('https://dev.docker.example.com/', 'docker-registry-login') { pcImg.withRun { petclinic -> sh "test -f /var/some_file" } } stage 'Promote' docker.withRegistry('https://staging.docker.example.com/', 'docker-registry-login') { pcImg.push() }
- 26. DOCKER WORKFLOW PLUGIN DEMO https://github.com/jenkinsci/docker-workflow- plugin/tree/master/demo
- 27. USING BATS Testing using shell scripts! bats tests/install-plugins.bats Examples from https://github.com/jenkinsci/docker/tree/master/tests
- 28. BATS #!/usr/bin/env bats SUT_IMAGE=bats-jenkins load 'test_helper/bats-support/load' load 'test_helper/bats-assert/load' load test_helpers @test "plugins are installed with plugins.sh" { run docker build -t $SUT_IMAGE-plugins $BATS_TEST_DIRNAME/plugins assert_success run bash -c "docker run --rm $SUT_IMAGE-plugins ls -1 /var/jenkins_home/plugins" assert_success assert_line 'maven-plugin.jpi' assert_line 'maven-plugin.jpi.pinned' assert_line 'ant.jpi' assert_line 'ant.jpi.pinned' }
- 29. BATS @test "test jenkins arguments" { local version=$(grep 'ENV JENKINS_VERSION' Dockerfile | sed -e 's/.*:-(.*)}/1/') # need the last line of output assert "${version}" docker run --rm --name $SUT_CONTAINER -P $SUT_IMAGE --help --version | tail -n 1 }
- 30. ISOLATION There is no 100% isolation
- 31. MEMORY ISSUES WITH CONTAINERS Scheduler needs to account for container memory requirements and host available memory Prevent containers for using more memory than allowed Memory constrains translate to Docker --memory
- 32. WHAT DO YOU THINK HAPPENS WHEN? Your container goes over memory quota?
- 34. WHAT ABOUT THE JVM? WHAT ABOUT THE CHILD PROCESSES?
- 35. END TO END TESTING OF MULTIPLE CONTAINERS
- 36. JENKINS PIPELINE WITH MULTIPLE CONTAINERS pcImg = docker.image("examplecorp/spring-petclinic:dev") stage 'Test' docker.withRegistry('https://dev.docker.example.com/', 'docker-registry-login') { pcImg.withRun {petclinic -> testImg.inside("--link=${petclinic.id}:petclinic") { wrap([$class: 'Xvnc', takeScreenshot: true, useXauthority: true]) { sh "mvn -B clean test" } } } } stage 'Promote' docker.withRegistry('https://staging.docker.example.com/', 'docker-registry-login') { pcImg.push() }
- 38. DOCKER COMPOSE Runs multiple containers Containers can access each other
- 39. DOCKER COMPOSE version: '2' services: example: image: acme/example:latest ports: - "5050:5050" maven: image: maven:3-jdk-8 command: mvn test
- 40. JENKINS KUBERNETES PLUGIN The Jenkins job can run in a Kubernetes Pod (group of containers) Containers in a Pod can access each other at localhost
- 41. KUBERNETES PLUGIN PIPELINES Ex. run maven tests against webapp with Selenium, using pre-made Docker images podTemplate(label: 'petclinic', containers: [ containerTemplate( name: 'maven', image: 'maven:3.3.9-jdk-8-alpine', ttyEnabled: true, command: 'cat'), containerTemplate( name: 'petclinic', image: 'csanchez/petclinic'), containerTemplate( name: 'selenium', image: 'selenium/standalone-firefox') ]) { node ('petclinic') { stage 'Get a Maven project' git 'https://github.com/jenkinsci/kubernetes-plugin.git' container('maven') { stage 'Build a Maven project' sh 'cd demo/test && mvn -B clean test' } } }
- 42. HOW ARE WE DOING IT
- 43. HOW ARE WE DOING IT On each commit and PR Provisioning of the infrastructure using Terraform Installation of the cluster scheduler (Mesos & Marathon) Continuously creating clusters from scratch
- 45. TERRAFORM resource "aws_instance" "worker" { count = 1 instance_type = "m3.large" ami = "ami-xxxxxx" key_name = "tiger-csanchez" security_groups = ["sg-61bc8c18"] subnet_id = "subnet-xxxxxx" associate_public_ip_address = true tags { Name = "tiger-csanchez-worker-1" "cloudbees:pse:cluster" = "tiger-csanchez" "cloudbees:pse:type" = "worker" } root_block_device { volume_size = 50 } }
- 46. TERRAFORM State is managed Runs are idempotent terraform apply Sometimes it is too automatic Changing image id will restart all instances
- 48. IF YOU HAVEN'T AUTOMATICALLY DESTROYED SOMETHING BY MISTAKE, YOU ARE NOT AUTOMATING ENOUGH
- 49. HOW ARE WE DOING IT In AWS and OpenStack 5 different combinations More combinations on demand
- 50. HOW ARE WE DOING IT A er creation we launch acceptance tests Some python scripts Some Selenium tests Clusters get destroyed at the end
- 51. MONITORING IS THE NEW TESTING We gather from the cluster: logs configuration outputs Attached to the build to diagnose errors in CI but also used by our customers to send us information
- 52. HOW ARE WE DOING IT Feedback is published to Github PR
- 55. SCALING New and interesting problems
- 56. AWS Resource limits: VPCs, S3 snapshots, some instance sizes Rate limits: affect the whole account Always use different accounts for testing/production and possibly different teams Retrying is your friend, but with exponential backoff
- 57. EMBRACE FAILURE!
- 58. OPENSTACK Custom flavors Custom images Different CLI commands There are not two OpenStack installations that are the same
- 59. DANKE! RATE THIS SESSION IN AGILETESTINGDAYS.COM csanchez.org csanchez carlossg