Highly efficient container orchestration and continuous delivery with DC/OS and Jenkins presentation
0 likes288 views
The document outlines the integration of Continuous Delivery with DC/OS and Jenkins, emphasizing the components of modern infrastructure and how they enhance developer agility by allowing for rapid application deployment and bug fixes. It provides an overview of Apache Mesos and DC/OS, including their architecture, resource management, and service discovery capabilities. Additionally, it addresses traditional versus modern release processes, demonstrating the efficiency gained through a continuous feedback loop and automation with tools like Jenkins and Docker.
Highly efficient container orchestration and continuous delivery with DC/OS and Jenkins presentation
- 1. © 2016 Mesosphere, Inc. All Rights Reserved. VELOCITY NYC | SEPTEMBER 20, 2016 CONTINUOUS DELIVERY WITH DC/OS AND JENKINS
- 2. © 2016 Mesosphere, Inc. All Rights Reserved. SUNIL SHAH Product Manager at Mesosphere @ssk2 ROGER IGNAZIO Tech Lead at Mesosphere @rogerignazio WHO WE ARE
- 3. © 2016 Mesosphere, Inc. All Rights Reserved. mesosinaction.com Use the code vecymes for 42% off! BUY MY BOOK!
- 4. © 2016 Mesosphere, Inc. All Rights Reserved. AGENDA Lecture ● Introduction to Apache Mesos and DC/OS ● Components that make up modern infra. ● Running Jenkins as a service on DC/OS ● Continuously deploying applications to DC/OS Demos & Lab ● Installing and configuring Jenkins ● Installing and configuring a load balancer ● Creating a new CI/CD pipeline ● Putting it all together (CD in practice)
- 5. © 2016 Mesosphere, Inc. All Rights Reserved. DEVELOPER AGILITY, DEFINED
- 6. © 2016 Mesosphere, Inc. All Rights Reserved. Developer agility empowers developers to ● ship their apps to production ● leverage the power of Mesos and DC/OS ● fix bugs rapidly without downtime! DEVELOPER AGILITY, DEFINED
- 7. © 2016 Mesosphere, Inc. All Rights Reserved. Continuous integration is not the same as continuous delivery or continuous deployment. DEVELOPER AGILITY, DEFINED
- 8. © 2016 Mesosphere, Inc. All Rights Reserved. An artifact repo stores built binaries, whereas an image registry stores templates for deployment. DEVELOPER AGILITY, DEFINED
- 9. © 2016 Mesosphere, Inc. All Rights Reserved. DEVELOPER AGILITY, DEFINED
- 10. © 2016 Mesosphere, Inc. All Rights Reserved. INTRO TO APACHE MESOS AND DC/OS
- 11. © 2016 Mesosphere, Inc. All Rights Reserved. A QUICK PRIMER ON CONTAINERS Physical Server Virtual Machine–Based Application Deployment Container–Based Application Deployment Hypervisor OS VM VM OS App App Physical Server Hypervisor OS VM VM OS App App Physical Server Operating System App App Physical Server Operating System App App Mesos Isolate apps by running multiple VMs per physical server; still need to manage each guest OS! Isolate apps using features of the host OS, such as Linux cgroups.
- 12. © 2016 Mesosphere, Inc. All Rights Reserved. A QUICK PRIMER ON CONTAINERS Linux cgroup Dependencies Linux Host OS Application Docker Container Dependencies Host OS Application Linux cgroupsDocker Containers VM Dependencies Hypervisor Application Virtual Machines Guest OS Docker Engine
- 13. © 2016 Mesosphere, Inc. All Rights Reserved. A BIT OF CLARIFICATION https://mesos.apache.org https://dcos.io
- 14. © 2016 Mesosphere, Inc. All Rights Reserved. ● General purpose cluster resource manager ● Represents many machines as a single entity ● Advertises resources directly to frameworks ● Works at scale: Apple, Twitter, Airbnb, Netflix, … WHAT IS MESOS?
- 15. © 2016 Mesosphere, Inc. All Rights Reserved. ● Two-tier scheduling across resource types ● cpus, mem, disk, and ports by default ● Masters are highly available, agents are fault tolerant ● Checkpointing, agent recovery ● Resource isolation between processes ● Linux cgroups, Docker, … ● Language bindings: C++, Java, Python, Go, … WHAT IS MESOS? (CONTINUED)
- 16. © 2016 Mesosphere, Inc. All Rights Reserved. MESOS ARCHITECTURE Launch task(s) Mesos Master Quorum Mesos Master (Standby) ZK Mesos Master (Standby) ZK Mesos Master (Leader) ZK Framework B Scheduler Agent 1 Framework A Executor Task(s) Agent n Framework A Executor Task(s) Framework B Executor Task(s) Resource offer Resource offer Framework A Scheduler Launch task(s)
- 17. © 2016 Mesosphere, Inc. All Rights Reserved. ANATOMY OF A RESOURCE OFFER (TWO-TIER SCHEDULING) Mesos Master(s) Mesos Agent 16 CPUs 128 GB RAM 1 TB disk Mesos Agent 16 CPUs 128 GB RAM 1 TB disk Mesos Agent 16 CPUs 128 GB RAM 1 TB disk Master offers resources to scheduler Scheduler Executor Available compute resources Resource offer accepted, launch executors/tasks Tasks Executor Tasks Executor Tasks Scheduler accepts or declines an offer
- 18. © 2016 Mesosphere, Inc. All Rights Reserved. ● Service discovery and load balancing ● BIND, Mesos-DNS, Consul-Mesos, Marathon-LB NEW (OLD) PROBLEMS
- 19. © 2016 Mesosphere, Inc. All Rights Reserved. ● Service discovery and load balancing ● BIND, Mesos-DNS, Consul-Mesos, Marathon-LB ● Monitoring and metrics collection ● Collectd, Nagios, Prometheus, Snap NEW (OLD) PROBLEMS
- 20. © 2016 Mesosphere, Inc. All Rights Reserved. ● Service discovery and load balancing ● BIND, Mesos-DNS, Consul-Mesos, Marathon-LB ● Monitoring and metrics collection ● Collectd, Nagios, Prometheus, Snap ● Persistent storage (filesystems, databases, etc) ● Ceph, HDFS, Amazon EBS / EFS / S3, NFS, Cassandra NEW (OLD) PROBLEMS
- 21. © 2016 Mesosphere, Inc. All Rights Reserved. ● Service discovery and load balancing ● BIND, Mesos-DNS, Consul-Mesos, Marathon-LB ● Monitoring and metrics collection ● Collectd, Nagios, Prometheus, Snap ● Persistent storage (filesystems, databases, etc) ● Ceph, HDFS, Amazon EBS / EFS / S3, NFS, Cassandra ● Administration: named URIs vs. ports, IPAM ● Nginx, HAProxy, Mesos-DNS, dhcpd, Minuteman NEW (OLD) PROBLEMS
- 22. © 2016 Mesosphere, Inc. All Rights Reserved. https://dcos.io https://github.com/dcos DC/OS: BUILT ON MESOS
- 23. © 2016 Mesosphere, Inc. All Rights Reserved. DC/OS: BUILT ON MESOS
- 24. © 2016 Mesosphere, Inc. All Rights Reserved. All of the benefits of Mesos, plus ● Built-in service discovery and load balancing ● Support for stateful services ● Turn-key installation of distributed systems ● Cloud-agnostic installer ● Web and command-line interfaces ● All components are integration tested and supported by Mesosphere, Inc. MESOS AND DC/OS: BETTER TOGETHER
- 25. © 2016 Mesosphere, Inc. All Rights Reserved. JENKINS ON DC/OS
- 26. © 2016 Mesosphere, Inc. All Rights Reserved. MULTIPLE JENKINS MASTERS Team A Jenkins . . . Jenkins Agent 1 Jenkins Agent 2 . . . Jenkins Agent N Team B Jenkins Jenkins Agent 1 Jenkins Agent 2 . . . Jenkins Agent N Team X Jenkins Jenkins Agent 1 Jenkins Agent 2 . . . Jenkins Agent N
- 27. © 2016 Mesosphere, Inc. All Rights Reserved. MULTIPLE JENKINS MASTERS → STATIC PARTITIONING Team A Jenkins . . . Jenkins Agent 1 Jenkins Agent 2 . . . Jenkins Agent N Team B Jenkins Jenkins Agent 1 Jenkins Agent 2 . . . Jenkins Agent N Team X Jenkins Jenkins Agent 1 Jenkins Agent 2 . . . Jenkins Agent N 90% Utilized (Normal) 140% Utilized (40 builds in queue) 0% Utilized (Idle)
- 28. © 2016 Mesosphere, Inc. All Rights Reserved. RUNNING JENKINS ON DC/OS → FAIR SHARING Team A Jenkins . . . DC/OS Agent 1 DC/OS Agent 4 DC/OS Agent N Team B Jenkins DC/OS Agent 2 DC/OS Agent 5 Team X Jenkins DC/OS Agent 3 DC/OS Agent 6 . . . DC/OS
- 29. © 2016 Mesosphere, Inc. All Rights Reserved. CONTINUOUSLY DEPLOYING APPLICATIONS TO DC/OS
- 30. © 2016 Mesosphere, Inc. All Rights Reserved. TRADITIONAL RELEASE PROCESS ● Planning & implementing new technologies ● Waiting for people & infrastructure ● Building environment specific CI/CD for each project ● Moving apps from dev to staging to prod DEV(OPS) TEAMS SPEND SIGNIFICANT TIME AND EFFORT ON: PROJECT PLANNING WEEK 1 WEEK 2 WEEK 3 WEEKS 4 TO 8 WEEK 9 CUSTOMER FEEDBACKCONFIGURATIONRESOURCE ALLOCATION Dev Test Stage Release
- 31. © 2016 Mesosphere, Inc. All Rights Reserved. MODERN RELEASE PROCESS WEEK 1 WEEK 2 WEEK 3 WEEK 4 PROJECT PLANNING CUSTOMER FEEDBACK CUSTOMER FEEDBACK CUSTOMER FEEDBACK RELEASE 1 RELEASE 2 RELEASE 3 RELEASE 4 Better products through a repeatable release cadence Happier customers through continuous feedback Dev Test Stage Release Dev Test Stage Release Dev Test Stage Release Dev Test Stage Release
- 32. © 2016 Mesosphere, Inc. All Rights Reserved. ● Scheduling — advertising available compute resources ● Deployments — getting an application onto a node ● Health checks — ensuring the app/service is healthy ● Service discovery — connecting to dependent services ● Persistence — running stateful services in containers DEPLOYING APPLICATIONS: BASIC REQUIREMENTS
- 33. © 2016 Mesosphere, Inc. All Rights Reserved. A sysadmin provisions one or more physical/virtual servers to host the app Mesos resource offers (two-tier scheduling) offers available resources directly to frameworks DEPLOYING APPLICATIONS: SCHEDULING Before DC/OS With DC/OS
- 34. © 2016 Mesosphere, Inc. All Rights Reserved. By hand or using Puppet / Chef / Ansible Jenkins SSHing to the machine and running a shell script Note: all dependencies must also be present! Marathon deploys containers, ideally using a CI/CD tool to create/update app definitions Docker containers packages app and dependencies DEPLOYING APPLICATIONS: DEPLOYMENTS Before DC/OS With DC/OS
- 35. © 2016 Mesosphere, Inc. All Rights Reserved. Nagios pages a sysadmin Marathon performs health checks, restarts unhealthy/failed instances DEPLOYING APPLICATIONS: HEALTH CHECKS Before DC/OS With DC/OS
- 36. © 2016 Mesosphere, Inc. All Rights Reserved. Static hostnames / IP addresses in a spreadsheet or config management A sysadmin configures a load balancer manually or with Puppet / Chef / Ansible Mesos-DNS provides DNS resolution for running services (hostname / IP address, ports, etc) Load balancer configs built dynamically using cluster state DEPLOYING APPLICATIONS: SERVICE DISCOVERY Before DC/OS With DC/OS
- 37. © 2016 Mesosphere, Inc. All Rights Reserved. Individual servers with RAID 1/5/6/10, expensive SANs, NFS, etc. Dedicated, statically partitioned Ceph or Gluster storage clusters Mesos external/persistent volumes (REX-Ray), HDFS, etc. Self-healing Ceph or Gluster on Mesos / DC/OS DEPLOYING APPLICATIONS: PERSISTENCE Before DC/OS With DC/OS
- 38. © 2016 Mesosphere, Inc. All Rights Reserved. Q: Given that we are all engineers, which tools would we like to work with?
- 39. © 2016 Mesosphere, Inc. All Rights Reserved. DEMOS & LAB
- 40. © 2016 Mesosphere, Inc. All Rights Reserved. 40 PIPELINE COMPONENTS Continuous Integration Container Registry Container Orchestrator Version Control System Load Balancer Production Environment Continuous Delivery Pipeline git push DC/OS
- 41. © 2016 Mesosphere, Inc. All Rights Reserved. 41 PIPELINE COMPONENTS Jenkins DockerHub MarathonGitHub Marathon-lbMesos Continuous Delivery Pipeline git push DC/OS
- 42. © 2016 Mesosphere, Inc. All Rights Reserved. 42 PIPELINE CONFIGURATION Jenkinsfile Dockerfile marathon.json Continuous Delivery Pipeline git push DC/OS
- 43. © 2016 Mesosphere, Inc. All Rights Reserved. A SNEAK PREVIEW
- 44. © 2016 Mesosphere, Inc. All Rights Reserved. ● Head over to github.com/mesosphere/vny ● Follow the exercises! ● Use the username/password: velocityuser/velocitypassword YOUR TURN
- 45. © 2016 Mesosphere, Inc. All Rights Reserved. Roger Ignazio [email protected] @rogerignazio Sunil Shah [email protected] @ssk2 THANK YOU! Learn more by visiting DCOS.io and Mesosphere.com