PaaSTA: Running applications at Yelp
1 like•1,165 views
Nathan Handler, a Site Reliability Engineer at Yelp, discusses the transition from a monolithic application to a service-oriented architecture using PaaSTA (Platform as a Service). He outlines the benefits of smaller services for faster deployments and easier issue resolution, and provides technical details on components like Docker and Marathon for deploying applications. The document includes code snippets and configurations for service monitoring and deployment pipelines.
![FROM ubuntu:xenial
MAINTAINER Nathan Handler <nhandler@yelp.com>
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get -y install
apache2
libapache2-mod-php
ENV APACHE_RUN_USER www-data
ENV APACHE_RUN_GROUP www-data
ENV APACHE_LOG_DIR /var/log/apache2
ENV APACHE_LOCK_DIR /var/lock/apache2
ENV APACHE_PID_FILE /var/run/apache2.pid
RUN rm -f /var/www/html/index.html
COPY htdocs /var/www/html/
CMD ["/usr/sbin/apache2", "-D", "FOREGROUND", "-C", "listen 8888"]
EXPOSE 8888 19
$ cat Dockerfile](https://image.slidesharecdn.com/nathanhandleroscon2016paasta-160523162721/85/PaaSTA-Running-applications-at-Yelp-19-320.jpg)
![---
demo:
advertise: [region]
discover: region
proxy_port: 20973
38
$ cat smartstack.yaml](https://image.slidesharecdn.com/nathanhandleroscon2016paasta-160523162721/85/PaaSTA-Running-applications-at-Yelp-38-320.jpg)
PaaSTA: Running applications at Yelp
- 1. Nathan Handler [email protected] / @nathanhandler PaaSTA Running Applications at Yelp
- 2. ● Nathan Handler / @nathanhandler ● Site Reliability Engineer at Yelp ● Ubuntu/Debian Developer and freenode staff ● PaaSTA Developer Who am I? 2
- 3. Yelp’s Mission Connecting people with great local businesses. 3
- 4. Yelp Stats As of Q1 2016 90M 3270%102M 4
- 5. ● Monolithic Python application (~3M LoC) ● Builds/deployments took a long time ○ Bottleneck on how often we can deploy ● Mistakes are painful ○ Large impact ○ Difficult to find ○ Slow to fix History 5
- 6. ● Split features into different applications ● Smaller services allowed for faster pushes ● Easier to reason about issues ● Able to scale services independently Service Oriented Architecture v1 6
- 7. ● Standalone application ● Stateless ● Separate git repository ● Typically at Yelp: ○ HTTP API ○ Python, Pyramid, uWSGI ○ virtualenv What is a service? 7
- 8. ● Statically defined list of hosts to deploy a service on ● Operations decides which hosts to deploy to ● Monitoring manually configured in Nagios ● Manual deployment system via rsync Deploying Services v1 8
- 11. ● Yelp's Platform as a Service ● Builds, Deploys, Connects, and Monitors services ● Glue around existing and established open source tools PaaSTA 11 https://github.com/yelp/paasta #paasta on irc.freenode.net
- 12. 12
- 13. PaaSTA Components 13 Docker Registry Developer git push git pull git push docker push 13 Marathon Sensu
- 15. . ├── Dockerfile ├── htdocs │ ├── index.php │ └── status └── Makefile 15 A simple service
- 16. DOCKER_TAG ?= $(USER)-dev test: @echo 'Unit testing' itest: cook-image paasta local-run --healthcheck --service devops cook-image: docker build -t $(DOCKER_TAG) . 16 $ cat Makefile
- 17. DOCKER_TAG ?= $(USER)-dev test: @echo 'Unit testing' itest: cook-image paasta local-run --healthcheck --service devops cook-image: docker build -t $(DOCKER_TAG) . 17 $ cat Makefile
- 18. • Containers: like lightweight VMs • Provides a language (Dockerfile) for describing container image • Reproducible builds (mostly) • Provides software flexibility 18 Docker docker.com
- 19. FROM ubuntu:xenial MAINTAINER Nathan Handler <[email protected]> RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get -y install apache2 libapache2-mod-php ENV APACHE_RUN_USER www-data ENV APACHE_RUN_GROUP www-data ENV APACHE_LOG_DIR /var/log/apache2 ENV APACHE_LOCK_DIR /var/lock/apache2 ENV APACHE_PID_FILE /var/run/apache2.pid RUN rm -f /var/www/html/index.html COPY htdocs /var/www/html/ CMD ["/usr/sbin/apache2", "-D", "FOREGROUND", "-C", "listen 8888"] EXPOSE 8888 19 $ cat Dockerfile
- 20. ✓ yelpsoa-config directory for devops found in /nail/etc/serv ✓ deploy.yaml exists for a Jenkins pipeline ✗ No 'security-check' entry was found in your deploy.yaml. Please add a security-check entry *AFTER* the itest entry in d so your docker image can be checked against known security vul More info: http://servicedocs.yelpcorp.com/docs/paasta_tools/c ✗ No 'performance-check' entry was found in your deploy.yaml. Please add a performance-check entry *AFTER* the security-chec so your docker image can be checked for performance regression More info: http://servicedocs.yelpcorp.com/docs/paasta_tools/c ✓ Jenkins build pipeline found ✓ Git repo found in the expected location. ✓ Found Dockerfile ✓ A Makefile is present ✓ The Makefile contains a tab character ✓ The Makefile contains a docker tag ✓ The Makefile responds to `make cook-image` ✓ The Makefile responds to `make itest` ✓ The Makefile responds to `make test` ✓ Found marathon.yaml file. ✓ All entries in deploy.yaml correspond to a marathon or chro ✓ All marathon instances have a corresponding deploy.yaml ent ✓ monitoring.yaml found for Sensu monitoring ✓ Your service uses Sensu and team 'nhandler' will get alerts ✓ Found smartstack.yaml file ✓ Instance 'demo' of your service is using smartstack port 20 balanced ✓ Successfully validated schema: marathon-nova-devc.yaml 20 $ paasta check
- 21. PaaSTA Components 21 Developer git push git pull 21
- 22. PaaSTA Components 22 Docker Registry Developer git push git pull git push docker push 22
- 23. --- pipeline: - instancename: itest - instancename: push-to-registry - instancename: dev.everything 23 $ cat deploy.yaml
- 26. • Describe end goal, not path • Helps us achieve fault tolerance. "Deploy 6de16ff2 to prod" vs. "Commit 6de16ff2 should be running in prod" Gas pedal vs. Cruise Control 26 Declarative control
- 27. Description: A demo PaaSTA service for OSCON 2016 External Link: http://conferences.oreilly.com/oscon/open-source- us/public/schedule/detail/49358 Monitored By: team nhandler Runbook: Please set a `runbook` field in your monitoring.yaml. Like "y/rb-m Docs: https://trac.yelpcorp.com/wiki/HowToService/Monitoring/monitoring.yam Git Repo: [email protected]:services/devops Jenkins Pipeline: https://jenkins.yelpcorp.com/view/services-devops Deployed to the following clusters: - nova-devc (N/A) Smartstack endpoint(s): - http://169.254.255.254:20973 (demo) Dashboard(s): - https://uchiwa.yelpcorp.com/#/events?q=devops (Sensu Alerts) 27 $ paasta info
- 28. PaaSTA Components 28 Docker Registry Developer git push git pull git push docker push 28 Marathon
- 29. • Mesos is an "SDK for distributed systems", not batteries-included. • Requires a framework • Marathon • Chronos for scheduled batches • Can run many frameworks on the same cluster • Supports Docker as task executor 29 mesosphere.io mesos.apache.org Scheduling: Mesos + Marathon
- 30. --- demo: cpus: 0.1 instances: 3 mem: 500 monitoring: alert_after: 0m bounce_method: brutal deploy_group: dev.everything 30 $ cat marathon-nova-devc.yaml
- 31. PaaSTA Components 31 Docker Registry Developer git push git pull git push docker push 31 Marathon
- 32. ● Brutal: Stops old versions and starts the new version, without regard to safety. ● Upthendown: Brings up the new version of the service and waits until all instances are healthy before stopping the old versions. ● Downthenup: Stops any old versions and waits for them to die before starting the new version. ● Crossover: Starts the new version, and gradually kills instances of the old versions as new instances become healthy. Bounce Strategies 32
- 33. 33
- 34. 34
- 35. 35
- 36. 36 Discovery in PaaSTA: Smartstack mesos slave box2 client nerve HAProxy synapse box1 service nerve mesos slave synapse HAProxy ZooKeeperMetadata HTTP request healthcheck
- 38. --- demo: advertise: [region] discover: region proxy_port: 20973 38 $ cat smartstack.yaml
- 39. Pipeline: https://jenkins.yelpcorp.com/view/services-devops cluster: nova-devc instance: demo Git sha: 6de16ff2 State: Running - Desired state: Started Marathon: Healthy - up with (3/3) instances. Status: Running. Mesos: Healthy - (3/3) tasks in the TASK_RUNNING state. Smartstack: Name LastCheck LastChange Status useast1-devc - Healthy - in haproxy with (3/3) total backends UP in this namespace. 39 $ paasta status
- 40. Cluster: mesosstage Dashboards: Marathon RO: http://marathon.paasta-mesosstage.yelp/ Smartstack: http://paasta-mesosstage.yelp:3212 Chronos RO: http://chronos.paasta-mesosstage.yelp/ Mesos: http://mesos.paasta-mesosstage.yelp Mesos Status: OK quorum: masters: 3 configured quorum: 2 frameworks: framework: chronos-2.4.0 count: 1 framework: marathon count: 1 CPUs: 1.00 / 7 in use (14.29%) Memory: 3.03 / 42.85GB in use (7.07%) Disk: 10.00 / 153.81GB in use (6.50%) tasks: running: 9 staging: 1 starting: 0 slaves: active: 7 inactive: 0 Marathon Status: OK marathon apps: 5 marathon tasks: 9 marathon deployments: 0 Chronos Status: OK Enabled chronos jobs: 1 40 $ paasta metastatus
- 41. PaaSTA Components 41 Docker Registry Developer git push git pull git push docker push 41 Marathon Sensu
- 42. --- team: nhandler page: true notification_email: [email protected] 42 $ cat monitoring.yaml
- 43. 43
- 44. PaaSTA Components 44 Docker Registry Developer git push git pull git push docker push 44 Marathon Sensu
- 45. Autoscaling 45