Immutable Infrastructure with Docker and EC2
Download as PPTX, PDF•1 like•555 views
This document discusses Gilt's strategy of using immutable infrastructure with Docker and EC2 to enable continuous delivery and minimize risk when deploying new software versions. Some key points made include: - Gilt builds Docker containers for each new application version, creates a new "stack" of infrastructure to run the container, and uses incremental rollout and automated rollback to reduce risk. - Immutable infrastructure emerges naturally with Docker since each version requires new containers and infrastructure rather than updating existing instances. - Automating deployment, rollback, and incremental rollout across new infrastructure stacks reduces probability, cost and occurrences of failures when deploying new versions. - Instant rollback is possible by moving traffic back to the previous version's infrastructure if
![Docker and Play Framework
$ sbt stage
$ more api/Dockerfile
FROM giltarchitecture/
ubuntu-openjdk-7-jre-headless:12.0.4
ADD . /apidoc
ENTRYPOINT ["/apidoc/bin/apidoc-api"]](https://image.slidesharecdn.com/michaelbryzek-gilt-140613152422-phpapp02-140801194911-phpapp01/85/Immutable-Infrastructure-with-Docker-and-EC2-19-320.jpg)
Immutable Infrastructure with Docker and EC2
- 1. Immutable Infrastructure With Docker and EC2 Docker Conf 2014 Michael Bryzek CTO & Co-Founder Gilt [email protected] / @mbryzek http://tech.gilt.com / @gilttech
- 2. What is Gilt? Founded in 2007 World’s best brands and products at 50-70% off New products launch at noon EST Limited inventory – products constantly sell out Over 1000 employees
- 3. Gilt Tech • ~150 people • Strategy to attract great people and enable them to innovate • Lots of Small Teams • Micro services architecture • 300+ services • ~1000 git repos • Busy days see > 100 production releases • > 10k requests / second
- 4. Immutable Infrastructure Why it Matters We believe innovation fuels growth. Part of our strategy to accelerate innovation Is to create truly autonomous teams Supported by tooling and automated processes to relentlessly decrease risk of change
- 5. Lots of Small Applications (LOSA) • Technology Strategy focused on: • Autonomy • Decentralization • Parallelism • Isolation
- 6. Teams and LOSA • Lots of Small Teams • 4-10 people / team • Have all “ingredients” to succeed • Deliver across stack for most projects
- 9. Move Fast with Minimal Risk What that Actually Means
- 10. Defining Risk Probability (event) * Cost(event) * Number of occurrences There is a risk to doing nothing
- 11. Reducing Probability(event) • Testing • Manual or Automated • Prefer automated for long term • Not making changes • Peer review • Kaizen • Immutability • Ownership / Pride • Experience
- 12. Reducing Cost(event) • Small change sets • Verification in target environment • Incremental rollout • Automated rollout / rollback
- 13. Reducing NumberOccurrences(event) • Instant Rollback • Great Monitoring and Alerting
- 14. Modern Software Deployment 1. Foundation of continuous delivery 2. Each deploy immutable 3. Incremental rollout 4. Metrics and alerting
- 15. Continuous Delivery @ Gilt Pre Docker sbt release-remote 1. Build an RPM in Jenkins 2. Deploy RPM to test environment 3. Run unit and integration tests 4. Deploy to one node in production 5. Run healthcheck, auto rollback if necessary 6. Repeat 4-6 on remaining nodes
- 16. Continuous Delivery @ Gilt w/ Docker ionblaster new api 1.2.3 ionblaster traffic api 1.2.2 90 1.2.3 10 1. Build docker container 2. Create new “stack” of infrastructure 3. Run container on each node in stack 4. Assign DNS to new stack 5. Manage traffic from old to new
- 17. ionblaster new api 0.4.2
- 18. Immutable Infrastructure / Docker Huge win w/ docker Dependencies in Dockerfile Focus instead on cloud and new stacks
- 19. Docker and Play Framework $ sbt stage $ more api/Dockerfile FROM giltarchitecture/ ubuntu-openjdk-7-jre-headless:12.0.4 ADD . /apidoc ENTRYPOINT ["/apidoc/bin/apidoc-api"]
- 20. Sample command to start play container image -run “ --expose 80 -p 9000:80 giltarchitecture/apidoc-api-1-2-3 –Dhttp.port=90 -Dconfig.resource=xxx.conf ”
- 21. Immutability w/ Docker Immutability emerges naturally when using Docker Upgrade Java? New version, new infrastructure, new containers. Security patch? New version, new infrastructure, new containers. Eliminate surprise for application owners.
- 22. Automate Incremental Rollout Core area of focus now ionroller api 1.2.3 1.2.4 “24 hours” Measure response time and status codes - triggers based on tolerance between versions
- 23. Instant Rollback If prior version around – just move traffic ionblaster traffic api 1.2.3 100 If not, same as before - deploy version But then can revise garbage collection policy for the app to decrease risk of a future event. (Kaizen)
- 24. Amazing Metrics and Alerting Reporting and alerting is hard Used nagios, graphite, open TSDB w/ limited success. We are now building a REST API for alerting on top of influxdb (open source time series db). Plan to open source if successful.
- 25. Lessons Learned: Incremental Rollout Minimize number of versions in production at any one time – e.g. “at most 2” Garbage collection important, but keep prior versions around for long enough (1 day? 1 week?) Different apps have different requirements on rollout time – back to calculation of Risk and the Cost(event)
- 26. Lessons Learned - PAAS You must have platform as a service; impossible to build well if not your core business. It’s tempting to build out a PAAS; but the number of tools needed to make this work reliably at scale is large.
- 27. Lessons Learned: Alerting Core interface: Send me at most one alert every n hours Core challenge always: • Signal to noise ratio critical and first class • Human tendency to ignore over time
- 28. Immutable Infra w/ Docker and EC2 • Decrease Probability(Event) • Immutability • Decrease Cost(Event) • Verification in target env w/ no user traffic • Incremental Rollout • Automated rollout/rollback • Reduce NumberOccurrences(event) • Instant Rollback
- 29. Thank You Michael Bryzek CTO & Co-Founder Gilt [email protected] / @mbryzek http://tech.gilt.com / @gilttech