Safe deployments with Blue-Green and Spinnaker
- 1. Safe deployments with Blue-Green and Spinnaker Platform Bucharest 23.02.2017
- 3. Deployment • getting product out to clients • agility matters - first to market • speed vs risk
- 4. Rolling deployment • classic/easiest way of doing things • on every server: git pull; restart • you are probably doing some variation of this
- 6. Rolling deployment (cont.) • Advantages: • simple • no extra hardware • can do partial deploys
- 7. Rolling deployment (cont.) • Disadvantages: • might drop requests • slow rollback • finding the "last good state" • testing in staging only
- 8. Blue-Green deployment • two identical production environments • only one serves requests • deploy to the new one; test; route traffic
- 10. Blue-Green deployment (cont.) • Advantages: • testing in production • no downtime • quick rollback • predictable release time
- 11. Blue-Green deployment (cont.) • Disadvantages • cost - 2x servers • databases require extra work • development time to implement
- 12. Fixing the database problem
- 13. Optimizations • connection draining • canary • sticky sessions • automated tests on green • reproducible deploys using AMIs/Docker images
- 14. Real world • using it at Hootsuite since summer 2015 • 1K deploys in prod with no issues* • has not affected our velocity • deploy to prod is a mundane task * trivial rollback in case we had any
- 15. Real world (cont.) • Tech stack: • Docker • Asgard • Jenkins • ELB • Hubot
- 17. Our deployment pipeline (cont.) • cluster check • conflict check • merge tests • live tests • git merge • manual canary release • manual blue-green switch
- 18. Spinnaker • successor of Asgard (thanks, Netflix) • custom build and deploy pipelines • multiple cloud providers supported
- 19. Why Spinnaker? • easy trigger integration with Jenkins • handles all the "cloud" stuff • server groups • load balancers • security groups • useful templates built-in
- 21. Spinnaker stack • Jenkins • Packer • .deb repo • Spinnaker core
- 22. Deploying .debs • not everybody used to this • lifesaver: fpm • running your stuff - upstart