Zero Downtime App Deployment using Hadoop
Download as PPTX, PDF1 like828 views
Hadoop can enable zero downtime app deployments by using microservices, continuous delivery, and real-time analytics. The presenters describe how Expedia saves $5M annually through zero downtime deployments. Their architecture uses microservices, continuous integration, deployment monitoring with Storm/Kafka/HDFS, and analytics in Solr/Hive to enable canary testing, fast feedback, and automated problem resolution. A live demo shows log processing, analytics, and using results to ensure smooth, high-quality deployments.
Zero Downtime App Deployment using Hadoop
- 1. ZERO DOWNTIME APP DEPLOYMENT USING HADOOP Hadoop Summit 2016 – San Jose Heman Duraiswamy Solutions Engineer Wei Wang Data Scientist Jun 30, 2016
- 2. Agenda Introduction Our Story that ends with “Happily ever after” “Zero downtime app deployment” Reference Architecture Demo
- 3. Introduction: Why? Expedia revenue in 2015 == $6.67 B >18.2MM/day (or) >$750,000/hr Cost of 15 min deployment window and having one deployment every other week?? ~5MM $$ Same figure for Amazon?? - $$80MM And offcourse, customer trust & confidence!
- 5. Zero downtime app deployment using Hadoop
- 6. Our Story: Once upon a time…
- 7. Our Story: Once upon a time…
- 8. Our Story: Once upon a time…
- 9. Our Story: Once upon a time…
- 10. Our Story: then they evolved Rolling deployment Near Zero downtime deployment Hampers Innovation Operational over-head
- 12. Innovate -- Micro services Header module Search module localDest module topDeal module Loyalty module
- 14. Learn & Succeed (or fail) Canary Deployment a/b testing Operational monitoring 24*7 Instant feedback
- 15. Reference Infrastructure layout Header module Search module localDest module topDeal module Loyalty module Server001 Server002 Server003 Server004 Server005 Server006 Server007 Server012 Server013 Server014 Server008 Server009 Server010 Server011 Use case 1: server in bad state – meaning serves higher proportion of 404 &503 pages Use case 2: server is slow – takes longer to process requests Use case 3: Deploy bad app version – serve high propotion of application CRIT and ERROR messages
- 16. Reference Architecture HDF (nifi) Kafka Storm Topology Application log files Access log files Server s Server s Solr Hive HDFS Banana view Ansible script
- 17. DEMO: Log prep
- 18. DEMO: HDF flow
- 19. DEMO: Storm Topology Storm Kafka Bolt Aggregation Bolt Timer Bolt Calculatio n Bolt Solr Bolt Hive Bolt
- 22. DEMO: Analytics Results in Banana
- 23. DEMO…