Kubernetes and lastminute.com: our course towards better scalability and processes (Codemotion Milan 2016)
3 likes1,949 views
Lastminute.com Group is transforming its infrastructure for better scalability and efficiency by migrating from a monolithic system to microservices, utilizing Docker and Kubernetes. This shift aims to enhance processes while maintaining business continuity, achieving significant operational improvements like faster deployment and self-healing systems. The initiative has successfully migrated 25 microservices in four months and can handle 1300 requests per second in the new setup.
Kubernetes and lastminute.com: our course towards better scalability and processes (Codemotion Milan 2016)
- 1. Kubernetes and lastminute.com group: our course towards better scalability and processes [email protected] Milan, 25-26 November 2016
- 2. The inspiring travel company
- 3. lastminute.com group in numbers 40 countries 17 languages 10M travellers per year* € 2.5B GTV* € 250M revenue* 43M users per month* *data as 31st December 2015 icons from http://www.flaticon.com
- 4. A tech company to the core Tech department: 300+ people Modules: ~100 Database: 150 schemas, 3300 tables, TB data Instances: 1400+ Locations: Chiasso, Milan, Madrid, London, Bengaluru
- 6. lastminute.com group: an agile company ● Scrum and Kanban ● TDD ● clean code ● continuous integration ● code review ● internal communities
- 7. Starting from the monolith ... https://www.flickr.com/photos/southtopia/5702790189
- 8. https://www.pexels.com/photo/gray-pebbles-with-green-grass-51168/ ... broken into microservices
- 9. The improvements needed ● alignment ● real pipelines ● infrastructure ● resilience ● monitoring ● remove constraints
- 10. An year-long endeavour ● build a new, modern infrastructure ● migrate the search (flight/hotel) product there ... without: ● impacting the business ● throwing away our whole datacenter
- 11. TODO list ● company framework ● docker ● kubernetes
- 12. How? Teams and peopleNew teams https://www.pexels.com/photo/blue-lego-toy-beside-orange-and-white-lego-toy-standing-during-daytime-105822/
- 13. Our infrastructure and technologyOur infrastructure and technology https://www.pexels.com/photo/colorful-toothed-wheels-171198/
- 14. ● build once, run everywhere ● externalised configuration Docker containers
- 15. Docker containers registry.intra/application:v2-090025112016 BASE OS JAVA SDK START/STOP SCRIPTS JAR APPLICATION ● build once, run everywhere ● externalised configuration
- 16. Kubernetes ● independent from OS/hosts ● isolated env, managed at scale ● self-healing ● externalised configuration Omega paper: http://research.google.com/pubs/pub41684.html
- 19. Kubernetes: logical representation NAMESPACE1 CPU 10 MEM 40GB NAMESPACE2 CPU 20 MEM 50GB NAMESPACE3 CPU 80 MEM 60GB NAMESPACE4 CPU 5 MEM 5GB cluster
- 21. Kubernetes: our architecture and choices APP1-PRODUCTION deployment replica-set POD 3 POD 2 POD 1 production
- 22. Kubernetes: our architecture and choices APP1-PRODUCTION deployment replica-set secret configmap POD 3 POD 2 POD 1 production
- 23. Kubernetes: our architecture and choices APP1-PRODUCTION deployment replica-set app1.lastminute.intra secret configmap POD 3 POD 2 POD 1 loadbalancer-app1 production
- 24. APP1-PRODUCTION Kubernetes: our architecture and choices POD collectd production application fluentd
- 25. Kubernetes: what’s left outside? ● datastores ● distributed caches ● distributed locking ● pub-sub ● logs and metrics storage
- 26. 1st try (with test app), it seemed to work https://www.flickr.com/photos/26516072@N00/2194001232
- 27. The self-healing term describes any application, service, or a system that can discover that it is not working correctly and, without any human intervention, make the necessary changes to restore itself to the normal or designed state. Self-healing ref: https://technologyconversations.com/2016/01/26/self-healing-systems
- 28. Kubernetes agnostic interfaces “When a container is dead I will restart it” “When a container is ready I will forward traffic to it”
- 29. Kubernetes probes: liveness & readiness Two questions for dev: ● when can I consider my container alive? ● when can I consider my container ready to receive traffic? spec: containers: livenessProbe: httpGet: path: /liveness successThreshold: 3 failureThreshold: 2 readinessProbe: httpGet: path: /readiness successThreshold: 3 failureThreshold: 2 deployment.yaml
- 30. /liveness: ● when tomcat container is up ● when ratio “active/max” threads are lower than a threshold /readiness: ● all the startup jobs have run ● no termination request has been received .. ongoing never-ending research .. Our choices: framework - k8s
- 31. ● zero downtime during rollout ● monitoring in place ● alerting ● centralized logging ● legacy infrastructure to the rescue in case of problem 2nd try (with production traffic)
- 32. ... failure ... the big one! https://www.flickr.com/photos/ghost_of_kuji/2763674926
- 33. Problems ● configuration ● infrastructure ● tools ● manual mistakes ● (external) scalability
- 34. ● temporary team focus on objective ● automation ● monitoring ● Go deeper in docker/kubernetes Another improvement step
- 35. Pipeline: a huge step forward microservice = factory.newDeployRequest() .withArtifact(“com.lastminute.application1”,2) lmn_deployCanaryStrategy(microservice,”qa”) lmn_deployStableStrategy(microservice,”preview”) lmn_deployCanaryStrategy(microservice,”production”) pipeline
- 36. APP1-PRODUCTION Monitoring: grafana/graphite/nagios cluster graphite n collectd Grafana nagios icons from http://www.flaticon.com
- 37. “Go” deep .. whatever language it takes https://www.pexels.com/photo/sea-man-person-ocean-2859/
- 38. There’s light ..There’s a light .. at the end https://www.pexels.com/photo/grayscale-photography-of-person-at-the-end-of-tunnel-211816/
- 39. ● lead and migration time ● resilience ● root cause analysis ● speed of deployment ● instant scaling ... benefits
- 40. ● 1300 req/sec in the new cluster ● 25 micro-services migrated in 4 months ● 1 week to migrate an application ● 10 minutes to create a new environment ● 11 min to gracefully roll-out a new version with 55 instances ● whole pipeline runs in 16 min ● 1.5M metrics/minute flows Give me the numbers!