WSA: Scaling Web Service to Handle Millions of Requests per Second
0 likes100 views
The presentation by Venkata Ramana Gollamudi covers strategies for scaling web services to handle massive traffic, particularly focusing on scenarios from Flipkart during significant sales events. It outlines two case studies: scaling a simple lookup web service and an order placement service, addressing the challenges and solutions involving Kubernetes and NoSQL databases. Key insights include the importance of containerization, independent service scaling, and the use of event-driven architecture for efficient processing.
WSA: Scaling Web Service to Handle Millions of Requests per Second
- 1. www.webstackacademy.com Scaling Web Service to Handle Millions of Requests per Second Presentation – 23rd March 2024 Venkata Ramana Gollamudi (Senior Principal Architect @ Flipkart)
- 2. www.webstackacademy.com About Me With 20+ Years of Software development Experience. Senior Principal Architect @ Flipkart solutioning Data Platform and Data Engineering. This platform support data and real time analytics on 100s of PB of data. Open Source Apache Committer. Key expertise in designing scalable distributed processing and analytical systems. New Interest is on NLP based data analytics on Big Data
- 3. www.webstackacademy.com Agenda 1. Scale for India, expectations (impact) 2. Case1: Journey of scaling a simple lookup web service a. Challenges b. Using K8s c. Using NoSQL stores 3. Case2: Journey of scaling a simple order placement service a. Challenges b. Queues and Event driven Architecture 4. Typical network Scalability model
- 4. www.webstackacademy.com Scale for India 1. 1.4 billion visits during Big Billion Day 2023 Sales 2. 4 million packages delivered in first 4 days of Sale.
- 5. www.webstackacademy.com www.webstackacademy.com Case 1: Journey of scaling a simple lookup web service
- 6. www.webstackacademy.com Problem statement Webservice GetProductDetail MySQL 1 VM 10 Requests/sec 10 Requests/sec
- 7. www.webstackacademy.com Try Scaling - Challenges Load Balancer Webservice GetProductDetail MySQL 10 VM 100 Requests/sec 100 Requests/sec Challenges - Request routing across VMS - Each VM load protection - Images and common data needs catching - VM is down - Load protection on MySQL - MySQL Instance is down vm1 vm2 vm10 … Multi Master, Master slave replica etc MySQL Router / Load Balancer
- 8. www.webstackacademy.com Scale further - Challenges Load Balancer Webservice GetProductDetail MySQL 1000 VMs 10000 Requests/sec 10000 Requests/sec Challenges - Request routing across VMs - Each VM load protection - Images and common data needs catching - VM is down - Load protection on MySQL - MySQL Instance is down - Cannot scale horizontally vm1 vm2 vm1000 … 100s of Nodes Multi Master, Master slave replica etc MySQL Router / Load Balancer
- 9. www.webstackacademy.com Scaling Web service using K8s Replaced with What it brings in ● Containerization ● Scalability ● High Availability ● Resource Efficiency ● Self-Healing ● Portability ● DevOps Enablement & Deployment Spec Webservice vm1 vm2 vm1000 …
- 10. www.webstackacademy.com Scaling Web service using K8s (contd..) Load Balancer Container1 (Bench marked) Container2 Container 1000 … ● Liveness Probes ● Readiness Probe ● Startup Probe Readiness Probes Control Plane (Autoscaler) Startup probe Liveness probe
- 11. www.webstackacademy.com Scaling Database using NoSQL Database Replaced with What it brings in ● Key-Value Store ● High Scalability ● High Availability ● Efficiency ● Row level Consistency ● In-Memory Cache MySQL 100s of Nodes Multi Master, Master slave replica etc MySQL Router / Load Balancer RegionServer Region Region RegionServer Region Region … RegionServer Region Region 1000s of Nodes Web Service (Clients)
- 12. www.webstackacademy.com Scaling Database using NoSQL Database (contd) What is brings in ● Key Sharding ● Fast lookup ● Fast Update Key Range 00 - 15 16 - 30 31 - 45 46 - 60 61 - 75 75 - 100
- 13. www.webstackacademy.com Scaling Database using NoSQL Database (contd) Load Balancer Web service GetProductDetail 1000s of Containers Million Requests/sec Million Requests/sec Cont1 … CDN Cont1 Cont1k RegionServer Region Region RegionServer Region Region … RegionServer Region Region 1000s of Nodes
- 14. www.webstackacademy.com www.webstackacademy.com Case 2: Journey of scaling a simple order placement service
- 15. www.webstackacademy.com Problem statement Order Service PlaceOrder MySQL Track Order Deliver Order 1 VM 10 Requests/sec 10 Requests/sec Read/Write
- 16. www.webstackacademy.com Try scaling Order Service PlaceOrder Track Order Deliver Order 1000s Place Order Requests/sec 1000s Deliver Order Requests/sec Million Track Order Requests/Sec 1000s of Containers 1000s Write Requests/sec Million Read Requests/sec Read/Write NoSQL Load Balancer
- 17. www.webstackacademy.com Try scaling - Challenges Order Service PlaceOrder Track Order Deliver Order 1000s Place Order Requests/sec 1000s Deliver Order Requests/sec Million Track Order Requests/Sec 1000s of Containers 1000s Write Requests/sec Million Read Requests/sec Read/Write NoSQL Load Balancer Challenges ● Different Services requests have different resources requirement ● Adding new Service is tough ● Latency of Place Order is high ● Track Order and Place Order have different scale requirements ● Not all operations are synchronous
- 18. www.webstackacademy.com User Queues and Event based design Order Service PlaceOrder Track Order Deliver Order NoSQL Load Balancer Order Tracking Service NoSQL Fulfillment Service NoSQL Load Balancer Load Balancer Fraud Detection Service Graph Database Load Balancer What it brings in ● Independent scaling ● Low latency Async Processing ● High throughput Scalable Queues ● Resource Efficiency ● Independent tech stack ● Easy New service addition ● DevOps independence Advertisement Service NoSQL Database Load Balancer
- 21. www.webstackacademy.com Typical network CDN placement
- 23. www.webstackacademy.com MCQ Questions 1. Which Databases preferred to Scale to Million Read/Write Requests per sec a. Transactional Databases b. NoSQL databases c. File Based Stores d. None of the above Answer: b
- 24. www.webstackacademy.com MCQ Questions 2. K8s brings in a. Scalability b. Self Healing c. Containerization and portability d. All of the above Answer: d
- 25. www.webstackacademy.com MCQ Questions 3. Example of NoSQL databases a. MySQL b. HBase c. K8s d. Kafka Answer: b
- 26. www.webstackacademy.com MCQ Questions 4. HBase scalability is achieved using a. Master Slave model b. Sharding key space c. Multiple clusters d. Distributed transactions e. All of the above Answer: b
- 27. www.webstackacademy.com MCQ Questions 5. Load balancer will identify Active Containers to route traffic using a. Readiness Probe b. Round robin distribution c. Errors d. Latency of request Answer: a
- 28. www.webstackacademy.com MCQ Questions 6. Event based async architecture requires ______ for inter service communication a. K8s b. NoSql Databases c. Queues d. Load balancer Answer: c
- 29. www.webstackacademy.com MCQ Questions 7. Event based async architecture benefits a. Services can independently scale b. Addition of new service is easy c. Can choose different tech stack and stores based on need d. All of the above Answer: d
- 30. www.webstackacademy.com MCQ Questions 8. Popular implementation of Queues a. HBase b. Cassandra c. Apache kafka d. K8s Answer: c
- 31. www.webstackacademy.com MCQ Questions 9. CDN useful to store a. Images and Videos b. Order status information c. Delivery status information d. All of the above Answer: a
- 32. www.webstackacademy.com MCQ Questions 10. Typical path for mobile to web service traffic is a. Mobile -> POP endpoint -> Internet -> Data center b. Mobile -> Internet -> POP endpoint -> Data center c. Mobile -> Internet -> POP endpoint -> Internet -> Data center d. All of the above Answer: b