Scaling APIs: Predict, Prepare for, Overcome the Challenges
Download as PPTX, PDF14 likes9,456 views
This document discusses strategies for scaling APIs to handle increasing volumes of traffic. It outlines considerations and approaches for handling throughput from 1 transaction per second up to 100,000 transactions per second. Lower volumes can typically be handled through database and application tuning. Higher volumes require caching, database sharding or NoSQL databases, and scaling out application servers and other infrastructure. API and client design can also impact scaling by reducing unnecessary calls and payloads. Latency and network bandwidth also become important concerns at high volumes.
Scaling APIs: Predict, Prepare for, Overcome the Challenges
- 1. Scaling APIs Feeding your Speeds April 5, 2012 Greg Brail @gbrail Brian Pagano @brianpagano
- 2. @gbrail @brianpagano
- 5. New! IRC Channel #api-craft on freenode
- 6. What is scale to an API? Developers: Lots of developers building apps Apps: Lots of apps for end users to use End users: Millions and millions of app users Versions: Lots of API versions to manage API calls: All of these things result in API calls…
- 7. Today’s Topic: API Calls Today we are going to focus on handling huge numbers of API calls in an API infrastructure Ever been in a meeting where someone said, “let’s not talk about ‘speeds and feeds’ today?” This is not that meeting.
- 8. Why do APIs need to scale? Thousands of app developers… Each one must be managed, signed up, etc. Thousands of apps… Each one has credentials that need to be validated Millions of end users… Each one has one or more OAuth access tokens Result in lots of API calls. For example: One million end users, Making 1000 API calls a day, Results in one billion API calls a day, Or about 11,000 API calls per second on average
- 9. Tracking API calls Today we’ll mainly talk about throughput Measured in API calls or transactions per second (tps) For an API, usually as the number of users increases, throughput increases As throughput increases, latency often increases too It’s not enough just to handle lots of throughput – it’s important to handle it with a reasonable amount of latency
- 10. What Limits Scale? Disk Database Network App server CPU API Proxy Memory Load balancer Cache servers
- 11. What are some limits? Seek time Database design & tuning Rotational speed App server coding & config Transfer speed Proxy configuration Clock speed Load balancer policies Number of cores Cache configuration Amount of RAM And many more…
- 12. Some examples We’re going to talk about things to look at as throughput grows from one level of traffic to another… 1 tps 10 tps 100 tps 1000 tps 10,000 tps 100,000 tps and beyond…..
- 14. At 1 transaction per second 86,400 per day / 2.5 million per month Almost everything can handle this. What about the database? If each API call makes several big SQL queries, it may not!
- 15. Strategies for 1 tps Test (always) Tune the database installation Tune the database design Monitor query performance Test again!
- 17. At 10 transactions per second 864,000 per day / 25 million per month Most infrastructure can still handle this. What about the application server? Is the app well-designed enough? Does it make an excessive number of database calls?
- 18. Strategies for 10 tps Ensure that the app server is properly optimized Do API calls make the minimum number of database calls? Do API calls depend on large numbers of external services?
- 20. At 100 transactions per second 8.6 million per day / 259 million per month (Now we are starting to get somewhere) RDBMS systems may struggle Less-efficient app servers may struggle “Free” tiers on hosting platforms aren’t an option
- 21. Strategies for 100 tps Database optimization and tuning is critical here Allocate fast storage, and lots of it Allocate lots of memory Tune the database to use it! Find bad queries and fix them or optimize them App server tuning is critical here Are there enough threads in the thread pool? Are there enough processes?
- 23. At 1000 transactions per second 86 million per day / 2.5 billion per month Now everything may start to break… What is the mix between reads and writes?
- 24. Strategies for 1000 tps Understand the mix between reads and writes Cache the reads as much as you can If you can cache them closer to the client, better Understand your app server performance Faster app servers should still be able to handle (like Java) RoR, Python, PHP, etc will require much bigger clusters Stateless app servers are your friend!
- 25. More strategies for 1000 tps Can the database handle the load? It can if most transactions are reads And you cache as much as you can Otherwise it’s time to scale the database layer Sharded RDBMSes Or a scalable NoSQL database works here
- 26. At 10,000 transactions per second 864 million per day / 25 billion per month If most transactions are reads, caching is your friend Otherwise, this is serious business No single database can handle this Few single app servers can handle this If API calls are large, what will the bandwidth be?
- 27. Strategies for 10,000 tps Caching is even more essential Even a simple cache can handle this load on one or two boxes Database writes are problematic No single database server can write 10,000 times per second Scalable, eventually-consistent databases can scale this big, (like Cassandra)
- 28. More for 10,000 App servers You’ll need a cluster of app servers no matter what! What about session management? What about load balancing?
- 29. 100,000 API calls per second 8.6 billion per day! Now your API is truly impressive (either that or it is very poorly designed!) You will need racks of infrastructure no matter what!
- 30. Some other considerations API design Client design Latency Bandwidth
- 31. What about API design? Every API call has overhead: TCP connection / SSL handshake / load balancer CPU / API proxy CPU / App server CPU and thread pool / database connections / disk I/O… Do you need to make so many? Can you design your APIs to support fewer high-value API calls? Can you have “batch” calls in your API?
- 32. What about the client? Can client apps use the API more efficiently? Don’t make the same API calls over and over Utilize compression Utilize conditional requests in HTTP Which means that the API server should support them! Request only the data that’s needed Which means that the API server should trim responses Or paginate them
- 33. What about latency? Latency kills user experience! How can the API server reduce it? Remove steps in the processing flow through caching Cache closer to the API clients
- 34. What about the network? What kind of network connection to you have? For instance, in EC2 you get 1Gbps, Or about 122 megabytes / second At 10,000 tps, for instance, that’s 12K per API call
- 35. THANK YOU Questions and ideas to: @gbrail @brianpagano
Editor's Notes
- #2: Creative Commons Attribution-Share Alike 3.0 United States License