whyPostgres, a presentation on the project choice for a storage system
- 1. First, We need a problem to Solve.
- 2. This is US
- 3. And we had the chance to be here and to impalement OTNEngine
- 4. And to display the analytics and data has some challenges Need to find a way to store and retrieve the data - Would be able to run for long time (as long as the session) - Can write real time traffic - Can read huge traffic for analytics and - Has a catchy name
- 5. Our user want to monitor traffic and see the progress
- 6. So how to monitor data and see the history of records
- 7. Focal Points for frames Monitoring • Number of frames generated • What are the headers most important value • Did it fit the allocated space • How long did the graph stall • Did the user get board? • How do these numbers compare to what is expected • How do these number up to the user expectation
- 8. This creates a lot of data • Not all data are important for each user • Sampling frames generated in one hour (approx): 360K frame • Samples from frames are trimmed to the header only so the size reduces by 63 • Those headers will be aggregated with some meta data like page number • After aggreges we are left with 360k header per hour • Each row has 256 measures of viewability metrics
- 9. Real-time accumulation and aggregation Consume header of frames as fast as possible Serving API calls continuously Single machine with - 6 physical cores - Low latency - Overprovisioned
- 10. What didn’t work: MySQL • Original DB choice • Performed very well, when daily volume was < 1% of current volume • Impossible to add new columns to database
- 11. What didn’t work LFS • Very fast compared to any db. • Not Optimized for Random Access • Needed local disk with very fast I/O
- 12. What did work: Postgres • Fault tolerance against interrupts • Can have a dynamic schema (json data-type) • Support multiple connection
- 13. Main Flow
- 14. DB fast? • No loops needed (used array to insert records) • Connection pool (for parallel writes to db) • Non-blocking code (used async + threads) • Prepared statement (fancy words) • In an 2.9 GHz 6-Core Intel Core i9 • Write 10k frames in 40 ms • Read 10k frames in 50 ms
- 16. What didn’t work: Compressing Payload • Our Data is pRPS • Reduce in both read and write • From 40 ms to 250 to the same size • With this rate it is estimated to have 200GB in 24