Using Postgres and Citus for Lightning Fast Analytics, also ft. Rollups | Live Demo | Sai Srirampur
0 likes263 views
The document outlines a live demo presented by Sai Srirampur on using Postgres and Citus for fast analytics and rollups. It emphasizes the challenges of scaling databases and outlines the benefits of using Citus to distribute data and improve performance for analytics workloads. The presentation also covers concepts like rollups and estimation techniques like HLL for efficient data processing.
Using Postgres and Citus for Lightning Fast Analytics, also ft. Rollups | Live Demo | Sai Srirampur
- 1. Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019 WITH FROM Thu 27 February at 10am PT Using Postgres & Citus for Lightning Fast Analytics PRESENTED BY: SAI SRIRAMPUR LIVE DEMO
- 2. Sai Srirampur | Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019 Ask private questions in Q&A panel (mouseover at bottom to see Q&A icon) We plan to do 1-2 quick polls. Logistics X
- 3. Sai Srirampur | PyConCA 2018 • Sai Srirampur a.k.a Sai • Engineer at Citus Data (now part of the Microsoft family!!) • Joined Citus to make it so developers never have to worry about scaling their database • Recently got married! • Follow me @saisrirampur @citusdata
- 4. Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019 Does this sound like you? Hitting resource limits with single-node Postgres, but don’t want to give up Postgres 1
- 5. Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019 Want to reduce dev costs by consolidating—& avoid separate stores for analytics (OLAP) & transactional workloads (OLTP) Does this sound like you? 2
- 6. Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019 Does this sound like you? Pre-aggregation or rollup pipeline lags and never catches up 3
- 7. Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019
- 8. Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019 Why ! Postgres? TLDR; Open source Constraints Extensions PostGIS / Geospatial HLL, TopN, Citus Foreign data wrappers Rich SQL CTEs Window functions Full text search Datatypes JSONB
- 9. Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019 Q: Why Citus? A: Citus transforms Postgres into a distributed database
- 10. Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019 TLDR; on scaling out Postgres w/Citus 1. Distributes data across multiple nodes 2. More memory, cpu, disk + parallelization 3. Extension to Postgres (not a fork) DATABASE AS A SERVICE ENTERPRISE SOFTWARE OPEN SOURCE
- 11. Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 201911
- 12. Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019 Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019
- 13. Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019 What is a rollup? Period Customer Country Site Hit Count SELECT… Pre-computed aggregates for a period and set of (group by) dimensions. Can be further filtered and aggregated to generate charts.
- 14. Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019 What are the benefits of rollups? • Fast (indexed) lookups of aggregates • Avoid expensive repeated computation • Rollups can be further aggregated • Rollups are compact, can be kept over longer periods
- 15. Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019 What is hll? • Approximation algorithm (“sketch” algorithm) • Estimates COUNT(DISTINCT)/cardinality of given data • Extension on postgres: hll • hll data type to store distinct values
- 16. Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019 Already provisioned a 4-node Citus database cluster for today’s demo / & regular Postgres on single node
- 17. Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019 Load data Create indexes (btree and GIN) Query patterns in analytics workload Rollup queries What I will show you in today’s analytics demo? Citus 4- node cluster Single node Postgres TOP BOTTOM Load data Create indexes (btree and GIN) Query patterns in analytics workload Rollup queries
- 18. Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019 Sai Srirampur | PyConCA 2018
- 19. Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019 1/ Parallelism 2/ Linear scale 3/ Freshness (real-time, concurrency) 4/ Both transactional && analytics
- 20. Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019 1/ Faster rollups - parallelism - no lag 2/ Increase granularity of period 3/ Store and query longer periods of data 4/ Extensions - hll, topn, pg_cron, pg_partman Enhance your rollup pipeline with
- 21. Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019 Sai Srirampur | PyConCA 2018 citusdata.com/customers/heap citusdata.com/customers/heap
- 22. Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 201922 citusdata.com/customers/algolia
- 23. Scaling Postgres for Time Series Data with Citus | Nov 15 2018 | Marco Slot | Claire Giordano Using Postgres and Citus for Lightning Fast Analytics | February 2019 Min Wei of Microsoft
- 24. Scaling Postgres for Time Series Data with Citus | Nov 15 2018 | Marco Slot | Claire Giordano Using Postgres and Citus for Lightning Fast Analytics | February 2019
- 25. Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 2019
- 26. Live Demo of Using Postgres and Citus for Lightning Fast Analytics, also featuring Rollups | March 201926 Any Questions?? (And, resources) DOWNLOAD CITUS OPEN SOURCE: https://www.citusdata.com/download CREATE CITUS CLOUD DEV ACCOUNT: https://www.citusdata.com/product/cloud JOIN OUR PUBLIC SLACK: https://slack.citusdata.com/
- 27. Live Demo of Citus Cloud | July 2018 www.citusdata.com @citusdata © 2019 Citus Data. All rights reserved. Sai Srirampur | @saisrirampur Thank you for your time