Real-time Analytics with Redis
Download as PPTX, PDF5 likes1,444 views
The document discusses Redis, an open-source in-memory database platform known for its high performance in operational, analytic, and hybrid use cases. It highlights Redis' versatility in handling real-time analytics, machine learning capabilities, and various data structures that support transactional processing and analytics. It also includes examples of companies using Redis for real-time analytics, such as Redfin and Home Depot, showcasing its effectiveness in high transaction environments.
Real-time Analytics with Redis
- 1. Real-time Analytics • CIHAN BIYIKOGLU – VP OF PRODUCT MANAGEMENT
- 2. Who We Are Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source home and commercial provider of Redis Enterprise (Redise) technology, platform, products & services. 2
- 3. Redise Pack Managed Fully managed Redise Pack in private datacenters Redise Pack Downloadable Redise software for any enterprise datacenter or cloud environment Redise Cloud Private Fully managed, server- less scaling Redise service in VPCs within AWS, MS Azure, GCP and IBM Softlayer Redise Cloud Fully managed, server- less Redise service on hosted resources within AWS, MS Azure, GCP, IBM Softlayer, Heroku, CF and OpenShift Redis Labs Products 3 or or or DBaaS Software
- 4. What Makes Redis So Popular? Simplicity VersatilityPerformance ListsSorted Sets Hashes Hyperloglog Geospatial Indexes Bitmaps SetsStrings Bit field Sub-millisecond Latency at High Throughput 1 Redis Data Structures 2 3 Redis Modules for Indexing, Query and more… 4
- 5. Redis is Versatile 5 Covering transactional, operational and real-time analytics use cases, such as: Authorization Authentication Price Management Advertising Bids Messaging Location-based Processing User Session Management Counting Leaderboards Page Ranking Recommendation Engine Time-series Analysis Session Analysis Secondary Index Accelerated Reporting Real-time Attribution Search Order History Inventory Tracking TRANSACTIONAL ANALYTICS OPERATIONAL
- 6. Redis is Versatile 6 Covering transactional, operational and real-time analytics use cases, such as: Authorization Authentication Price Management Advertising Bids Messaging Location-based Processing User Session Management Counting Leaderboards Page Ranking Recommendation Engine Time-series Analysis Session Analysis Secondary Index Accelerated Reporting Real-time Attribution Search Order History Inventory Tracking TRANSACTIONAL ANALYTICS OPERATIONAL
- 7. Why Redis for Analytical Processing? 7 Analytics Data Structures • Probabilistic Data Types • Indexed Types • Geo Searches Real-time Machine Learning • Native ML Commands for Serving Recommendations and Predictions Data Processing on RAM + Flash • Data processing over RAM + Flash for fast but cost efficient storage
- 8. Redis Data Types for Analytics Processing
- 9. Data Structures Enable Embedded Analytics Hyperlog-logs Probabilistic estimates of counts for anomaly detection Sorted Sets Real time range analyses, top scorers, bid ranges Sets Cardinality for fraud detection Geospatial Indexes Location based searches INLINE ANALYTICS WITH DATA STRUCTURES Bitmaps Real-time population counting for activity monitoring
- 10. Modules Expand Analytic Capabilities Rebloom Topk Countminsketch T-digest INLINE ANALYTICS WITH MODULES Probabilistic membership queries Rank-based statistics estimator Track the top-k most frequent elements in a stream Approximate frequency counter Redis-cell Rate-limiting
- 11. Redis Enterprise + Modules Deliver Extended Capabilities Extremely fast text based search, used for secondary indexing Graph query processing Range analyses, built in aggregations (min, max, sum,avg) OPERATIONAL ANALYTICS WITH MODULES RediSearch Redis Graph Time-series
- 12. Real-time Machine Learning with Redis
- 13. Machine Learning/Deep Learning Modules NEURAL REDIS REDIS-ML • Simple neural network as a native data type for Redis • Training and classification/regression in one place with simple, intuitive APIs • Very fast and accurate for realtime ML use cases • Models can be stored, retrieved and updated natively with Redis-ML • Accelerates complex ML models by x100 • Unified and simplified ML serving operation • Currently available with Spark ML • Other ML/DL platforms coming soon
- 14. The Machine / Deep Learning (ML/DL) World Creating a Model Serving the ModelTraining1 2 3 14
- 15. Serving SolutionsTraining and Creation Solutions Homegrown The Machine / Deep Learning (ML/DL) World Creating a Model Serving the ModelTraining1 2 3 15
- 16. Serving SolutionTraining and Creation Solutions How ML/DL Should be Served Creating a Model Serving the ModelTraining1 2 3 16
- 17. 17 Scale Machine Learning with Redis -ML Saves 97% on Costs ‐ Requirements: ‐ Serve 20,000 ads/sec @ 50msec data-center latency ‐ Runs 1k campaigns 1K random forest ‐ Each forest has 15K trees with an average depth of 7 levels ‐ Needs 1 trillion ops/second • Redis –ML serves models 2000 times faster • Cuts computing infrastructure needed by 97% x2,000 FASTER COST-EFFECTIVE msec msec Custom App Redise # of AWS instances 1,247xc8.xlarge 35xc8.xlarge Reserved instances costs $11,448,611 $322,455 Savings 97% savings
- 18. Data Processing on RAM + Flash Redise Flash
- 19. Price/Performance of Memory Technology Cost per GB vs. Read/Write Performance 1.00 90.00 250.00 0.10 25.00 500.00 0.10 1.00 10.00 100.00 1,000.00 $0 $1 $2 $3 $4 $5 $6 $7 $8 $9 $10 DRAM NV-DIMM/PM NVMe SSD SATA SSD 1 GB cost Read latency Write latency $9/GB $0.4/GB$1/GB $2/GB microseconds 19
- 20. User customizes the RAM-to-Flash ratio How to Achieve Optimal Price/Performance 20 Redis Enterprise performs cluster-wide orchestration of RAM- Flash distribution
- 21. Redise Flash: 10TB Redis Deployment on EC2 * Redis enterprise only needs 1 copy of the data because quorum issues are solved at the node level Redis on Ram Redise Flash Dataset size 10 TB 10 TB Database size w/ replication 30 TB 20 TB* AWS instance type X1.32xlarge I3.16xlarge Actual instance size (RAM and RAM+Flash) 1.46 TB 3.66 TB # of instances needed 21 6 Persistent Storage (EBS) 154 TB 110 TB 1 year cost (reserved instances) $1,595,643 $298,896 Annual Savings -- 81.27% 21
- 22. Who is Using Redis for Real-time Analytics? 22
- 23. RedFin: Redis for Real-time Analytics and More • Real-time analytics used for: – Error aggregation and analysis (health dashboards are powered by Redis) – Personalized experience to users based on location, preferences, etc. – Estimation of active users using hyperloglog – Breakdown of users by city/neighborhood
- 24. Home Depot • Redis powers Home Depot’s omni-channel order management system which is designed to process 30,000 transactions per second • Sorted Sets and Geo data structures used for inventory sourcing and ATP (Available to Promise) calculation in real-time • Hyperloglog used for counting unique customers, sales per item, and sales per category https://redislabs.com/webinars/implementation-patterns- leverage-redis-turbo-charge-existing-legacy-applications/
- 25. Redislabs.Com Download Redis Enterprise today! Signup for Redis Enterprise Cloud today! Get Started Now!
- 26. Real-time Analytics • CIHAN BIYIKOGLU – VP OF PRODUCT MANAGEMENT
Editor's Notes
- #18: Calculation for ops/second requirement : 20K (ads/sec) x 1K (forests) x 15K(trees) x 7 x 0.5 (levels)
- #20: DRAM prices have been relatively stable over the years – and it continues to be expensive. Technologies such as Flash offer performance that is 3-4 orders of magnitude slower but 10 times cheaper. Emerging technologies such as Flash offer performance that is only an order of magnitude slower at 3 times lower cost. This makes for quite an attractive cost-performance tradeoff!