What's new with enterprise Redis - Leena Joshi, Redis Labs
Download as PPTX, PDF2 likes1,773 views
The document outlines the features and benefits of Enterprise Redis, highlighting its capabilities for infinite scaling, high availability, and expert support. It elaborates on implementations like Redis on Flash, enabling enhanced performance and cost savings for data management, and discusses the integration with analytics tools like Spark. Additionally, it introduces the Redis Module Hub, which expands Redis functionality through various modules for diverse use cases.
What's new with enterprise Redis - Leena Joshi, Redis Labs
- 1. Home of RedisWhat’s New with Enterprise Redis Leena Joshi
- 2. 2 Agenda • What is Enterprise Redis? • Redis on Flash • Redis and Spark • Redis Module Hub
- 4. 4 But What It Really Means Is:
- 5. 5 Which In Turn Means: Infinite Seamless Scaling True High Availability Top notch expert support
- 6. 6 Redis Labs Enhances OSS Redis Redis Labs Node Open Source Zero latency proxy Cluster Manager REST API Odd number of nodes needed to handle network splits- not three copies of data Redis Labs Cluster • Shared nothing cluster architecture • Fully compatible with open source commands & data structures Proprietary
- 7. 7 The Same Technology Runs Redis Cloud Cluster Management Path Proxies Node Watchdog Cluster Watchdog Node 1 Node 2 Node N (uneven number)… Redis Shards Unique multi-tenant “Docker” like architecture enables running hundreds of databases over a single, average cloud instance without performance degradation and with maximum security provisions Data Path Distributed Proxies Single or Multiple Endpoints 50,000+ Customers
- 8. 8 Tremendous Customer Traction Redis Cloud Available since mid-2013 6000+ enterprise customers Redis Labs Enterprise Cluster (RLEC) Available since early-2015 100+ enterprise customers
- 9. 9 Always On - Highly Available & Persistent Simple, Seamless Clustering. Linear Scalability. Enterprise-Class Management and Support Enterprise-Class Redis – The Benefits Stable & Predictable Top Performance Operational Cost Savings
- 10. 10 Simple, Seamless Scaling and Clustering Auto- scaling/re-sharding/re-balancing No downtime while scaling Supports cross-shard operations Simple, Seamless Clustering. Linear Scalability. Linear Scalability
- 11. 11 Always On - Highly Available & Persistent Seamless cross datacenter/region/cloud replication Instant auto-failover Persistence, backups and DR Always On - Highly-Available & Persistent
- 12. 12 Stable & Predictable Top Performance Consistent high performance achieved under any load or cluster size Database processed by multiple cores Built-in performance enhancement techniques Stable & Predictable Top Performance
- 13. 13 Operational Cost Savings OSS Redis Redis Labs More efficient hardware utilization: fewer servers, lower power & cooling and operational costs Reduced manual labor through automation - reduced time writing scripts, scaling, configuration, monitoring, re-balancing and more Run Redis on flash memory as RAM extender – up to 10 times cheaper Reduced downtime incidents Shorten time to deploy Redis by over 50%
- 14. 14 Enterprise Management and Support UI, CLI, REST API -based management & alerting Proven technology supporting thousands of customers 24x7 enterprise support, top notch Redis expertise Enterprise-Class Management & Support
- 15. 15 Redis Labs: Fastest Recovery, No Data Loss %oftimesdatawaslost Averagetimetorecoverinseconds Redis Labs recovers in 5 seconds and does not lose data. All other vendors lose data and take many minutes to recover Vendors evaluated include (not in order) • Heroku Redis • AWS ElastiCache • Microsoft AzureCache • Compose.io
- 16. 16 Redis Labs: The Only True HA Redis 16 Failure Event In-memory Replication Multi-DC/Zone replication Auto-failover AOF Data Persistence Backup (using snapshots) Multi- region/Cloud replication Process failure Instant recovery* Slow recovery Node failure Instant recovery* Slow recovery Multi-node failure Instant recovery* Network split Instant recovery* Zone/Rack failure Instant recovery* Slow recovery Fast recovery Region/Cloud failure Slow recovery Fast recovery Typeofoutage Essential features for high availability *Auto-failover should run on same nodes as Redis deployment Redis Labs provides all the essential HA features that protect against every type of outage
- 17. Redis on Flash
- 18. 18 Why Analyze Data In-Memory? “Information is the oil of 21st century, and analytics is the combustion engine”- Peter Sondergaard, Gartner Analyst
- 19. 19 Decision Speeds Are Accelerating.. “Big Data” gains popularity as tools become available to harness it Batch insights start to drive business Real time insights automate decision- making 2005 2012 - 2015 2016… THE DATA REVOLUTION IS MATURING..
- 20. 20 The Race Is On.. INSIGHTS FROM YOUR DATA NEED TO BE INSTANTANEOUS COST EFFECTIVE
- 21. 21 Price/Performance of Memory Technology
- 22. 22 Redis on Flash Flash used as a RAM extender and NOT as persistent storage
- 23. 23 Redis on Flash Concepts • Flash used as a RAM extender (NOT as a persistent storage) • Global key list in RAM; ‘hot’ values in RAM; ‘cold’ values on Flash. • Multi-threaded & async Redis when accessing objects on Flash. Utilizes multi-core and Flash concurrency architecture • 100% compatibility with Redis
- 24. 24 How to Achieve Optimal Price/Performance By dynamically setting RAM/Flash ratio
- 25. 25 Single Server Results with Dell & Samsung NVMe read write read write Avg: 2.04M ops/sec Max: 2.14M ops/sec Avg: 0.91msec Max: 0.98 msec % below 1msec: 100% Avg: 313RMB / 9.4WMB Max: 1.71RGB / 96WMB Avg: 1.45Gbps (Tx) / 0.97Gbps (Rx) Max: 1.6Gbps (Tx) / 1.2Gbps (Rx) Test setup: • Redis Labs Enterprise Cluster v3.2 • Dell Xeon CPU E5- 2670 v3 @ 2.50GHz • 4x Samsung NVMe PM1725 • Memtier benchmark- open source tool • 100B object size • 80% read • 20% write Throughput – ops/sec Latency – msec Disk Bandwidth – MB/sec NW Bandwidth – Gb/sec >2M Ops/sec, <1 ms latency, > 1GB disk bandwidth
- 26. 26 A Real Life Example With Redis On Flash Customer Scenario: • Genome dataset • Key sizes: 32B, value sizes : 5-12B • No of keys: 250 x109… 250 x1012 Key1: AAAAAAAAAAAAAAAAAAAAAAAACCCCAAA Value1 = Freq=4 IE=A OE=A Key2: AAAAAAAAAAAAAAAAAAAAAAAAACAACCC Value2 = Freq=7 IE=A,C,T,G OE=A,C,T,G *IE – Inside End sequences *OE – Outside End sequences
- 27. 27 Optimizing Redis Usage RAW • # of keys 250x10^9 • Key size = 32B • Value size = 8B (average) • Overhead per object (key+value) = ±61B (key) + 9 (value) = 70 • Internal fragmentation per object = 14B • RAM size = ±31TB
- 28. 28 Further Optimizations Encoding keys and values to compress sizes • # of keys 1.25x10^9 • Key size = 4B • Value size = 3612B • RAM overhead per object (key+value) = ±40B • RAM size = ±55GB // for optimal performance we used 500GB to keep 10/90 RAM/Flash ratio • Flash size = ±4.5TB Using Redis Hashes to store compressed keys/values (200 keys and values per hash) 1 2
- 29. 29 Memory Usage and Cost Comparison Redis on RAM Strings Redis on Flash Hashes RAM size 31TB 0.5 TB Flash size - 4.5TB EC2 instances 155 x r3.8xlarge 2 x i2.8xlarge 1yr costs (reserved instances) $2,017,325 $49,862 Savings with Flash & Hashes % 97.6%
- 30. Home of Redis Redis in Analytics
- 31. 31 Spark & Redis - Connector & Service Layer Data Source Serving Layer Spark SQL & Data Frame RDD, Data Source, Data Set RDD, Data Source, Data Set Analytics & BI
- 32. 32 Spark & Redis – Internal Accelerator Data Source RDD, Data Source, Data Set RDD, Data Source, Data Set Spark SQL & Data Frame Analytics & BI RDD, Data Source, Data Set, Redis API Data Sink
- 33. 33 Accelerate Spark Time-Series with Redis Redis sorted sets accelerate time series data processing by 100 times compared to other in- memory K/V stores Example time series data: Stock prices for 1024 stocks over 32 years
- 34. 34 Spark-Redis Package : The Results Redis faster by upto 100 times compared to HDFS and over 45 times compared to Tachyon or Spark
- 35. Home of Redis Redis Modules
- 36. 36 36 Modules Extend Redis’ Use Case Coverage MongoDB Cassandra/ Datastax Couchbase Redis (original) Redis + Modules Single View Coming Soon Personalization Catalog Coming Soon IoT Real-Time Analytics Content Management Coming Soon Messaging Fraud Detection Coming Soon Graph Coming Soon Time Series Caching Text Search Coming Soon Image Processing Coming Soon Machine Learning Coming Soon Linear Algebra Coming Soon Probabilistic data structures for processing continuous, Coming Soon More
- 37. 37 Modules Turn Redis into a Multi-Model Database 37 MongoDB Cassandra/ Datastax Couchbase Redis (original) Redis + Modules Document-Based Coming Soon Column-Based Coming Soon Key-Value Data Structures Data Structures Graph Coming Soon
- 38. 38 Redis Module Hub • A Redis Module Marketplace – for everyone • Every problem a developer solves with Redis – now extended to the Enterprise • Will help developers monetize their work and reach enterprise Redis users • Will give enterprise Redis users the confidence and peace of mind to easily deploy modules www.redismodules.com
- 39. 39
- 40. 40 40 3.15 2.40 21.00 8.70 24.57 10.61 0.00 5.00 10.00 15.00 20.00 25.00 30.00 Full text search Prefix search Average Latency (msec) RLEC Elasticsearch Solr 20,045 6,831 690 3,686 621 3,133 0 5,000 10,000 15,000 20,000 25,000 Full text search Prefix search Ops/sec RLEC Elasticsearch Solr 85% higher 32x higher 7.8x faster 4.1x faster redisearch The world fastest text search engine
- 41. 41 What Can Modules Do 41 • All modules are certified by Redis Labs for full compliance with OSS Redis, Redis Cloud and Redis Labs Enterprise Cluster (RLEC) Full Text Search Enhanced JSON Graph Operations Secondary Indexes Linear Algebra SQL Support Image Processing N-Dimension Queries …
- 42. Thank You!
Editor's Notes
- #9: Insert Version Number Here
- #22: DRAM prices have been relatively stable over the years – and it continues to be expensive. Technologies such as Flash offer performance that is 3-4 orders of magnitude slower but 10 times cheaper. Emerging technologies such as Flash offer performance that is only an order of magnitude slower at 3 times lower cost. This makes for quite an attractive cost-performance tradeoff!
- #23: We extended Redis to take advantage of the multithreaded and asyn nature of Flash/other slower memory. Not only that, we added the capability to recognize “fast” and “slow” memory – with a configurable ratio so that all keys and hot values can be stored in the fast memory and cold values in slow memory such as Flash, 3 D Cross point or Storage Class memory for optimum performance.
- #26: NVMe – is easily x40 the throughput of SATA based Flash
- #29: We encoded and compressed the keys and values–keys were 31 bytes of string of 4 nucleobases. We encoded so that they could be represented with 2 bits per nucleobase – 62 bits and values were similarly compressed. Flash works at 4KB blocks size, so hash sizes < 4KB. Limited hashes to 200 entries to achieve the 4kb size per hash