SlideShare a Scribd company logo
Redis Memory Optimization
Store More Data in Less Memory
Sripathi Krishnan
CTO, HashedIn Inc. / Rdbtools
@srithedabbler
- Using redis since ~2010
- Top redis answers on StackOverflow
- Author of redis-rdb-tools
- Redis meetups across India
- Now launching RdbTools.com in beta
sripathi@rdbtools.com
github.com/sripathikrishnan
RAM prices are going
through the roof
Users want faster applications,
which needs more memory
And yet, data stored
keeps growing
Store more data in less memory.
That’s what this talk is about.
Optimizing Key Value Pairs
Part I
#1: “Normalize” Your Objects
USER
ID
First Name
Last Name
Photo
* Fav Articles
ARTICLE
ID
Title
Body
Author
Images
Videos
1
⋈
What happens when several users
favourite the same articles?
#1: “Normalize” Your Objects
users:<id>
ID
First Name
Last Name
Photo
articles:<id>
ID
Title
Body
Author
Images
Videos
Instead, store under 3 keys - users, articles, and
favourite_articles
users:<id>:favourite_articles
- Java / Python / Php have poor serializers
#2: Use a Better Serializer
- Java / Python / Php have poor serializers
- In Java, don’t want to change code?
- Try Kryo
#2: Use a Better Serializer
- Java / Python / Php have poor serializers
- In Java, don’t want to change code?
- Try Kryo
- Willing to modify code schema?
- Try a binary format - ProtoBuf / FlatBuffers / MsgPack
#2: Use a Better Serializer
#3: Compress Data!
- Easily save 50% memory + Network Bandwidth
- Latency Sensitive?
- LZO, Google Snappy
- Maximum Compression?
- Gzip, Brotli
- For comparison of algorithms, see
https://quixdb.github.io/squash-benchmark/#results
#4: JSON -> MsgPack
- It’s like JSON, but fast and small.
- Libraries in 50+ languages
- Plus, lua scripts in redis can parse MsgPack!
#5: Combine Small Objects
Key Value
zipcodes:90210 {"city":.. , "state": .. }
zipcodes:90421 {"city":.. , "state": .. }
zipcodes:43232 {"city":.. , "state": .. }
zipcodes:43010 {"city":.. , "state": .. }
zipcodes:43593 {"city":.. , "state": .. }
zipcodes:32142 {"city":.. , "state": .. }
zipcodes:32113 {"city":.. , "state": .. }
zipcodes:32431 {"city":.. , "state": .. }
commands: set / get
#5: Combine Small Objects
Key Value
zipcodes:90210 {"city":.. , "state": .. }
zipcodes:90421 {"city":.. , "state": .. }
zipcodes:43232 {"city":.. , "state": .. }
zipcodes:43010 {"city":.. , "state": .. }
zipcodes:43593 {"city":.. , "state": .. }
zipcodes:32142 {"city":.. , "state": .. }
zipcodes:32113 {"city":.. , "state": .. }
zipcodes:32431 {"city":.. , "state": .. }
Key Field Value
zipcodes:90 210 {"city":.. , "state": .. }
421 {"city":.. , "state": .. }
zipcodes:43 232 {"city":.. , "state": .. }
010 {"city":.. , "state": .. }
593 {"city":.. , "state": .. }
zipcodes:32 142 {"city":.. , "state": .. }
113 {"city":.. , "state": .. }
431 {"city":.. , "state": .. }
commands: set / get commands: hset / hget
#5: Combine Small Objects
- Objects between 512 bytes to 1 KB can be combined in a
larger hash
- Caveat: Expiry isn’t supported
See: Instagram’s Blog on Storing hundreds of millions of keys
#6: Prefer Hashes to JSON
- If your JSON is flat / not nested - just use a redis hash
- You won’t save much memory…
- … But you get the ability to read/update parts of the object
Wrap your Redis Library
- Build a wrapper around redis library
- Transparently compress / combine / change serializer etc.
without affecting business logic
- Easy to test various algorithms
Optimizing Hashes
Part II
#7: Small Hashes? Tweak Your Config
hash-max-ziplist-entries & hash-max-ziplist-value
- Use redis-rdb-tools to find the right values
- Monitor latency using info commanstats
TODO:
Insert optimization chart
#8: Few Large Fields?
users:<ID> : {“name”: …,
“age”:..,
“role”: …,
“about-me”: <LONG TEXT>
}
Does NOT use ziplist encoding
#8: Few Large Fields?
users:<ID> {“name”: …, “age”:..,“role”:…,}
users:about-me {
<ID2>: <large text>,
<ID2>: <large text>
}
Move the large field to a separate hash altogether.
#9: Large Hashes?
1 Large Hash "zipcodes"
Field Value
90210 {"city":.. , "state": .. }
90421 {"city":.. , "state": .. }
43232 {"city":.. , "state": .. }
43010 {"city":.. , "state": .. }
43593 {"city":.. , "state": .. }
32142 {"city":.. , "state": .. }
32113 {"city":.. , "state": .. }
32431 {"city":.. , "state": .. }
#9: Large Hashes? Shard Them!
1 Large Hash "zipcodes"
Field Value
90210 {"city":.. , "state": .. }
90421 {"city":.. , "state": .. }
43232 {"city":.. , "state": .. }
43010 {"city":.. , "state": .. }
43593 {"city":.. , "state": .. }
32142 {"city":.. , "state": .. }
32113 {"city":.. , "state": .. }
32431 {"city":.. , "state": .. }
Multiple Smaller Hashes
Key Field Value
zipcodes:90 210 {"city":.. , "state": .. }
421 {"city":.. , "state": .. }
zipcodes:43 232 {"city":.. , "state": .. }
010 {"city":.. , "state": .. }
593 {"city":.. , "state": .. }
zipcodes:32 142 {"city":.. , "state": .. }
113 {"city":.. , "state": .. }
431 {"city":.. , "state": .. }
Smaller hashes use the efficient ziplist encoding
Combine Strings or Split Large Hash
#5
Thousands of Small
Strings
Hundreds of Small Hashes
and then adjust hash-max-ziplist-*
#9
1 Large Hash with
Thousands of Elements
It’s the same idea!
#10: Many Similar Hashes?
- Use shorter field names / integer indexes
- 1M hashes, 10 fields x 10 bytes each = Savings of 100MB
- Wrap redis client library, so no change to application code
Complicates application code, do the math before implementing!
Optimizing Set
Part III
#11: Prefer Integer IDs in Sets
users:13232 v/s users:sripathi
125 198 243 398 432 457 598 607 643 743 879
Redis Set stored using IntSet data structure:
category:<name>:users
#12: Map Strings IDs to Ints if necessary
125 198 243 398 432 457 598 607 643 743 879
users_by_name: { “Sripathi” => 125
“Tom” => 198
... }
category:books:users
125 243 457 607 864 879
category:electronics:users
#13: Adjust set-max-intset-entries
- Default is 512
- You can increase it as much as 1500.
- Check for latency increase using info commandstats
- Counting Unique Objects? Use HyperLogLog
- Check for Existence? Use Bloom Filter (see rebloom module)
You won’t get exact results, but the memory savings are worth it.
#14: Probabilistic Data Structures
See RedisConf 2018 talks:
- Deduplicating Data Streams with Bloom Filters
- Real-Time Log Analytics Using Probabilistic Data Structures in
Redis
#14: Probabilistic Data Structures
#15: Use Bitmaps to Store Binary Flags
#15: Use Bitmaps to Store Binary Flags
Bitmaps are efficient IF there is 40-60% probability of an
element being present. Otherwise, just use regular set.
Optimizing List
Part IV
#16: Enable Compression on Lists
Redis does not compress list elements by default. If you store
large values, enable compression.
Set list-compress-depth to 1 or higher in your
configuration.
#17: Upgrade Redis if < 3.2
Redis 3.2 introduced a new encoding - quicklist - which saves
a LOT of memory. Upgrade highly recommended!
Bit Packing & App Specific
Data Formats
Part V
#18: Use Bitfield command
Credits: Reddit’s 2017 talk on building /r/place using bitfield
#18: Use Bitfield command
If your data structure has lots of integers, floats or fixed-width
strings, you can use bitfield. It’s like a struct in C.
Great for large matrix of numbers, large arrays, large number
of small objects that are fixed width.
- See bitfield command.
- See reddit’s 2017 talk on building /r/place using bitfield
#19: Create your own data structures
Redis strings are extremely versatile, you can build complex
data structures.
- Use getrange / setrange and similar commands from app
- Write lua scripts
- Write a custom module
Optimizing Streams
Part VI
#20: Try keeping same field names
Server: 101
CPU: 83
Memory: 90
Server: 201
CPU: 43
Memory: 20
Server: 301
CPU: 55
Memory: 39
Server: 101
CPU: 70
Server: 301
CPU: 50
Memory: 43
<unixtime>.<seq> 123.0 125.0 130.0 130.1
Reference
Record
SameFields =
True
SameFields = False
Field names are stored again in memory.
time
Advance Configuration
Part VI
#21: Adjust maxmemory-samples
If your keys have short expiry, and you have a large number of
keys - redis may not reclaim memory.
Increase maxmemory-samples to instruct redis to spend more cpu
cycles to free memory.
#22: Enable Active Defragmentation
activedefrag yes
If you have high defragmentation, Redis can defragment without
having to restart. This can be enabled at runtime also.
#23: Switch to 32 Bit Redis
If your dataset is less than 3GB, switching to 32 bit redis will
save memory.
#24: Watch LUA memory usage
Redis caches LUA scripts, and never flushes them. If you generate
scripts dynamically, you will end up wasting a lot of memory.
- Don’t generate dynamic scripts, parameterize
- script flush to manually clear cache
#25: Use rdbtools.com!
Take the guesswork out of your memory analysis. Visit
https://rdbtools.com
Thank You

More Related Content

What's hot (20)

PPTX
DirectX 11 Rendering in Battlefield 3
Electronic Arts / DICE
 
PPTX
Using Time Window Compaction Strategy For Time Series Workloads
Jeff Jirsa
 
PDF
Introduction to Cassandra
Gokhan Atil
 
PDF
Materialized Column: An Efficient Way to Optimize Queries on Nested Columns
Databricks
 
PDF
MongoDB World 2019: Finding the Right MongoDB Atlas Cluster Size: Does This I...
MongoDB
 
PDF
Forward+ (EUROGRAPHICS 2012)
Takahiro Harada
 
PPTX
Mongodb introduction and_internal(simple)
Kai Zhao
 
PPTX
Game object models - Game Engine Architecture
Shawn Presser
 
PPTX
MongoDB (Advanced)
TO THE NEW | Technology
 
PDF
Container Performance Analysis
Brendan Gregg
 
PDF
ClickHouse Deep Dive, by Aleksei Milovidov
Altinity Ltd
 
PDF
MongodB Internals
Norberto Leite
 
PDF
Redis - Usability and Use Cases
Fabrizio Farinacci
 
PPTX
Optimizing the Graphics Pipeline with Compute, GDC 2016
Graham Wihlidal
 
PDF
Intro to Neo4j and Graph Databases
Neo4j
 
PDF
Redis cluster
iammutex
 
PDF
Introducing DataFrames in Spark for Large Scale Data Science
Databricks
 
PPTX
Mongo DB Presentation
Jaya Naresh Kovela
 
PDF
Webinar: Secrets of ClickHouse Query Performance, by Robert Hodges
Altinity Ltd
 
PPTX
Introduction to .NET Core
Marco Parenzan
 
DirectX 11 Rendering in Battlefield 3
Electronic Arts / DICE
 
Using Time Window Compaction Strategy For Time Series Workloads
Jeff Jirsa
 
Introduction to Cassandra
Gokhan Atil
 
Materialized Column: An Efficient Way to Optimize Queries on Nested Columns
Databricks
 
MongoDB World 2019: Finding the Right MongoDB Atlas Cluster Size: Does This I...
MongoDB
 
Forward+ (EUROGRAPHICS 2012)
Takahiro Harada
 
Mongodb introduction and_internal(simple)
Kai Zhao
 
Game object models - Game Engine Architecture
Shawn Presser
 
MongoDB (Advanced)
TO THE NEW | Technology
 
Container Performance Analysis
Brendan Gregg
 
ClickHouse Deep Dive, by Aleksei Milovidov
Altinity Ltd
 
MongodB Internals
Norberto Leite
 
Redis - Usability and Use Cases
Fabrizio Farinacci
 
Optimizing the Graphics Pipeline with Compute, GDC 2016
Graham Wihlidal
 
Intro to Neo4j and Graph Databases
Neo4j
 
Redis cluster
iammutex
 
Introducing DataFrames in Spark for Large Scale Data Science
Databricks
 
Mongo DB Presentation
Jaya Naresh Kovela
 
Webinar: Secrets of ClickHouse Query Performance, by Robert Hodges
Altinity Ltd
 
Introduction to .NET Core
Marco Parenzan
 

Similar to RedisConf18 - Redis Memory Optimization (20)

PPTX
Introduction to MongoDB at IGDTUW
Ankur Raina
 
PDF
Simplifying & accelerating application development with MongoDB's intelligent...
Maxime Beugnet
 
PDF
HBaseCon 2015: S2Graph - A Large-scale Graph Database with HBase
HBaseCon
 
PDF
MongoDB Meetup
Maxime Beugnet
 
PDF
MongoDB.local DC 2018: Tutorial - Data Analytics with MongoDB
MongoDB
 
PDF
MongoDB Europe 2016 - Graph Operations with MongoDB
MongoDB
 
PDF
What's new in Redis v3.2
Itamar Haber
 
PPTX
Eagle6 mongo dc revised
MongoDB
 
PPTX
Eagle6 Enterprise Situational Awareness
MongoDB
 
PPTX
Benefits of Using MongoDB Over RDBMSs
MongoDB
 
PDF
mongodb-introduction
Tse-Ching Ho
 
PPTX
Jumpstart: MongoDB BI Connector & Tableau
MongoDB
 
PDF
Analytics with MongoDB Aggregation Framework and Hadoop Connector
Henrik Ingo
 
PPT
Distributed Queries in IDS: New features.
Keshav Murthy
 
PPTX
SenchaCon 2016: Handle Real-World Data with Confidence - Fredric Berling
Sencha
 
PPTX
MongoDB: Comparing WiredTiger In-Memory Engine to Redis
Jason Terpko
 
PDF
MongoDB Tips and Tricks
M Malai
 
PDF
Managing your black friday logs - Code Europe
David Pilato
 
ODP
Perchè potresti aver bisogno di un database NoSQL anche se non sei Google o F...
Codemotion
 
PDF
Virtual training optimizing the tick stack
InfluxData
 
Introduction to MongoDB at IGDTUW
Ankur Raina
 
Simplifying & accelerating application development with MongoDB's intelligent...
Maxime Beugnet
 
HBaseCon 2015: S2Graph - A Large-scale Graph Database with HBase
HBaseCon
 
MongoDB Meetup
Maxime Beugnet
 
MongoDB.local DC 2018: Tutorial - Data Analytics with MongoDB
MongoDB
 
MongoDB Europe 2016 - Graph Operations with MongoDB
MongoDB
 
What's new in Redis v3.2
Itamar Haber
 
Eagle6 mongo dc revised
MongoDB
 
Eagle6 Enterprise Situational Awareness
MongoDB
 
Benefits of Using MongoDB Over RDBMSs
MongoDB
 
mongodb-introduction
Tse-Ching Ho
 
Jumpstart: MongoDB BI Connector & Tableau
MongoDB
 
Analytics with MongoDB Aggregation Framework and Hadoop Connector
Henrik Ingo
 
Distributed Queries in IDS: New features.
Keshav Murthy
 
SenchaCon 2016: Handle Real-World Data with Confidence - Fredric Berling
Sencha
 
MongoDB: Comparing WiredTiger In-Memory Engine to Redis
Jason Terpko
 
MongoDB Tips and Tricks
M Malai
 
Managing your black friday logs - Code Europe
David Pilato
 
Perchè potresti aver bisogno di un database NoSQL anche se non sei Google o F...
Codemotion
 
Virtual training optimizing the tick stack
InfluxData
 
Ad

More from Redis Labs (20)

PPTX
Redis Day Bangalore 2020 - Session state caching with redis
Redis Labs
 
PPTX
Protecting Your API with Redis by Jane Paek - Redis Day Seattle 2020
Redis Labs
 
PPTX
The Happy Marriage of Redis and Protobuf by Scott Haines of Twilio - Redis Da...
Redis Labs
 
PPTX
SQL, Redis and Kubernetes by Paul Stanton of Windocks - Redis Day Seattle 2020
Redis Labs
 
PPTX
Rust and Redis - Solving Problems for Kubernetes by Ravi Jagannathan of VMwar...
Redis Labs
 
PPTX
Redis for Data Science and Engineering by Dmitry Polyakovsky of Oracle
Redis Labs
 
PPTX
Practical Use Cases for ACLs in Redis 6 by Jamie Scott - Redis Day Seattle 2020
Redis Labs
 
PPTX
Moving Beyond Cache by Yiftach Shoolman Redis Labs - Redis Day Seattle 2020
Redis Labs
 
PPTX
Leveraging Redis for System Monitoring by Adam McCormick of SBG - Redis Day S...
Redis Labs
 
PPTX
JSON in Redis - When to use RedisJSON by Jay Won of Coupang - Redis Day Seatt...
Redis Labs
 
PPTX
Highly Available Persistent Session Management Service by Mohamed Elmergawi o...
Redis Labs
 
PPTX
Anatomy of a Redis Command by Madelyn Olson of Amazon Web Services - Redis Da...
Redis Labs
 
PPTX
Building a Multi-dimensional Analytics Engine with RedisGraph by Matthew Goos...
Redis Labs
 
PPTX
RediSearch 1.6 by Pieter Cailliau - Redis Day Bangalore 2020
Redis Labs
 
PPTX
RedisGraph 2.0 by Pieter Cailliau - Redis Day Bangalore 2020
Redis Labs
 
PPTX
RedisTimeSeries 1.2 by Pieter Cailliau - Redis Day Bangalore 2020
Redis Labs
 
PPTX
RedisAI 0.9 by Sherin Thomas of Tensorwerk - Redis Day Bangalore 2020
Redis Labs
 
PPTX
Rate-Limiting 30 Million requests by Vijay Lakshminarayanan and Girish Koundi...
Redis Labs
 
PDF
Three Pillars of Observability by Rajalakshmi Raji Srinivasan of Site24x7 Zoh...
Redis Labs
 
PPTX
Solving Complex Scaling Problems by Prashant Kumar and Abhishek Jain of Myntr...
Redis Labs
 
Redis Day Bangalore 2020 - Session state caching with redis
Redis Labs
 
Protecting Your API with Redis by Jane Paek - Redis Day Seattle 2020
Redis Labs
 
The Happy Marriage of Redis and Protobuf by Scott Haines of Twilio - Redis Da...
Redis Labs
 
SQL, Redis and Kubernetes by Paul Stanton of Windocks - Redis Day Seattle 2020
Redis Labs
 
Rust and Redis - Solving Problems for Kubernetes by Ravi Jagannathan of VMwar...
Redis Labs
 
Redis for Data Science and Engineering by Dmitry Polyakovsky of Oracle
Redis Labs
 
Practical Use Cases for ACLs in Redis 6 by Jamie Scott - Redis Day Seattle 2020
Redis Labs
 
Moving Beyond Cache by Yiftach Shoolman Redis Labs - Redis Day Seattle 2020
Redis Labs
 
Leveraging Redis for System Monitoring by Adam McCormick of SBG - Redis Day S...
Redis Labs
 
JSON in Redis - When to use RedisJSON by Jay Won of Coupang - Redis Day Seatt...
Redis Labs
 
Highly Available Persistent Session Management Service by Mohamed Elmergawi o...
Redis Labs
 
Anatomy of a Redis Command by Madelyn Olson of Amazon Web Services - Redis Da...
Redis Labs
 
Building a Multi-dimensional Analytics Engine with RedisGraph by Matthew Goos...
Redis Labs
 
RediSearch 1.6 by Pieter Cailliau - Redis Day Bangalore 2020
Redis Labs
 
RedisGraph 2.0 by Pieter Cailliau - Redis Day Bangalore 2020
Redis Labs
 
RedisTimeSeries 1.2 by Pieter Cailliau - Redis Day Bangalore 2020
Redis Labs
 
RedisAI 0.9 by Sherin Thomas of Tensorwerk - Redis Day Bangalore 2020
Redis Labs
 
Rate-Limiting 30 Million requests by Vijay Lakshminarayanan and Girish Koundi...
Redis Labs
 
Three Pillars of Observability by Rajalakshmi Raji Srinivasan of Site24x7 Zoh...
Redis Labs
 
Solving Complex Scaling Problems by Prashant Kumar and Abhishek Jain of Myntr...
Redis Labs
 
Ad

Recently uploaded (20)

PPTX
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
PDF
Bitcoin for Millennials podcast with Bram, Power Laws of Bitcoin
Stephen Perrenod
 
PDF
The 2025 InfraRed Report - Redpoint Ventures
Razin Mustafiz
 
PDF
What’s my job again? Slides from Mark Simos talk at 2025 Tampa BSides
Mark Simos
 
PDF
Staying Human in a Machine- Accelerated World
Catalin Jora
 
PDF
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
PDF
UiPath DevConnect 2025: Agentic Automation Community User Group Meeting
DianaGray10
 
PPTX
New ThousandEyes Product Innovations: Cisco Live June 2025
ThousandEyes
 
PPTX
Agentforce World Tour Toronto '25 - Supercharge MuleSoft Development with Mod...
Alexandra N. Martinez
 
PPTX
Seamless Tech Experiences Showcasing Cross-Platform App Design.pptx
presentifyai
 
PPTX
The Project Compass - GDG on Campus MSIT
dscmsitkol
 
PDF
Peak of Data & AI Encore AI-Enhanced Workflows for the Real World
Safe Software
 
PDF
“Squinting Vision Pipelines: Detecting and Correcting Errors in Vision Models...
Edge AI and Vision Alliance
 
PDF
NASA A Researcher’s Guide to International Space Station : Physical Sciences ...
Dr. PANKAJ DHUSSA
 
PDF
Newgen Beyond Frankenstein_Build vs Buy_Digital_version.pdf
darshakparmar
 
PPTX
MuleSoft MCP Support (Model Context Protocol) and Use Case Demo
shyamraj55
 
PDF
[Newgen] NewgenONE Marvin Brochure 1.pdf
darshakparmar
 
PDF
Agentic AI lifecycle for Enterprise Hyper-Automation
Debmalya Biswas
 
PDF
“Computer Vision at Sea: Automated Fish Tracking for Sustainable Fishing,” a ...
Edge AI and Vision Alliance
 
PDF
How do you fast track Agentic automation use cases discovery?
DianaGray10
 
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
Bitcoin for Millennials podcast with Bram, Power Laws of Bitcoin
Stephen Perrenod
 
The 2025 InfraRed Report - Redpoint Ventures
Razin Mustafiz
 
What’s my job again? Slides from Mark Simos talk at 2025 Tampa BSides
Mark Simos
 
Staying Human in a Machine- Accelerated World
Catalin Jora
 
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
UiPath DevConnect 2025: Agentic Automation Community User Group Meeting
DianaGray10
 
New ThousandEyes Product Innovations: Cisco Live June 2025
ThousandEyes
 
Agentforce World Tour Toronto '25 - Supercharge MuleSoft Development with Mod...
Alexandra N. Martinez
 
Seamless Tech Experiences Showcasing Cross-Platform App Design.pptx
presentifyai
 
The Project Compass - GDG on Campus MSIT
dscmsitkol
 
Peak of Data & AI Encore AI-Enhanced Workflows for the Real World
Safe Software
 
“Squinting Vision Pipelines: Detecting and Correcting Errors in Vision Models...
Edge AI and Vision Alliance
 
NASA A Researcher’s Guide to International Space Station : Physical Sciences ...
Dr. PANKAJ DHUSSA
 
Newgen Beyond Frankenstein_Build vs Buy_Digital_version.pdf
darshakparmar
 
MuleSoft MCP Support (Model Context Protocol) and Use Case Demo
shyamraj55
 
[Newgen] NewgenONE Marvin Brochure 1.pdf
darshakparmar
 
Agentic AI lifecycle for Enterprise Hyper-Automation
Debmalya Biswas
 
“Computer Vision at Sea: Automated Fish Tracking for Sustainable Fishing,” a ...
Edge AI and Vision Alliance
 
How do you fast track Agentic automation use cases discovery?
DianaGray10
 

RedisConf18 - Redis Memory Optimization

  • 1. Redis Memory Optimization Store More Data in Less Memory
  • 2. Sripathi Krishnan CTO, HashedIn Inc. / Rdbtools @srithedabbler - Using redis since ~2010 - Top redis answers on StackOverflow - Author of redis-rdb-tools - Redis meetups across India - Now launching RdbTools.com in beta [email protected] github.com/sripathikrishnan
  • 3. RAM prices are going through the roof
  • 4. Users want faster applications, which needs more memory
  • 5. And yet, data stored keeps growing
  • 6. Store more data in less memory. That’s what this talk is about.
  • 7. Optimizing Key Value Pairs Part I
  • 8. #1: “Normalize” Your Objects USER ID First Name Last Name Photo * Fav Articles ARTICLE ID Title Body Author Images Videos 1 ⋈ What happens when several users favourite the same articles?
  • 9. #1: “Normalize” Your Objects users:<id> ID First Name Last Name Photo articles:<id> ID Title Body Author Images Videos Instead, store under 3 keys - users, articles, and favourite_articles users:<id>:favourite_articles
  • 10. - Java / Python / Php have poor serializers #2: Use a Better Serializer
  • 11. - Java / Python / Php have poor serializers - In Java, don’t want to change code? - Try Kryo #2: Use a Better Serializer
  • 12. - Java / Python / Php have poor serializers - In Java, don’t want to change code? - Try Kryo - Willing to modify code schema? - Try a binary format - ProtoBuf / FlatBuffers / MsgPack #2: Use a Better Serializer
  • 13. #3: Compress Data! - Easily save 50% memory + Network Bandwidth - Latency Sensitive? - LZO, Google Snappy - Maximum Compression? - Gzip, Brotli - For comparison of algorithms, see https://quixdb.github.io/squash-benchmark/#results
  • 14. #4: JSON -> MsgPack - It’s like JSON, but fast and small. - Libraries in 50+ languages - Plus, lua scripts in redis can parse MsgPack!
  • 15. #5: Combine Small Objects Key Value zipcodes:90210 {"city":.. , "state": .. } zipcodes:90421 {"city":.. , "state": .. } zipcodes:43232 {"city":.. , "state": .. } zipcodes:43010 {"city":.. , "state": .. } zipcodes:43593 {"city":.. , "state": .. } zipcodes:32142 {"city":.. , "state": .. } zipcodes:32113 {"city":.. , "state": .. } zipcodes:32431 {"city":.. , "state": .. } commands: set / get
  • 16. #5: Combine Small Objects Key Value zipcodes:90210 {"city":.. , "state": .. } zipcodes:90421 {"city":.. , "state": .. } zipcodes:43232 {"city":.. , "state": .. } zipcodes:43010 {"city":.. , "state": .. } zipcodes:43593 {"city":.. , "state": .. } zipcodes:32142 {"city":.. , "state": .. } zipcodes:32113 {"city":.. , "state": .. } zipcodes:32431 {"city":.. , "state": .. } Key Field Value zipcodes:90 210 {"city":.. , "state": .. } 421 {"city":.. , "state": .. } zipcodes:43 232 {"city":.. , "state": .. } 010 {"city":.. , "state": .. } 593 {"city":.. , "state": .. } zipcodes:32 142 {"city":.. , "state": .. } 113 {"city":.. , "state": .. } 431 {"city":.. , "state": .. } commands: set / get commands: hset / hget
  • 17. #5: Combine Small Objects - Objects between 512 bytes to 1 KB can be combined in a larger hash - Caveat: Expiry isn’t supported See: Instagram’s Blog on Storing hundreds of millions of keys
  • 18. #6: Prefer Hashes to JSON - If your JSON is flat / not nested - just use a redis hash - You won’t save much memory… - … But you get the ability to read/update parts of the object
  • 19. Wrap your Redis Library - Build a wrapper around redis library - Transparently compress / combine / change serializer etc. without affecting business logic - Easy to test various algorithms
  • 21. #7: Small Hashes? Tweak Your Config hash-max-ziplist-entries & hash-max-ziplist-value - Use redis-rdb-tools to find the right values - Monitor latency using info commanstats TODO: Insert optimization chart
  • 22. #8: Few Large Fields? users:<ID> : {“name”: …, “age”:.., “role”: …, “about-me”: <LONG TEXT> } Does NOT use ziplist encoding
  • 23. #8: Few Large Fields? users:<ID> {“name”: …, “age”:..,“role”:…,} users:about-me { <ID2>: <large text>, <ID2>: <large text> } Move the large field to a separate hash altogether.
  • 24. #9: Large Hashes? 1 Large Hash "zipcodes" Field Value 90210 {"city":.. , "state": .. } 90421 {"city":.. , "state": .. } 43232 {"city":.. , "state": .. } 43010 {"city":.. , "state": .. } 43593 {"city":.. , "state": .. } 32142 {"city":.. , "state": .. } 32113 {"city":.. , "state": .. } 32431 {"city":.. , "state": .. }
  • 25. #9: Large Hashes? Shard Them! 1 Large Hash "zipcodes" Field Value 90210 {"city":.. , "state": .. } 90421 {"city":.. , "state": .. } 43232 {"city":.. , "state": .. } 43010 {"city":.. , "state": .. } 43593 {"city":.. , "state": .. } 32142 {"city":.. , "state": .. } 32113 {"city":.. , "state": .. } 32431 {"city":.. , "state": .. } Multiple Smaller Hashes Key Field Value zipcodes:90 210 {"city":.. , "state": .. } 421 {"city":.. , "state": .. } zipcodes:43 232 {"city":.. , "state": .. } 010 {"city":.. , "state": .. } 593 {"city":.. , "state": .. } zipcodes:32 142 {"city":.. , "state": .. } 113 {"city":.. , "state": .. } 431 {"city":.. , "state": .. } Smaller hashes use the efficient ziplist encoding
  • 26. Combine Strings or Split Large Hash #5 Thousands of Small Strings Hundreds of Small Hashes and then adjust hash-max-ziplist-* #9 1 Large Hash with Thousands of Elements It’s the same idea!
  • 27. #10: Many Similar Hashes? - Use shorter field names / integer indexes - 1M hashes, 10 fields x 10 bytes each = Savings of 100MB - Wrap redis client library, so no change to application code Complicates application code, do the math before implementing!
  • 29. #11: Prefer Integer IDs in Sets users:13232 v/s users:sripathi 125 198 243 398 432 457 598 607 643 743 879 Redis Set stored using IntSet data structure: category:<name>:users
  • 30. #12: Map Strings IDs to Ints if necessary 125 198 243 398 432 457 598 607 643 743 879 users_by_name: { “Sripathi” => 125 “Tom” => 198 ... } category:books:users 125 243 457 607 864 879 category:electronics:users
  • 31. #13: Adjust set-max-intset-entries - Default is 512 - You can increase it as much as 1500. - Check for latency increase using info commandstats
  • 32. - Counting Unique Objects? Use HyperLogLog - Check for Existence? Use Bloom Filter (see rebloom module) You won’t get exact results, but the memory savings are worth it. #14: Probabilistic Data Structures
  • 33. See RedisConf 2018 talks: - Deduplicating Data Streams with Bloom Filters - Real-Time Log Analytics Using Probabilistic Data Structures in Redis #14: Probabilistic Data Structures
  • 34. #15: Use Bitmaps to Store Binary Flags
  • 35. #15: Use Bitmaps to Store Binary Flags Bitmaps are efficient IF there is 40-60% probability of an element being present. Otherwise, just use regular set.
  • 37. #16: Enable Compression on Lists Redis does not compress list elements by default. If you store large values, enable compression. Set list-compress-depth to 1 or higher in your configuration.
  • 38. #17: Upgrade Redis if < 3.2 Redis 3.2 introduced a new encoding - quicklist - which saves a LOT of memory. Upgrade highly recommended!
  • 39. Bit Packing & App Specific Data Formats Part V
  • 40. #18: Use Bitfield command Credits: Reddit’s 2017 talk on building /r/place using bitfield
  • 41. #18: Use Bitfield command If your data structure has lots of integers, floats or fixed-width strings, you can use bitfield. It’s like a struct in C. Great for large matrix of numbers, large arrays, large number of small objects that are fixed width. - See bitfield command. - See reddit’s 2017 talk on building /r/place using bitfield
  • 42. #19: Create your own data structures Redis strings are extremely versatile, you can build complex data structures. - Use getrange / setrange and similar commands from app - Write lua scripts - Write a custom module
  • 44. #20: Try keeping same field names Server: 101 CPU: 83 Memory: 90 Server: 201 CPU: 43 Memory: 20 Server: 301 CPU: 55 Memory: 39 Server: 101 CPU: 70 Server: 301 CPU: 50 Memory: 43 <unixtime>.<seq> 123.0 125.0 130.0 130.1 Reference Record SameFields = True SameFields = False Field names are stored again in memory. time
  • 46. #21: Adjust maxmemory-samples If your keys have short expiry, and you have a large number of keys - redis may not reclaim memory. Increase maxmemory-samples to instruct redis to spend more cpu cycles to free memory.
  • 47. #22: Enable Active Defragmentation activedefrag yes If you have high defragmentation, Redis can defragment without having to restart. This can be enabled at runtime also.
  • 48. #23: Switch to 32 Bit Redis If your dataset is less than 3GB, switching to 32 bit redis will save memory.
  • 49. #24: Watch LUA memory usage Redis caches LUA scripts, and never flushes them. If you generate scripts dynamically, you will end up wasting a lot of memory. - Don’t generate dynamic scripts, parameterize - script flush to manually clear cache
  • 50. #25: Use rdbtools.com! Take the guesswork out of your memory analysis. Visit https://rdbtools.com