SlideShare a Scribd company logo
The Versatility of Redis
Powering our critical business using Redis
Luciano Sabença
Eiti Kimura
ABOUT US
● Software Engineer at Movile/Wavy
● Developer and architect of distributed systems
● Main interest areas are: Computer Theory, Machine
Learning and Distributed System.
● BS degree with praise in Computer Science by
University of Campinas.
Luciano Sabença
lucianosabenca
ABOUT US
● IT Coordinator and Software Architect at Movile
● Msc. in Electrical Engineering
● Apache Cassandra MVP (2014/2015 and 2015/2016)
● Apache Cassandra Contributor (2015)
● Cassandra Summit Speaker (2014 and 2015)
● Strata Hadoop World Singapore Speaker (2016)
● Spark Summit Speaker (2017)
Eiti Kimura
eitikimura
Movile is the company behind several
apps that makes the life easier
WE MAKE LIFE BETTER
THROUGH OUR APPS
is a Movile company
BALANCE INFORMATION
Account: X3254
Balance: $1,564.20
100852
BALANCE INFORMATION
Account: X3254
Balance: $1,564.20
100852
A deposit of $95.00 was
made to your account on
April 23th at 7:59pm.
Your balance is:
$1,659.20. Text STOP to
cancel or HELP for more
information.
PLAYKIDS
685-63
PLAYKIDS
685-63
You have opted to
subscribe for Play Kids
access fo $14,99/mo.
Reply YES to confirm. Text
STOP to cancel or HELP for
more info.
PLAYKIDS
685-63
You have opted to
subscribe for Play Kids
access fo $14,99/mo.
Reply YES to confirm. Text
STOP to cancel or HELP for
more info.
YES
You successfully
subscribed to PK.
Download the app:
http://bit.ly/pp and enter
this validation code:
528-985-963 to access.
RedisConf18 - The Versatility of Redis - Powering our critical business using Redis
AGENDA
● What/Why Redis?
● 1st Using Redis as Database
● 2nd Traditional Cache Usage
● 3rd A Distributed Lock case
● Lessons Learned / Pros & Cons
WHAT IS REDIS ?
● Redis is what is called a key-value store often
referred to as a NoSQL database
● It can handle concurrency problems very well
● Records can be set to expire after sometime
● It can be used in several different applications
WHY REDIS ?
● Most Redis Operations are atomic!
○ No need for external synchronization!
● Redis Async Replication is fast and makes Redis
reliable!
● All dataset is in memory: fast access!
● Data persistence: Durability!
1st Use Case
How about to try Redis as
a Database?
1st Use Case Intro
Exploring the Redis feature as a datastore we are using
in this case as a distributed database for our blacklist
datalayer service.
60 million phone
numbers
> 1 billion
requests/mo
latency < 5ms
Blacklist Usage by Our Services
Messaging Platform
BLACKLIST DATABASE LAYER
check
PLAY KIDS
685-63
Blacklist Usage by Our Services
Messaging Platform
BLACKLIST DATABASE LAYER
check
PLAY KIDS
You have opted to
subscribe for Play
Kids access fo
$14,99/mo. Reply YES
to confirm.
685-63
Blacklist Usage by Our Services
Billing Platform
BLACKLIST DATABASE LAYER
check
YES
685-63
PLAY KIDS
You have opted to
subscribe for Play
Kids access fo
$14,99/mo. Reply YES
to confirm.
Blacklist Usage by Our Services
Subscription Platform
BLACKLIST DATABASE LAYER
check
YES
You successfully
subscribed to PK.
Enter this validation
code: 528-985-963
685-63
PLAY KIDS
You have opted to
subscribe for Play
Kids access fo
$14,99/mo. Reply YES
to confirm.
Blacklist Usage by Our Services
Subscription Platform Billing Platform Messaging Platform
BLACKLIST DATABASE LAYER
check check check
685-63
YES
You successfully
subscribed to PK.
Enter this validation
code: 528-985-963
685-63
PLAY KIDS
You have opted to
subscribe for Play
Kids access fo
$14,99/mo. Reply YES
to confirm.
Blacklist Usage by Our Services
Web Pages Marketplaces
Broadcast
Applications
Third-Part
Applications
Mobile Apps
Blacklist Usage by Our Services
Subscription Platform Billing Platform Messaging Platform
BLACKLIST DATABASE LAYER
Web Pages Marketplaces
Broadcast
Applications
Third-Part
Applications
Mobile Apps
charge
subscribe
subscribe
charge
send SMS
send message
broadcast
check check check
APPLICATION LAYER
SERVICE LAYER
Old Architecture
Old Architecture
Old Architecture - Failure
● Large amount of memory
being used (~5GB/instance)
● High frequency of GC pauses
● General low performance
noticed
● No more memory available
(scale up limitation)
New Architecture using Redis as Main Database
New Architecture using Redis as Main Database
Redis Advantages for the New Architecture Solution
Configurable
Persistence
Configurable
Replication
Strategies
Fast In-memory
data access
Redis Architecture Setup
Application Application Application Application Application Application
DATACENTER 1 DATACENTER 2
Read data
Write data
Results
● We saved more than 100 GB of memory
● Saved thousands Dollars in memory
● Distributed Data reliability
High message throughput
High scalability
Low costs
Low latency and
business impact
2nd Use Case
Using Redis as a Cache for
Distributed Systems
2nd Use Case Intro
One of the most common Redis usages is as a cache.
A well known case to use in our solution to provide fast
access to our high performance services.
> 160 million
records
essential for the
high throughput of
our platforms
direct impact in
company's
revenue
By the way what is a Grey list concept?
We create a score to our users based on the product it
is subscribed, it intent to classify the user as a good
customer.
Phone Number: + 55 19 0000 1111 Score: 0,8
Phone Number: + 55 19 2323 0101 Score: 0,2
Phone Number: + 55 19 0202 0000 Score: 0,4
Phone Number: + 55 19 4242 9090 Score: 0,1
GREYLIST DATA LAYER
By the way what is a Greylist concept?
Would you like to
receive Sports News
for your location
$2,99/week. Reply
YES to confirm.
685-63
Media Partner
By the way what is a Greylist concept?
Would you like to
receive Sports News
for your location
$2,99/week. Reply
YES to confirm.
YES
685-63
Subscription Platform
Greylist Service Layer
By the way what is a Greylist concept?
Would you like to
receive Sports News
for your location
$2,99/week. Reply
YES to confirm.
YES
685-63
Subscription Platform
Greylist Service Layer
SORRY service
unavailable for your
phone number.
Congratulations now
you are subscribed to
receive Sports News
for your location. Text
STOP to cancel.
YES
SCORE > 0,5NO
By the way what is a Greylist concept?
Would you like to
receive Sports News
for your location
$2,99/week. Reply
YES to confirm.
YES
SORRY service
unavailable for your
phone number.
685-63
Greylist Service Layer
SORRY service
unavailable for your
phone number.
Congratulations now
you are subscribed to
receive Sports News
for your location. Text
STOP to cancel.
YES
SCORE > 0,5NO
By the way what is a Greylist concept?
Would you like to
receive Sports News
for your location
$2,99/week. Reply
YES to confirm.
YES
SORRY service
unavailable for your
phone number.
685-63
Subscription Platform
Greylist Service Layer
SORRY service
unavailable for your
phone number.
Media Partner
Congratulations now
you are subscribed to
receive Sports News
for your location. Text
STOP to cancel.
YES
NO SCORE > 0,5
Solution Architecture
Redis In the Solution
● We disabled Redis persistence and reached write rate over
300k/second.
● Peak of 400k writes/second
● Redis response is below ms.
● We have no impact in our subscription flow
Results
● Blocked more than 1MM subscriptions from bad
payers users.
● Saved around U$60k in media in a few months.
● Block Rate > 40% for some campaigns.
3rd Use Case
Have you tried to orchestrate a
distributed system?
3rd Use Case Intro
Exploring Redis as an orchestrator for our distributed
systems managing SMPP connections
Stateful
connection with
limited number of
binds
A connection can
be consumed by
more than one
instance
We should be
capable of load
balancing the
connections
Normal Scenario
Platform Instance
Platform Instance
Mobile Carrier
SMPP / BIND 1
SMPP / BIND 2
WAVY DATACENTERS INTERNET
Failure Scenario
Platform Instance
Platform Instance
Mobile Carrier
SMPP / BIND 1
SMPP / BIND 2
WAVY DATACENTERS INTERNET
Expected Scenario
Platform Instance
Platform Instance
Mobile Carrier
SMPP / BIND 1
SMPP / BIND 2
WAVY DATACENTERS INTERNET
Solution Scenario
Platform Instance
Platform Instance
Mobile Carrier
SMPP / BIND 1
SMPP / BIND 2
WAVY DATACENTERS INTERNET
Components trying to acquire a lock
Set The Lock:
SET resource_lock_name some_random_value NX PX <expiration>
How to set/release the Lock?
Set The Lock:
SET resource_lock_name some_random_value NX PX <expiration>
Release The Lock:
if redis.call("get",KEYS[1]) == ARGV[1] then
return redis.call("del",KEYS[1])
else
return 0
end
Source and More Information: https://redis.io/topics/distlock
How to set/release the Lock?
Redis Strengths for the Solution
Atomic Operations Powerful Data Structures
Results
Low downtime for
a connection
Increase reliability for
our integrations
Increase resilience to
failures in our systems
Lessons Learned
1st Use case Lessons Learned
● Lack of multi DC support:
○ We replicate all writes which might cause
consistency problems
Application
DATACENTER 1 DATACENTER 2
2nd Use case Lessons Learned
● Persistence Might Slow Down Redis
Main Memory Access Send 2KB Over Commodity
Network
Read Sequentially 1MB from
SSD
100 ns 900 ns 400 µs
Source: https://people.eecs.berkeley.edu/~rcs/research/interactive_latency.html
Date reference 2010-2012
Architecture Setup Mistake (do not do this)
Master
KeepAliveD
Slave
KeepAliveD
Virtual IP
Platform
Instance
writes
Master Failure Expected Scenario
Down
KeepAliveD
Slave
KeepAliveD
Virtual IP
Platform
Instance
writes
Master
KeepAliveD
Slave
KeepAliveD
Virtual IP
Platform
Instance
writes
Failure Expected Scenario
Slave
KeepAliveD
Master
KeepAliveD
Virtual IP
Platform
Instance
writes
What really happened
Master
KeepAliveD
Slave
KeepAliveD
Virtual IP
Platform
Instance
writes
● The old master goes up
assuming as master and
synchronizing old
information
● We lost a huge amount data
● Causing overall
consistencies problems
Lesson Learned: Don't try to use
home-made solutions. Use Redis
Sentinel instead.
3rd Use case Lessons Learned
● Single point of failure
Platform
Platform
Platform
3rd Use case Lessons Learned
● Single point of failure
Platform
Platform
Platform
Delay
Slave
Solutions for These Problems: Use a more complex lock algorithm (e.g. RedLock)
● Not very suitable for high throughput locks
Platform
Platform
Platform
Thank you!
Questions?
luciano.sabenca@wavy.global
eiti.kimura@wavy.global
linkedin.com/in/eitikimura
linkedin.com/in/luciano-sabenca

More Related Content

What's hot (20)

PPTX
Scaling HDFS at Xiaomi
DataWorks Summit
 
PDF
RedisConf18 - Making Real-Time Predictive Decisions with Redis
Redis Labs
 
PDF
RedisConf18 - Scaling Whitepages With Redison Flash
Redis Labs
 
PPTX
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
DataWorks Summit
 
PPTX
Redis TimeSeries
Redis Labs
 
PPTX
Redis Reliability, Performance & Innovation
Redis Labs
 
PPTX
RedisConf18 - Re-architecting Redis-on-Flash with Intel 3DX Point™ Memory
Redis Labs
 
PDF
RedisConf18 - Auto-Scaling Redis Caches - Observability, Efficiency & Perform...
Redis Labs
 
PPTX
Webinar | Building Apps with the Cassandra Python Driver
DataStax Academy
 
PDF
Redis Tames The Caching Herd: Jon Hyman
Redis Labs
 
PPTX
Project Ouroboros: Using StreamSets Data Collector to Help Manage the StreamS...
Pat Patterson
 
PPTX
Building Continuously Curated Ingestion Pipelines
Arvind Prabhakar
 
PDF
Thoughts on kafka capacity planning
JamieAlquiza
 
PPTX
HBaseConAsia2018 Track2-3: Bringing MySQL Compatibility to HBase using Databa...
Michael Stack
 
PPTX
Data Streaming with Apache Kafka & MongoDB - EMEA
Andrew Morgan
 
PDF
Big Data Tools in AWS
Shu-Jeng Hsieh
 
PPTX
Extending Twitter's Data Platform to Google Cloud
DataWorks Summit
 
PPTX
MongoDB 3.4 webinar
Andrew Morgan
 
PPTX
In Flux Limiting for a multi-tenant logging service
DataWorks Summit/Hadoop Summit
 
PDF
HBaseConAsia2018 Track2-6: Scaling 30TB's of data lake with Apache HBase and ...
Michael Stack
 
Scaling HDFS at Xiaomi
DataWorks Summit
 
RedisConf18 - Making Real-Time Predictive Decisions with Redis
Redis Labs
 
RedisConf18 - Scaling Whitepages With Redison Flash
Redis Labs
 
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
DataWorks Summit
 
Redis TimeSeries
Redis Labs
 
Redis Reliability, Performance & Innovation
Redis Labs
 
RedisConf18 - Re-architecting Redis-on-Flash with Intel 3DX Point™ Memory
Redis Labs
 
RedisConf18 - Auto-Scaling Redis Caches - Observability, Efficiency & Perform...
Redis Labs
 
Webinar | Building Apps with the Cassandra Python Driver
DataStax Academy
 
Redis Tames The Caching Herd: Jon Hyman
Redis Labs
 
Project Ouroboros: Using StreamSets Data Collector to Help Manage the StreamS...
Pat Patterson
 
Building Continuously Curated Ingestion Pipelines
Arvind Prabhakar
 
Thoughts on kafka capacity planning
JamieAlquiza
 
HBaseConAsia2018 Track2-3: Bringing MySQL Compatibility to HBase using Databa...
Michael Stack
 
Data Streaming with Apache Kafka & MongoDB - EMEA
Andrew Morgan
 
Big Data Tools in AWS
Shu-Jeng Hsieh
 
Extending Twitter's Data Platform to Google Cloud
DataWorks Summit
 
MongoDB 3.4 webinar
Andrew Morgan
 
In Flux Limiting for a multi-tenant logging service
DataWorks Summit/Hadoop Summit
 
HBaseConAsia2018 Track2-6: Scaling 30TB's of data lake with Apache HBase and ...
Michael Stack
 

Similar to RedisConf18 - The Versatility of Redis - Powering our critical business using Redis (20)

PDF
Redis as a Cache Boosting Performance and Scalability
Inexture Solutions
 
PPT
New York REDIS Meetup Welcome Session
Aleksandr Yampolskiy
 
PDF
Redis Everywhere - Sunshine PHP
Ricard Clau
 
PPTX
Add Redis to Postgres to Make Your Microservices Go Boom!
Dave Nielsen
 
PDF
Redis everywhere - PHP London
Ricard Clau
 
PPTX
Redis meetup
Nikhil Dole
 
PPTX
Microservices - Is it time to breakup?
Dave Nielsen
 
PPTX
Introduction to Redis and its features.pptx
Knoldus Inc.
 
PPTX
Introduction to Redis
TO THE NEW | Technology
 
KEY
Redis overview for Software Architecture Forum
Christopher Spring
 
PDF
Mini-Training: Redis
Betclic Everest Group Tech Team
 
PDF
Redis - The Universal NoSQL Tool
Eberhard Wolff
 
PPTX
What's new with enterprise Redis - Leena Joshi, Redis Labs
Redis Labs
 
PPTX
Redis Labcamp
Angelo Simone Scotto
 
PPTX
Get more than a cache back! - ConFoo Montreal
Maarten Balliauw
 
PDF
Speed up your Symfony2 application and build awesome features with Redis
Ricard Clau
 
PPTX
Redis and it's data types
Aniruddha Chakrabarti
 
PDF
PostgreSQL and Redis - talk at pgcon 2013
Andrew Dunstan
 
PPTX
Big Data Day LA 2016/ NoSQL track - Analytics at the Speed of Light with Redi...
Data Con LA
 
PPTX
Redis Labs and SQL Server
Lynn Langit
 
Redis as a Cache Boosting Performance and Scalability
Inexture Solutions
 
New York REDIS Meetup Welcome Session
Aleksandr Yampolskiy
 
Redis Everywhere - Sunshine PHP
Ricard Clau
 
Add Redis to Postgres to Make Your Microservices Go Boom!
Dave Nielsen
 
Redis everywhere - PHP London
Ricard Clau
 
Redis meetup
Nikhil Dole
 
Microservices - Is it time to breakup?
Dave Nielsen
 
Introduction to Redis and its features.pptx
Knoldus Inc.
 
Introduction to Redis
TO THE NEW | Technology
 
Redis overview for Software Architecture Forum
Christopher Spring
 
Mini-Training: Redis
Betclic Everest Group Tech Team
 
Redis - The Universal NoSQL Tool
Eberhard Wolff
 
What's new with enterprise Redis - Leena Joshi, Redis Labs
Redis Labs
 
Redis Labcamp
Angelo Simone Scotto
 
Get more than a cache back! - ConFoo Montreal
Maarten Balliauw
 
Speed up your Symfony2 application and build awesome features with Redis
Ricard Clau
 
Redis and it's data types
Aniruddha Chakrabarti
 
PostgreSQL and Redis - talk at pgcon 2013
Andrew Dunstan
 
Big Data Day LA 2016/ NoSQL track - Analytics at the Speed of Light with Redi...
Data Con LA
 
Redis Labs and SQL Server
Lynn Langit
 
Ad

More from Redis Labs (20)

PPTX
Redis Day Bangalore 2020 - Session state caching with redis
Redis Labs
 
PPTX
Protecting Your API with Redis by Jane Paek - Redis Day Seattle 2020
Redis Labs
 
PPTX
The Happy Marriage of Redis and Protobuf by Scott Haines of Twilio - Redis Da...
Redis Labs
 
PPTX
SQL, Redis and Kubernetes by Paul Stanton of Windocks - Redis Day Seattle 2020
Redis Labs
 
PPTX
Rust and Redis - Solving Problems for Kubernetes by Ravi Jagannathan of VMwar...
Redis Labs
 
PPTX
Redis for Data Science and Engineering by Dmitry Polyakovsky of Oracle
Redis Labs
 
PPTX
Practical Use Cases for ACLs in Redis 6 by Jamie Scott - Redis Day Seattle 2020
Redis Labs
 
PPTX
Moving Beyond Cache by Yiftach Shoolman Redis Labs - Redis Day Seattle 2020
Redis Labs
 
PPTX
Leveraging Redis for System Monitoring by Adam McCormick of SBG - Redis Day S...
Redis Labs
 
PPTX
JSON in Redis - When to use RedisJSON by Jay Won of Coupang - Redis Day Seatt...
Redis Labs
 
PPTX
Highly Available Persistent Session Management Service by Mohamed Elmergawi o...
Redis Labs
 
PPTX
Anatomy of a Redis Command by Madelyn Olson of Amazon Web Services - Redis Da...
Redis Labs
 
PPTX
Building a Multi-dimensional Analytics Engine with RedisGraph by Matthew Goos...
Redis Labs
 
PPTX
RediSearch 1.6 by Pieter Cailliau - Redis Day Bangalore 2020
Redis Labs
 
PPTX
RedisGraph 2.0 by Pieter Cailliau - Redis Day Bangalore 2020
Redis Labs
 
PPTX
RedisTimeSeries 1.2 by Pieter Cailliau - Redis Day Bangalore 2020
Redis Labs
 
PPTX
RedisAI 0.9 by Sherin Thomas of Tensorwerk - Redis Day Bangalore 2020
Redis Labs
 
PPTX
Rate-Limiting 30 Million requests by Vijay Lakshminarayanan and Girish Koundi...
Redis Labs
 
PDF
Three Pillars of Observability by Rajalakshmi Raji Srinivasan of Site24x7 Zoh...
Redis Labs
 
PPTX
Solving Complex Scaling Problems by Prashant Kumar and Abhishek Jain of Myntr...
Redis Labs
 
Redis Day Bangalore 2020 - Session state caching with redis
Redis Labs
 
Protecting Your API with Redis by Jane Paek - Redis Day Seattle 2020
Redis Labs
 
The Happy Marriage of Redis and Protobuf by Scott Haines of Twilio - Redis Da...
Redis Labs
 
SQL, Redis and Kubernetes by Paul Stanton of Windocks - Redis Day Seattle 2020
Redis Labs
 
Rust and Redis - Solving Problems for Kubernetes by Ravi Jagannathan of VMwar...
Redis Labs
 
Redis for Data Science and Engineering by Dmitry Polyakovsky of Oracle
Redis Labs
 
Practical Use Cases for ACLs in Redis 6 by Jamie Scott - Redis Day Seattle 2020
Redis Labs
 
Moving Beyond Cache by Yiftach Shoolman Redis Labs - Redis Day Seattle 2020
Redis Labs
 
Leveraging Redis for System Monitoring by Adam McCormick of SBG - Redis Day S...
Redis Labs
 
JSON in Redis - When to use RedisJSON by Jay Won of Coupang - Redis Day Seatt...
Redis Labs
 
Highly Available Persistent Session Management Service by Mohamed Elmergawi o...
Redis Labs
 
Anatomy of a Redis Command by Madelyn Olson of Amazon Web Services - Redis Da...
Redis Labs
 
Building a Multi-dimensional Analytics Engine with RedisGraph by Matthew Goos...
Redis Labs
 
RediSearch 1.6 by Pieter Cailliau - Redis Day Bangalore 2020
Redis Labs
 
RedisGraph 2.0 by Pieter Cailliau - Redis Day Bangalore 2020
Redis Labs
 
RedisTimeSeries 1.2 by Pieter Cailliau - Redis Day Bangalore 2020
Redis Labs
 
RedisAI 0.9 by Sherin Thomas of Tensorwerk - Redis Day Bangalore 2020
Redis Labs
 
Rate-Limiting 30 Million requests by Vijay Lakshminarayanan and Girish Koundi...
Redis Labs
 
Three Pillars of Observability by Rajalakshmi Raji Srinivasan of Site24x7 Zoh...
Redis Labs
 
Solving Complex Scaling Problems by Prashant Kumar and Abhishek Jain of Myntr...
Redis Labs
 
Ad

Recently uploaded (20)

PDF
Mastering Financial Management in Direct Selling
Epixel MLM Software
 
PDF
Smart Trailers 2025 Update with History and Overview
Paul Menig
 
PDF
Exolore The Essential AI Tools in 2025.pdf
Srinivasan M
 
PPTX
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
PDF
Building Real-Time Digital Twins with IBM Maximo & ArcGIS Indoors
Safe Software
 
PDF
"AI Transformation: Directions and Challenges", Pavlo Shaternik
Fwdays
 
PPTX
AI Penetration Testing Essentials: A Cybersecurity Guide for 2025
defencerabbit Team
 
PDF
Agentic AI lifecycle for Enterprise Hyper-Automation
Debmalya Biswas
 
PDF
Newgen Beyond Frankenstein_Build vs Buy_Digital_version.pdf
darshakparmar
 
PDF
“NPU IP Hardware Shaped Through Software and Use-case Analysis,” a Presentati...
Edge AI and Vision Alliance
 
PDF
Using FME to Develop Self-Service CAD Applications for a Major UK Police Force
Safe Software
 
PPTX
AUTOMATION AND ROBOTICS IN PHARMA INDUSTRY.pptx
sameeraaabegumm
 
PDF
DevBcn - Building 10x Organizations Using Modern Productivity Metrics
Justin Reock
 
PDF
Biography of Daniel Podor.pdf
Daniel Podor
 
PPTX
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
PDF
"Beyond English: Navigating the Challenges of Building a Ukrainian-language R...
Fwdays
 
PDF
CIFDAQ Token Spotlight for 9th July 2025
CIFDAQ
 
PDF
Go Concurrency Real-World Patterns, Pitfalls, and Playground Battles.pdf
Emily Achieng
 
PPTX
"Autonomy of LLM Agents: Current State and Future Prospects", Oles` Petriv
Fwdays
 
PDF
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
Mastering Financial Management in Direct Selling
Epixel MLM Software
 
Smart Trailers 2025 Update with History and Overview
Paul Menig
 
Exolore The Essential AI Tools in 2025.pdf
Srinivasan M
 
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
Building Real-Time Digital Twins with IBM Maximo & ArcGIS Indoors
Safe Software
 
"AI Transformation: Directions and Challenges", Pavlo Shaternik
Fwdays
 
AI Penetration Testing Essentials: A Cybersecurity Guide for 2025
defencerabbit Team
 
Agentic AI lifecycle for Enterprise Hyper-Automation
Debmalya Biswas
 
Newgen Beyond Frankenstein_Build vs Buy_Digital_version.pdf
darshakparmar
 
“NPU IP Hardware Shaped Through Software and Use-case Analysis,” a Presentati...
Edge AI and Vision Alliance
 
Using FME to Develop Self-Service CAD Applications for a Major UK Police Force
Safe Software
 
AUTOMATION AND ROBOTICS IN PHARMA INDUSTRY.pptx
sameeraaabegumm
 
DevBcn - Building 10x Organizations Using Modern Productivity Metrics
Justin Reock
 
Biography of Daniel Podor.pdf
Daniel Podor
 
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
"Beyond English: Navigating the Challenges of Building a Ukrainian-language R...
Fwdays
 
CIFDAQ Token Spotlight for 9th July 2025
CIFDAQ
 
Go Concurrency Real-World Patterns, Pitfalls, and Playground Battles.pdf
Emily Achieng
 
"Autonomy of LLM Agents: Current State and Future Prospects", Oles` Petriv
Fwdays
 
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 

RedisConf18 - The Versatility of Redis - Powering our critical business using Redis

  • 1. The Versatility of Redis Powering our critical business using Redis Luciano Sabença Eiti Kimura
  • 2. ABOUT US ● Software Engineer at Movile/Wavy ● Developer and architect of distributed systems ● Main interest areas are: Computer Theory, Machine Learning and Distributed System. ● BS degree with praise in Computer Science by University of Campinas. Luciano Sabença lucianosabenca
  • 3. ABOUT US ● IT Coordinator and Software Architect at Movile ● Msc. in Electrical Engineering ● Apache Cassandra MVP (2014/2015 and 2015/2016) ● Apache Cassandra Contributor (2015) ● Cassandra Summit Speaker (2014 and 2015) ● Strata Hadoop World Singapore Speaker (2016) ● Spark Summit Speaker (2017) Eiti Kimura eitikimura
  • 4. Movile is the company behind several apps that makes the life easier WE MAKE LIFE BETTER THROUGH OUR APPS
  • 5. is a Movile company
  • 7. BALANCE INFORMATION Account: X3254 Balance: $1,564.20 100852 A deposit of $95.00 was made to your account on April 23th at 7:59pm. Your balance is: $1,659.20. Text STOP to cancel or HELP for more information.
  • 9. PLAYKIDS 685-63 You have opted to subscribe for Play Kids access fo $14,99/mo. Reply YES to confirm. Text STOP to cancel or HELP for more info.
  • 10. PLAYKIDS 685-63 You have opted to subscribe for Play Kids access fo $14,99/mo. Reply YES to confirm. Text STOP to cancel or HELP for more info. YES You successfully subscribed to PK. Download the app: http://bit.ly/pp and enter this validation code: 528-985-963 to access.
  • 12. AGENDA ● What/Why Redis? ● 1st Using Redis as Database ● 2nd Traditional Cache Usage ● 3rd A Distributed Lock case ● Lessons Learned / Pros & Cons
  • 13. WHAT IS REDIS ? ● Redis is what is called a key-value store often referred to as a NoSQL database ● It can handle concurrency problems very well ● Records can be set to expire after sometime ● It can be used in several different applications
  • 14. WHY REDIS ? ● Most Redis Operations are atomic! ○ No need for external synchronization! ● Redis Async Replication is fast and makes Redis reliable! ● All dataset is in memory: fast access! ● Data persistence: Durability!
  • 15. 1st Use Case How about to try Redis as a Database?
  • 16. 1st Use Case Intro Exploring the Redis feature as a datastore we are using in this case as a distributed database for our blacklist datalayer service. 60 million phone numbers > 1 billion requests/mo latency < 5ms
  • 17. Blacklist Usage by Our Services Messaging Platform BLACKLIST DATABASE LAYER check PLAY KIDS 685-63
  • 18. Blacklist Usage by Our Services Messaging Platform BLACKLIST DATABASE LAYER check PLAY KIDS You have opted to subscribe for Play Kids access fo $14,99/mo. Reply YES to confirm. 685-63
  • 19. Blacklist Usage by Our Services Billing Platform BLACKLIST DATABASE LAYER check YES 685-63 PLAY KIDS You have opted to subscribe for Play Kids access fo $14,99/mo. Reply YES to confirm.
  • 20. Blacklist Usage by Our Services Subscription Platform BLACKLIST DATABASE LAYER check YES You successfully subscribed to PK. Enter this validation code: 528-985-963 685-63 PLAY KIDS You have opted to subscribe for Play Kids access fo $14,99/mo. Reply YES to confirm.
  • 21. Blacklist Usage by Our Services Subscription Platform Billing Platform Messaging Platform BLACKLIST DATABASE LAYER check check check 685-63 YES You successfully subscribed to PK. Enter this validation code: 528-985-963 685-63 PLAY KIDS You have opted to subscribe for Play Kids access fo $14,99/mo. Reply YES to confirm.
  • 22. Blacklist Usage by Our Services Web Pages Marketplaces Broadcast Applications Third-Part Applications Mobile Apps
  • 23. Blacklist Usage by Our Services Subscription Platform Billing Platform Messaging Platform BLACKLIST DATABASE LAYER Web Pages Marketplaces Broadcast Applications Third-Part Applications Mobile Apps charge subscribe subscribe charge send SMS send message broadcast check check check APPLICATION LAYER SERVICE LAYER
  • 26. Old Architecture - Failure ● Large amount of memory being used (~5GB/instance) ● High frequency of GC pauses ● General low performance noticed ● No more memory available (scale up limitation)
  • 27. New Architecture using Redis as Main Database
  • 28. New Architecture using Redis as Main Database
  • 29. Redis Advantages for the New Architecture Solution Configurable Persistence Configurable Replication Strategies Fast In-memory data access
  • 30. Redis Architecture Setup Application Application Application Application Application Application DATACENTER 1 DATACENTER 2 Read data Write data
  • 31. Results ● We saved more than 100 GB of memory ● Saved thousands Dollars in memory ● Distributed Data reliability High message throughput High scalability Low costs Low latency and business impact
  • 32. 2nd Use Case Using Redis as a Cache for Distributed Systems
  • 33. 2nd Use Case Intro One of the most common Redis usages is as a cache. A well known case to use in our solution to provide fast access to our high performance services. > 160 million records essential for the high throughput of our platforms direct impact in company's revenue
  • 34. By the way what is a Grey list concept? We create a score to our users based on the product it is subscribed, it intent to classify the user as a good customer. Phone Number: + 55 19 0000 1111 Score: 0,8 Phone Number: + 55 19 2323 0101 Score: 0,2 Phone Number: + 55 19 0202 0000 Score: 0,4 Phone Number: + 55 19 4242 9090 Score: 0,1 GREYLIST DATA LAYER
  • 35. By the way what is a Greylist concept? Would you like to receive Sports News for your location $2,99/week. Reply YES to confirm. 685-63 Media Partner
  • 36. By the way what is a Greylist concept? Would you like to receive Sports News for your location $2,99/week. Reply YES to confirm. YES 685-63 Subscription Platform Greylist Service Layer
  • 37. By the way what is a Greylist concept? Would you like to receive Sports News for your location $2,99/week. Reply YES to confirm. YES 685-63 Subscription Platform Greylist Service Layer SORRY service unavailable for your phone number. Congratulations now you are subscribed to receive Sports News for your location. Text STOP to cancel. YES SCORE > 0,5NO
  • 38. By the way what is a Greylist concept? Would you like to receive Sports News for your location $2,99/week. Reply YES to confirm. YES SORRY service unavailable for your phone number. 685-63 Greylist Service Layer SORRY service unavailable for your phone number. Congratulations now you are subscribed to receive Sports News for your location. Text STOP to cancel. YES SCORE > 0,5NO
  • 39. By the way what is a Greylist concept? Would you like to receive Sports News for your location $2,99/week. Reply YES to confirm. YES SORRY service unavailable for your phone number. 685-63 Subscription Platform Greylist Service Layer SORRY service unavailable for your phone number. Media Partner Congratulations now you are subscribed to receive Sports News for your location. Text STOP to cancel. YES NO SCORE > 0,5
  • 41. Redis In the Solution ● We disabled Redis persistence and reached write rate over 300k/second. ● Peak of 400k writes/second ● Redis response is below ms. ● We have no impact in our subscription flow
  • 42. Results ● Blocked more than 1MM subscriptions from bad payers users. ● Saved around U$60k in media in a few months. ● Block Rate > 40% for some campaigns.
  • 43. 3rd Use Case Have you tried to orchestrate a distributed system?
  • 44. 3rd Use Case Intro Exploring Redis as an orchestrator for our distributed systems managing SMPP connections Stateful connection with limited number of binds A connection can be consumed by more than one instance We should be capable of load balancing the connections
  • 45. Normal Scenario Platform Instance Platform Instance Mobile Carrier SMPP / BIND 1 SMPP / BIND 2 WAVY DATACENTERS INTERNET
  • 46. Failure Scenario Platform Instance Platform Instance Mobile Carrier SMPP / BIND 1 SMPP / BIND 2 WAVY DATACENTERS INTERNET
  • 47. Expected Scenario Platform Instance Platform Instance Mobile Carrier SMPP / BIND 1 SMPP / BIND 2 WAVY DATACENTERS INTERNET
  • 48. Solution Scenario Platform Instance Platform Instance Mobile Carrier SMPP / BIND 1 SMPP / BIND 2 WAVY DATACENTERS INTERNET
  • 49. Components trying to acquire a lock
  • 50. Set The Lock: SET resource_lock_name some_random_value NX PX <expiration> How to set/release the Lock?
  • 51. Set The Lock: SET resource_lock_name some_random_value NX PX <expiration> Release The Lock: if redis.call("get",KEYS[1]) == ARGV[1] then return redis.call("del",KEYS[1]) else return 0 end Source and More Information: https://redis.io/topics/distlock How to set/release the Lock?
  • 52. Redis Strengths for the Solution Atomic Operations Powerful Data Structures
  • 53. Results Low downtime for a connection Increase reliability for our integrations Increase resilience to failures in our systems
  • 55. 1st Use case Lessons Learned ● Lack of multi DC support: ○ We replicate all writes which might cause consistency problems Application DATACENTER 1 DATACENTER 2
  • 56. 2nd Use case Lessons Learned ● Persistence Might Slow Down Redis Main Memory Access Send 2KB Over Commodity Network Read Sequentially 1MB from SSD 100 ns 900 ns 400 µs Source: https://people.eecs.berkeley.edu/~rcs/research/interactive_latency.html Date reference 2010-2012
  • 57. Architecture Setup Mistake (do not do this) Master KeepAliveD Slave KeepAliveD Virtual IP Platform Instance writes
  • 58. Master Failure Expected Scenario Down KeepAliveD Slave KeepAliveD Virtual IP Platform Instance writes Master KeepAliveD Slave KeepAliveD Virtual IP Platform Instance writes
  • 60. What really happened Master KeepAliveD Slave KeepAliveD Virtual IP Platform Instance writes ● The old master goes up assuming as master and synchronizing old information ● We lost a huge amount data ● Causing overall consistencies problems Lesson Learned: Don't try to use home-made solutions. Use Redis Sentinel instead.
  • 61. 3rd Use case Lessons Learned ● Single point of failure Platform Platform Platform
  • 62. 3rd Use case Lessons Learned ● Single point of failure Platform Platform Platform Delay Slave Solutions for These Problems: Use a more complex lock algorithm (e.g. RedLock) ● Not very suitable for high throughput locks Platform Platform Platform