RedisConf17 - Lyft - Geospatial at Scale - Daniel Hochman
4 likes•2,818 views
This document discusses Lyft's migration from a monolithic architecture to using Redis to power a geospatial indexing system at scale. It describes Lyft's original architecture, issues they faced, and how they iterated on their data model and use of Redis over time. It also discusses how Lyft uses Redis across their platform, including operations, monitoring, capacity planning, and their work contributing to open source projects like Envoy.
![API
Monolith
PUT /v1/locations
5 sec
loop
{
"lat": 37.61,
"lng": -122.38
}
{
"drivers": [ ... ],
"primetime": 0,
"eta": 30,
...
}
200 OK
Index on users collection
(driver_mode, region)
Lyft backend in 2012
Location stored on user record](https://image.slidesharecdn.com/lyft-redisconf-2017-dhochman-geospatialatscale-170619223814/85/RedisConf17-Lyft-Geospatial-at-Scale-Daniel-Hochman-3-320.jpg)
![Monolithic architecture issues
Global write lock, not shardable, difficult refactoring, region boundary issues
drivers_in_region = db.users.find(
driver_mode: {$eq: True},
region: {$eq: "SFO"},
ride_id: {$eq: None}
)
eligible_drivers = sort_and_filter_by_distance(
drivers_in_region, radius=.5
)
dispatch(eligible_drivers[0])](https://image.slidesharecdn.com/lyft-redisconf-2017-dhochman-geospatialatscale-170619223814/85/RedisConf17-Lyft-Geospatial-at-Scale-Daniel-Hochman-4-320.jpg)
![loc = {'lat': 23.2, 'lng': -122.3, 'ride_id': None}
geohash = compute_geohash(loc['lat'], loc['lng'], level=5)
with nutcracker.pipeline() as pipeline:
# string per user
pipeline.set(user_id, json.dumps(loc))
# sorted set per geohash with last timestamp
if driver_mode is True:
pipeline.zset(geohash, user_id, now_seconds())
pipeline.zremrangebyscore(geohash, -inf, now_seconds() - 30) # expire!
Data model with geohashing
Sorted set tells you where a driver might be. Use string as source of truth.
On query, look in neighboring geohashes based on desired radius.](https://image.slidesharecdn.com/lyft-redisconf-2017-dhochman-geospatialatscale-170619223814/85/RedisConf17-Lyft-Geospatial-at-Scale-Daniel-Hochman-14-320.jpg)
![Envoy Redis Roadmap
Additional command support
Error handling
Pipeline management features
Performance optimization
Replication
- Failover
- Mitigate hot read shard issues with large objects
- Zone local query routing
- Quorum
- Protocol overloading? e.g. SET key value [replication factor] [timeout]
More!](https://image.slidesharecdn.com/lyft-redisconf-2017-dhochman-geospatialatscale-170619223814/85/RedisConf17-Lyft-Geospatial-at-Scale-Daniel-Hochman-25-320.jpg)
RedisConf17 - Lyft - Geospatial at Scale - Daniel Hochman
- 1. Daniel Hochman, Engineer @ Geospatial Indexing at Scale The 10 15 Million QPS Redis Architecture Powering Lyft
- 2. Agenda Case Study: scaling a geospatial index - Original Lyft architecture - Migrating to Redis - Iterating on data model Redis on the Lyft platform - Service integration - Operations and monitoring - Capacity planning - Open source work and roadmap Q&A
- 3. API Monolith PUT /v1/locations 5 sec loop { "lat": 37.61, "lng": -122.38 } { "drivers": [ ... ], "primetime": 0, "eta": 30, ... } 200 OK Index on users collection (driver_mode, region) Lyft backend in 2012 Location stored on user record
- 4. Monolithic architecture issues Global write lock, not shardable, difficult refactoring, region boundary issues drivers_in_region = db.users.find( driver_mode: {$eq: True}, region: {$eq: "SFO"}, ride_id: {$eq: None} ) eligible_drivers = sort_and_filter_by_distance( drivers_in_region, radius=.5 ) dispatch(eligible_drivers[0])
- 5. Horizontally scalable and highly available from day zero Unwritten architecture rule at Lyft
- 6. nutcracker locations_cluster: listen: locations.sock distribution: ketama hash: md5 eject_hosts: true failure_limit: 3 servers: - 10.0.0.1:6379:1 - 10.0.0.2:6379:1 ... - 10.0.0.255:6379:1 Redis Cluster . . . n shards Application Instance Nutcracker overview Ketama provides consistent hashing! Lose a node, only lose 1/n data
- 7. nutcracker Pipelining PIPELINE ( HGETALL foo SET hello world INCR lyfts ) RESPONSE ( (k1, v1, k2, v2) OK 12121986 ) md5(foo) % 3 = 2 md5(hello) % 3 = 0 md5(lyfts) % 3 = 2 0: SET hello world 0 1 2 2: PIPELINE ( HGETALL foo INCR lyfts ) return ordered_results 1. hash the keys 2. send concurrent requests to backends 3. concatenate and return results
- 8. location = json.dumps({'lat': 23.2, 'lng': -122.3, 'ride_id': None}) with nutcracker.pipeline() as pipeline: pipeline.set(user_id, location) # string per user if driver_mode is True: pipeline.hset(region_name, user_id, location) # hash per region Parity data model Fetching inactive drivers when doing HGETALL. Network and serialization overhead. The obvious fix? Expiration.
- 10. - pipeline.hset(region_name, user_id, location) # hash per region # hash per region per 30 seconds bucket = now_seconds() - (now_seconds() % 30) hash_key = '{}_{}'.format(region_name, bucket) pipeline.hset(hash_key, user_id, location) pipeline.expire(hash_key, 15) Expiring a hash HGETALL current bucket plus next and previous to handle clock drift and boundary condition, merge in process. 12:00:00 12:00:30 12:01:00 12:01:30 ...
- 11. Region is a poor hash key (hot shards) Bulk expiration blocks Redis for longer than expected Redis used for inter-service communication with new dispatch system Growing pains Let's fix it!
- 12. Redis is used for inter-service communication - Replicate existing queries and writes in new service - Replace function calls that query or write to Redis with calls to new service - With contract in place between existing services and new service, refactor data model - Migrate relevant business logic Proper service communication
- 13. fd a Region is a poor hash key Geohashing is an algorithm that provides arbitrary precision with gradual precision degradation Geohashing b c d e f ihg >>> compute_geohash(lat=37.7852, lng=-122.4044, level=9) 9q8yywefd fa fb fc fe ff fifhfg Google open-sourced an alternative geohashing algorithm, S2
- 14. loc = {'lat': 23.2, 'lng': -122.3, 'ride_id': None} geohash = compute_geohash(loc['lat'], loc['lng'], level=5) with nutcracker.pipeline() as pipeline: # string per user pipeline.set(user_id, json.dumps(loc)) # sorted set per geohash with last timestamp if driver_mode is True: pipeline.zset(geohash, user_id, now_seconds()) pipeline.zremrangebyscore(geohash, -inf, now_seconds() - 30) # expire! Data model with geohashing Sorted set tells you where a driver might be. Use string as source of truth. On query, look in neighboring geohashes based on desired radius.
- 15. Why not GEO? Stable release in May 2016 with Redis 3.2.0 Point-in-radius, position of key Uses geohashing and a sorted set under-the-hood No expiration or sharding No metadata storage Great for prototyping Much more data model complexity behind the scenes - Additional metadata - Writing to multiple indices to lower cost of high frequency queries - Balancing scattered gets and hot shards with geohash level GEOADD GEODIST GEOHASH GEOPOS GEORADIUS GEORADIUSBYMEMBER
- 16. Redis on the Lyft platform
- 17. Creating a new cluster is a self-service process that takes less than one engineer hour. 2015: 1 cluster of 3 instances 2017: 50 clusters with a total of 750 instances Cluster creation
- 18. Golang and Python are the two officially supported backend languages at Lyft Python features - Fully compatible with redis-py StrictRedis - Stats - Retry - Pipeline management for interleave and targeted retry from lyftredis import NutcrackerClient redis_client = NutcrackerClient('locations') Internal libraries
- 19. {% macro redis_cluster_stats(redis_cluster_name, alarm_thresholds) -%} Observability
- 20. Capacity planning Combine APIs and stats for global capacity plan - Difficult to track 50 clusters - Google Sheets API for display - Internal stats for actual usage, EC2 API for capacity - Automatically determine resource constraint (CPU, memory, network) - Currently aim for 4x headroom due to difficulty of cluster resize - At-a-glance view of peak resource consumption, provisioned resources, cost, resource constraint
- 21. Object serialization Benefits - Lower memory consumption, I/O - Lower network I/O - Lower serialization cost to CPU 708 bytes 69% 1012 bytes (original) 190 bytes 18%
- 22. Nutcracker issues - Deprecated internally at Twitter and unmaintained - Passive health checks - No hot restart (config changes cause downtime) - Difficult to extend (e.g. to integrate with service discovery) - When EC2 instance of Redis dies, we page an engineer to fix the problem The road ahead
- 23. Open-source C++11 service mesh and edge proxy As an advanced load balancer, provides: - Service discovery integration - Retry, circuit breaking, rate limiting - Consistent hashing - Active health checks - Stats, stats, stats - Tracing - Outlier detection, fault injection Envoy was designed to be extensible! What is Envoy? envoy by
- 24. In production at Lyft as of May 2017 Support for INCR, INCRBY, SET, GET (ratelimit service) Service discovery integration (autoscaling!) Active healthcheck using PING (self-healing!) Pipeline splitting, concurrency Basic stats Introducing Envoy Redis
- 25. Envoy Redis Roadmap Additional command support Error handling Pipeline management features Performance optimization Replication - Failover - Mitigate hot read shard issues with large objects - Zone local query routing - Quorum - Protocol overloading? e.g. SET key value [replication factor] [timeout] More!
- 26. Q&A - Thanks! - Email technical inquiries to dhochman lyft.com - Participate in Envoy open source community! lyft/envoy - Lyft is hiring. If you want to work on scaling problems in a fast-moving, high-growth company visit https://www.lyft.com/jobs