Stream Processing in Uber
14 likes•2,408 views
The document discusses Uber's approach to stream processing, highlighting the need for rapid event availability and scalability for handling large volumes of data. It outlines the challenges of data management and the limitations of traditional storage systems, advocating for the use of Apache Samza and Elasticsearch for effective data processing and querying. The content emphasizes the importance of real-time data processing and complex event analysis in enhancing Uber's operational efficiency.
![/driverAcceptanceRate?
geo_dist(10,
[37,
22])&
time_range(2015-‐02-‐04,2015-‐03-‐06)&
aggregate(timeseries(7d))&
eq(msg.driverId,1)](https://image.slidesharecdn.com/untitled-151207121236-lva1-app6891/85/Stream-Processing-in-Uber-92-320.jpg)
Stream Processing in Uber
- 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /uber-stream-processing
- 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
- 4. What is Uber
- 5. Transportation at your fingertips
- 7. Stream Data Allows Us To Feel The Pulse Of Cities
- 9. What’s Going on Now
- 10. What’s Happened?
- 11. Status Tracking
- 16. Uber’s Platform Is a Distributed State Machine Rider States
- 17. Uber’s Platform Is a Distributed State Machine Rider States Driver States
- 18. Applications can’t do everything
- 19. Instead, Applications Emit Events
- 20. Events Should Be Available In Seconds
- 21. Events Should Rarely Get Lost
- 22. Events Should Be Cheap And Scalable
- 24. Where are the challenges?
- 25. Many Dimensions Dozens of fields per event
- 26. Granular Data
- 27. Granular Data
- 28. Granular Data Over 10,000 hexagons in the city
- 29. Granular Data 7 vehicle types
- 30. Granular Data 1440 minutes in a day
- 31. Granular Data 13 driver states
- 33. Granular Data 1 day of data: 300 x 10,000 x 7 x 1440 x 13 = 393 billion possible combinations
- 34. Unknown Query Patterns Any combination of dimensions
- 35. Variety of Aggregations - Heatmap - Top N - Histogram - count(), avg(), sum(), percent(), geo
- 37. Large Data Volume • Hundreds of thousands of events per second, or billions of events per day • At least dozens of fields in each event
- 38. Tight Schedule
- 40. Data Type • Dimensional Temporal Spatial Data Dimension Value state driver_arrived vehicle type uber X timestamp 13244323342 lattitude 12.23 longitude 30.00
- 41. Data Query • OLAP on single-table temporal-spatial data SELECT <agg functions>, <dimensions> FROM <data_source> WHERE <boolean filter> GROUP BY <dimensions> HAVING <boolean filter> ORDER BY <sorting criterial> LIMIT <n> DO <post aggregation>
- 42. Finding the Right Storage System
- 43. Minimum Requirements • OLAP with geospatial and time series support • Support large amount of data • Sub-second response time • Query of raw data
- 44. It can’t be a KV store
- 45. Challenges to KV Store Pre-computing all keys is O(2n) for both space and time
- 46. It can’t be a relational database
- 47. Challenges to Relational DB • Managing multiple indices is painful • Scanning is not fast enough
- 48. A System That Supports • Fast scan • Arbitrary boolean queries • Raw data • Wide range of aggregations
- 49. Elasticsearch
- 50. Highly Efficient Inverted-Index For Boolean Query
- 52. Fast Scan with Flexible Aggregations
- 53. Storage
- 54. Are We Done?
- 55. Transformation e.g. (Lat, Long) -> (zipcode, hexagon)
- 56. Dynamic Pricing
- 57. Trend Prediction
- 58. Supply and Demand Distribution
- 59. Technically Speaking: Clustering & Pr(D, S, E)
- 60. New Use Cases —> New Requirements
- 61. Pre-aggregation
- 63. Sessionization
- 65. State Management
- 66. Apache Samza
- 68. DAG on Kafka
- 69. Excellent Integration with Kafka
- 70. Excellent Integration with Kafka
- 74. What If Storage Is Down?
- 75. What If Processing Takes Long?
- 77. Are We Done?
- 80. Post Processing
- 81. Results Transformation and Smoothing
- 82. Scale of Post Processing 10,000 hexagons in a city
- 83. Scale of Post Processing 331 neighboring hexagons to look at
- 84. Scale of Post Processing 331 x 10,000 = 3.1 Million Hexagons to Process for a Single Query
- 85. Scale of Post Processing 99%-ile Processing Time: 70ms
- 86. Post Processing • Each processor is a pure function • Processors can be composed by combinators
- 87. Post Processing • Highly parallelized execution • Pipelining
- 88. Post Processing • Each processor is a pure function • Processors can be composed by combinators • Highly parallelized execution
- 90. Data Discovery
- 91. Elasticsearch Query Can Be Complex
- 92. /driverAcceptanceRate? geo_dist(10, [37, 22])& time_range(2015-‐02-‐04,2015-‐03-‐06)& aggregate(timeseries(7d))& eq(msg.driverId,1)
- 93. Elasticsearch Query Can Be Optimized • Pipelining • Validation • Throttling
- 94. Timeinseconds
- 95. Elasticsearch Can Be Replaced
- 97. There’s one more thing
- 98. There are always patterns in streams
- 99. There is always need for quick exploration
- 100. How many drivers cancel a request 10 times in a row within a 5-minute window?
- 101. Which riders request a pickup from 100 miles apart within a half hour window?
- 103. Complex Event Processing FROM driver_canceled#window.time(10 min) SELECT clientUUID, count(clientUUID) as cancelCount GROUP BY clientUUID HAVING cancelCount > 10 INSERT INTO hipchat(room);
- 105. Thank You!
- 106. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/uber- stream-processing