How LinkedIn Democratizes Big Data Visualization
3 likes1,222 views
The document discusses LinkedIn's data analytics infrastructure, focusing on the use of real-time analytics through a system called Pinot, which supports low-latency queries and complex dimensions. It highlights various interactive analytics products for both external members and internal reporting, emphasizing fresh data access and ease of use. The architecture involves technologies like Kafka and Hadoop for data ingestion and management, aiming to enhance user experiences with quick and insightful visualizations.
How LinkedIn Democratizes Big Data Visualization
- 1. ORGANIZATION NAME Photo: Courtesy of O'Reilly Conference on Flickr How LinkedIn Democratizes Big Data Visualization
- 2. Democratizes Big Data Visualization How Jonathan Wu Praveen Neppalli Naga Chi-Yi Kuan
- 3. 313,000,000 Members End of Q2 2014
- 4. 25,000,000,000 Page Views Q2 2014
- 7. What can we do with Linkedin data ?
- 8. Sales Talent flow between companies
- 10. Is it simple? Member attributes Page View events data
- 11. Photo Credit: https://www.flickr.com/photos/johnjoh/1060267344 Data is the new vineyard
- 12. Photo Credit: https://www.flickr.com/photos/johnjoh/1060267344 Data is the new vineyard
- 13. Data infra: collect & prepare data Collect & Prepare Data Mysql, Oracle, Kafka + Hadoop Serve Data Pinot Taste Data Easy-to-use visualization
- 14. Data Computation ETL HDFS Y A R N Map-Reduce Spark Tez Pig Hive Cubert Kafka Data Stores Hadoop
- 15. Data infra: Serve data Collect & Prepare Data Kafka + Hadoop Serve Data Pinot Taste Data Easy-to-use visualization
- 16. Products for members/customers with real-time interactive analytics •Who’s Viewed Your Profile •Ads Reporting •Jobs Analytics Categories of interactive analytics products Interactive business analytics for internal use •How feature X is performing Real-time business monitoring •Page view changes across mobile devices in different regions
- 17. Requirements for real-time interactive analytics Slice and dice billions of records, hundreds of dimensions End to end freshness of minutes not hours Sub-second query response times e.g. Which are top regions that contribute to my profile views? Which industries in those regions?
- 18. Pinot Distributed Analytics Infrastructure that serves Interactive Analytics products at Linkedin.
- 19. Data Indexes Distributed System Ingestion What is Pinot? Compressed Columnar indexes (supports Mmap and In-memory) Apache Helix for cluster management Apache Kafka (for near real-time) and Hadoop
- 20. Data Indexes Single Value Index Multi Value Index Inverted Index •Fixed bit length encoding •Sorted Index •Secondary Sorted Index •Multi-value Fixed bit length encoding •BitMap Multi-value Index •P4Delta •Modified P4Delta •BitMap
- 21. Cluster Management •Create Resources •Update Resource meta data •Expand/Contract partitions dynamically •Query Router
- 22. Data Ingestion Kafka for Realtime Hadoop for Historical
- 23. High Level Architecture PINOT Hadoop Kafka Historical Realtime CLUSTER MANAGER Controller Helix Zookeeper Broker 1 Broker 2 Server 1 Server 2 Server 3
- 24. Core Features Low latency and high QPS OLAP Queries with real-time ingestion Support complex dimensions Operational simplicity Data bootstrapping & reconciliation
- 25. Usage @ Linkedin About 18 member facing products on Linkedin.com Internal Reporting Open-source.…coming soon
- 26. Reporting UI: serve & taste data Collect & Prepare Data Kafka + Hadoop Serve Data Pinot Taste Data Easy-to-use visualization
- 27. I want to access big data without running SQL Business need
- 28. Start a new dashboard with one click
- 29. Select what metrics/dimensions you want
- 30. Charts are rendered in just a few seconds
- 31. Zoom into a single chart
- 32. Filter on various dimensions
- 34. Portal that connects dashboards, internal reports, and internal Wiki Pages Enterprise analytics portal
- 35. Scale of the data Pinot for interactive analysis Self service visualization for insights Summary
- 37. We are hiring Jonathan Wu www.linkedin.com/in/pneppalli www.linkedin.com/in/jiyewu www.linkedin.com/in/chiyikuan [email protected] Praveen Neppalli Naga [email protected] Chi-yi Kuan [email protected] 650-605-2184 650-962-3299 650-426-6301