Entity Resolution Service - Bringing Petabytes of Data Online for Instant Access
Download as PPTX, PDF2 likes232 views
eBay's entity resolution service (ERS) addresses the challenge of managing hundreds of millions of unstructured user accounts to enhance customer personalization and service while fighting fraud. The system utilizes various linking strategies and handles billions of user identities, offering rapid access to customer data with a high request capacity. Future challenges involve managing large datasets, complex timelines, and the need for real-time processing and redundancy.
Entity Resolution Service - Bringing Petabytes of Data Online for Instant Access
- 1. Entity Resolution Service Bringing Petabytes of Data Online for Instant Access Gurpreet Singh, eBay Ying Zhang, eBay
- 2. Problem Statement 2 eBay maintains hundreds of millions of accounts* across our properties and partners, that are sometimes unstructured, in different formats, different character sets, and are changing over time. Identifying which accounts belong to the same person enables us to personalize each customer's experience, deliver great customer service, and fight fraud. ERS brings xID along with device information online.
- 4. xID 4 Links User Credentials and transient identities to recognize a customer: –Hundreds of millions of user accounts. –20+ different linking strategies with the flexibility of adding more. –Billions of cookies, browser fingerprints, all mobile identities. –Customer recognition across sessions, sites, browsers, devices.
- 5. xID Modular Solution 5 Sources Edges Graph Table Account Entity 102832 10921 236896 10921 786273 10921 324324 23987 349709 73652 152631 73652 543273 37726
- 6. SOURCES – Overall Data Flow 6
- 9. ERS 9 Enables access to consolidated customer Identity –RESTFul service with ~2ms response time –Optimized access to customer identities –10K+ requests per second – 70 Nodes Couchbase Cluster with 11 TB memory in 3 data centers
- 10. Challenges 10 LARGE DATASET SKILLSET COMPLEX TIMELINE FUTURE GROWTH Raw clickstream + PII ~ in PB 20+ edge types 100+ billion edges
- 11. Challenge 11 LARGE DATASET SKILLSET COMPLEX TIMELINE FUTURE GROWTH Graph Join Process Geo Redundancy Requirement
- 12. Challenge 12 LARGE DATASET SKILLSET COMPLEX TIMELINE FUTURE GROWTH Downstream waiting Infrastructure Deliver it ASAP!
- 13. Challenge 13 LARGE DATASET SKILLSET COMPLEX TIMELINE FUTURE GROWTH NoSQL knowledge Near Real Time Processing
- 14. Challenge 14 LARGE DATASET SKILLSET COMPLEX TIMELINE FUTURE GROWTH More business Usecases Data Volume Data Variability Data Velocity
- 15. Key Take Away’s 1 5 • Optimize CB data load to prevent negative service impact • Reduce JSON size to increase CB Active Resident Ratio • Geo-Redundancy by data replication • Enterprise BigData solutions are complex (Velocity, Volume and Variability)
- 16. THE END