Cassandra and Riak at BestBuy.com
- 2. WHO WE ARE • Best Buy is the world’s largest multi-channel consumer electronics retailer, with stores in the United States, Canada, China and Mexico. • We are the 10th largest online retailer in the United States • More than 1.6 billion visitors come to our stores and BestBuy.com each year. • Our My Best Buy loyalty program is among the largest loyalty programs of its kind, with more than 40 million active members. • We provide customers with outstanding choice, unbiased advice and unmatched support for the tech needs. January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 3. A UNIQUE CUSTOMER PROMISE • THE LATEST DEVICES AND SERVICES, ALL IN ONE PLACE • IMPARTIAL & KNOWLEDGEABLE ADVICE • COMPETITIVE PRICES • THE ABILITY TO SHOP WHEN AND WHERE YOU WANT • SUPPORT FOR THE LIFE OF YOUR PRODUCTS January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 4. JOEL CRABB KANNAN SWAMINATHAN Chief Architect, Director, Web BestBuy.com Architecture January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 5. NOSQL AT BESTBUY.COM • Schema-less design —Key–value systems —Sparse column systems —Non-relational data • Distributed nature —Eventual consistency —Active-Active across clouds and datacenters —High reliability —Horizontal scaling January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 6. BESTBUY.COM January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 7. SCALABILITY GOALS • Near-Infinite • Bursts • 7X traffic spikes • Bursts > 50,000 rps • #4 in eCommerce traffic during 2013 Holiday Walmart Best Buy January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 8. FLEXIBILITY GOALS Low cost of change Fast concepts to site Daily releases Multiple versions One day of work vs. two months
- 9. RELIABILITY GOALS • 100% availability • Zero defects • Achieved 100% cloud uptime during Holiday • ~ 2s response times January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 10. CLOUD ARCHITECTURE CONCEPTS • Clouds fail; plan for it —Multiple availability zones —Multiple regions —Multiple vendors • Datacenter connections fail; plan for it —Serve pages completely from cloud —Browse-only fallback mode January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 11. CLOUD ARCHITECTURE CONCEPTS CDN: Global Traffic Manager Browse Cloud Browse Cloud Vendor 1 Vendor 2 Best Buy Datacenter January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 12. RIAK January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 13. PRODUCT CATALOG - REQUIREMENTS • Business Requirements —Easily add new attributes to products —Use existing product content feeding systems —Provide enhanced product data to all of Best Buy • Technical Requirements —Scale to BestBuy.com’s needs —One-way replication from DC to cloud —Provide a Product Catalog API usable by Best Buy applications internally (DC) and externally (cloud) January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 14. BROWSE CLOUD - ARCHITECTURE Persistent Cache Load Balancer Web App Web App Service Aggregator Product Data Product Data Datacenter Legacy Services and Product Data January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 15. WHY RIAK FOR THE PRODUCT CATALOG • Key-value Store —Easily add new attributes —Different attribute sets for different product categories • Hub and Spoke Replication —Multiple cloud instances can be out-of-synch for seconds or minutes; this is OK —Connection to Datacenter can be lost • Resilience within all tiers —Riak’s ring architecture allows single instances to fail with little impact to the system January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 16. WHY RIAK FOR THE PRODUCT CATALOG • Legacy Bridge —Extract data from legacy system —Expose to anyone who needs the data —Extends decision to retire legacy system • Input into Product Evolution —Best Buy is stretching Riak in multiple areas —Feedback on Search and Replication —Direct connection to Riak engineers January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 17. PRODUCT CATALOG DEPLOYMENT Cloud Replicant Cloud Replicant V1.2.1 Each Cloud Replicant is a self-contained instance which can intermittently lose connectivity to the Datacenter Datacenter Master V1.4.2 January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 18. PRODUCT CATALOG - DATA FLOW Cloud Web Application Granular Service Aggregator Service NoSQL Product Data Replication Extraction Datacenter Legacy Product Data NoSQL Product Data January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 19. RIAK LESSONS LEARNED • Search eats up the CPU (Pre-Yokozuna) Search Endpoint Removed January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 20. RIAK LESSONS LEARNED • Multi-datacenter replication is hard at scale • Object replication fails silently (v1.4.2) 1.5 hours for partial update 4000 objects/sec January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 21. DID THE PRODUCT CATALOG SUCCEED? • Scale – generally < 5 ms response times in cloud • Scope – provides APIs in cloud and DC Single zone 1.5 hours response update failure, for partial times jump to 8-10 ms 4 ms @ 95% 4000 objects/sec January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 22. CASSANDRA January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 23. CUSTOMER GRAPH - REQUIREMENTS • Business Requirements —Create a single identity provider —Build an adaptive model to support a 360 degree view of the customer, customer segmentation and multi-channel personalization —Extensible framework to support federated identity interoperability with external providers • Technical Requirements —Scale to BestBuy.com’s needs with a distributed service oriented architecture —Support risk based authentication and secure service interfaces for consumers January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 24. CUSTOMER GRAPH - ARCHITECTURE CDN + GTM + WAF Load Balancer API Load Balancer API Topology: NTS RF: 3 Partitioner: Random Data Center 1 / Region 1 API API Cassandra Ring Data Center n / Region n January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 25. CUSTOMER GRAPH - CONCEPTUAL Create Connection: POST /customers/Kannan/following/customers/Joel Get all Kannan’s connections: GET /customers/Kannan/connections Get Joel’s “followers”: GET customers/Joel/followers Get customers connecting to a specific store: GET /stores/{store id}/connecting/customer January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 26. CUSTOMER GRAPH – CONCEPTUAL Topics Products Blogs January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 27. WHY CASSANDRA? • No single point of failure • Delivers on Atomicity, Isolation & Durability • Eventual Consistency • Tunable Consistency for Reads vs. Writes • Linear Scalability • Querying a column slice or a range of row keys • Data can have expiration set • Reliable multiple data center replication January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 28. LESSONS LEARNED – DATA MODELING Use composite column names; static composite types Column names are stored physically sorted and indexed Store “values” as column names Wide rows in conjunction with composite columns can be used to build indices, but… For larger data sets, distribute the columns among rows A secondary index is best modeled as a separate column family January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 29. LESSONS LEARNED – DATA MODELING Do not store an entity as a single column blob Cannot index and query on entity attributes Updates to an entity attribute would require a read and then a write Mutate just the required columns on an entity row Do not read and then write January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 30. LESSONS LEARNED - MONITORING Heap Size and Use GC Mutation Stage (Writes) & Read Stage (Reads) Active and Pending AE Service, Stream & Message-Streaming-Pool Stages Especially during scheduled repair or rebuild (while adding nodes to a new data center) Streaming requires keep-alive connections (watch out for firewalls terminating idle established connections; update periodic tcp keep-alive ping rate) Compaction Stage & Compaction Count IO Wait, Limits - nofiles, nproc January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 31. IMPLEMENTATION STRATEGY • Start with a project team —Riak – Product Catalog —Cassandra – Customer Graph • Build expertise in development and operations • As usage grows, create a Platform team —Combine engineering into one team —Project team can then focus on business features —Focus on automation January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 32. CONCLUSIONS • Riak and Cassandra have both been successful —Riak – No complete outages in two years —Cassandra – Flawless in its first Holiday in 2013 • Differences —Replication patterns are different —Write and read treatment is different —Deployment pattern are different • High Availability —Both have highly resilient architectures —Both scale linearly January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.
- 33. January 16, 2014 @Copyright 2014 – Best Buy, Inc. All rights reserved.