From SQL to NoSQL - StampedeCon 2015
1 like538 views
The presentation discusses T-Mobile's transition from SQL to NoSQL to enhance customer service through big data, focusing on efficiently diagnosing device issues. It details the evolution of their technology from using traditional relational databases to implementing Cassandra and a Lambda architecture for real-time data processing. Key lessons include the importance of monitoring, organization of data, and maintaining transparency with customers regarding data collection and usage.
![Technology: Version 2
11
Device
C*
Second attempt: Cassandra!
Findings:
Stellar write performance
Stellar read performance
Relatively low TCO
X Application misused [beta] C* driver
X Application not designed to scale
X Heterogeneous architecture
(.NET/Linux)](https://image.slidesharecdn.com/fromsqltonosql-stampedecon2015-150724184010-lva1-app6892/85/From-SQL-to-NoSQL-StampedeCon-2015-11-320.jpg)
From SQL to NoSQL - StampedeCon 2015
- 1. FROM SQL TO NOSQL Timothy Shelton, T-Mobile USA StampedeCon, 2015 1
- 2. Outline • BUSINESS GOALS: Why do we need Big Data? • TECHNOLOGY: What did we build? • PRIVACY: Customer preference management • LESSONS LEARNED: Toes were stubbed. • Q&A 2
- 3. WHY DO WE NEED BIG DATA? 3
- 4. Business Goals 4 We want you to have a great experience with your device, and we want to provide world-class service to quickly resolve any device problems.
- 5. Example Use Case Symptom: “Battery keeps going dead” Possible Root Causes: • Bad application • Misconfigured application • Misconfigured device • Incorrect charger • Improper expectations • Faulty hardware 5
- 6. Tackling Complexity At the beginning of 2015, there were an estimated 1.5 million applications in the Google Play Store 6 1Q14 4Q14 1Q15 Smartphones 6.9 8.0 8.0 Non-Smartphones 0.5 0.6 0.5 Mobile Broadband 0.1 0.4 0.3 Total Company 7.5 9.0 8.8 (Device Sales, Millions) Challenge: Accurately diagnose device problems, educate customers, solve problems quickly. We sell a ton of devices:
- 7. By the numbers…. 7 30,000 Average number of technical calls received daily 32 Average number of customer-installed applications per device 56,836,000 Total customers, as of 1Q2015 (combined) 100% Percentage of customers expecting problem resolution
- 8. Solution: BIG DATA! 8 Device GOAL Provide Customer Care Representatives visibility into the device so they are able to resolve problems. REQUIREMENTS 1. Ability to gather metrics from a device 2. Ingest the metrics 3. Store for a period of time 4. Fast data retrieval
- 9. TECHNOLOGY 9
- 10. Technology: Version 1 10 Device SQL Server First attempt: RDBMS to store data. Findings: X Tables had to be de-normalized and indices removed to maintain write performance X Reads were barely achieved by batch processing snapshots of the database Effectively deployed a write-only database. ? ? ? ?
- 11. Technology: Version 2 11 Device C* Second attempt: Cassandra! Findings: Stellar write performance Stellar read performance Relatively low TCO X Application misused [beta] C* driver X Application not designed to scale X Heterogeneous architecture (.NET/Linux)
- 12. Technology: Version 3 12 Device C* Evolution: Lambda Architecture! Goals: Enables multiple consumers of incoming data Allows for near-realtime processing (Spark Streaming) Archival to S3 for offline processing (Spark) Layered architecture takes advantage of Cloud
- 13. Culture: Fast and Lean • Preference is rapid prototyping and proofs-of-concept • We set aggressive dates, then iterate • Cloud deployments align nicely with our culture 13 Embrace automation • CloudFormation Templates to reduce risk of error and increase velocity • Chef for deployment and configuration
- 15. Transparency and Choice • Clearly describe what is being collected • Be transparent about why it is being collected • What are the intended uses of the data? • Be clear with whom the data will be shared (if anyone) • Allow the customer to opt-out • Opt-in for Location- Based Services 15
- 16. Transparency: How not to do it 16 “It’s in our privacy policy!”
- 18. Scale Thy Monitoring 18 This looks bad! OMG, traffic is being rejected! What Worked What Didn’t Chart all the things! In this instance, statsd was being overloaded with traffic
- 19. Organize Thy Data 19 What Didn’t No temporal organization made querying this data more difficult every day What Worked Archive all the things!
- 20. Cassandra: df -u • Monitor your capacity • Leave enough disk for compaction • Disabling compactions is a terrible strategy 20
- 21. QUESTIONS? 21
- 22. Thanks! T-Mobile is actively recruiting engineers – come change the world with us! 22 @TimothyAShelton http://www.t-mobile.com