A/B testing at Spotify
14 likes11,658 views
The presentation discusses A/B testing at Spotify, emphasizing its role in improving product functionality and user engagement among millions of subscribers. It outlines the challenges faced by Spotify's autonomous teams and proposes a structured approach using metrics to optimize features to avoid software complications. Key takeaways include the importance of automation and education to enhance the A/B testing process.
A/B testing at Spotify
- 1. February 12, 2015 A/B testing at Spotify Ali Sarrafi & Evan Shrubsole
- 2. Who is this? 2 Ali Sarrafi Evan Shrubsole Software EngineerProduct Owner
- 3. Section name Source: Agenda 3 Spotify intro How Spotify works Our challenges How we manage A/B tests How we implement A/B tests Conclusions + Q&A
- 8. February 12, 2015 The right music for every moment 8
- 9. 9 15 million subscribers 60 million monthly active users 30 million songs
- 10. Tribes Provide fast and reliable access to all the world's music Enable high product development speed while maintaining a highly available service P P P P Squad Squad Squad Squad Chapter Chapter P P P P Squad Squad Squad Squad Chapter Chapter Our team is growing rapidly 400 30 100 10
- 11. 11 Autonomous fast moving teams Feature 1 Feature 3 Feature 2 Feature 4 Feature 5
- 12. February 12, 2015 More Context Over 50 autonomous teams working on features Over 7 platforms with multiple features Over 3000 source repositories
- 13. Section name 13 How to make sure we improve the product?
- 14. 14 Hippo Approach Highest Paid Person’s Opinion
- 15. Do A/B testing! 15 A/B testing is a simple randomised experiment with at least two variants, which are the control and treatment in the controlled experiment
- 17. 17 We don’t want a Frankenstein software Challenge
- 18. 18 Well at least a good looking 2014 edition
- 19. Section name 19 How can we align teams’ optimisation to avoid the Frankenstein effect?
- 21. Source: 21 DAU (Daily Active Users) MAU (Monthly Active Users)
- 22. Section name 22 Evolving Metrics
- 23. Focus on functionality as a metric We want people to play music and enjoy it 23
- 24. 24 DAU/MAU Function 1 Function 2 Function 3 Function 4 CTR/Team Metrics Importance (High to low) Order of optimisation
- 25. Section name 25 How should we implement this efficiently?
- 26. Code 26 if (shuffle_button = “green_square") show_green_square_shuffle() else show_old_shuffle()
- 27. Managing tests 27 A/B test Name: My test Hypothesis: … …. …. Registry
- 29. 29 Making magic happen with desired metrics Vital Sign Group A Group B Functionality 1 +1.5% -0.5% Functionality 2 +2.5% -3% Functionality 3 +0.5% 0% Functionality 4 +1% -7%
- 30. Section name 30 How can we make the process more successful?
- 31. 31 It’s all about automation
- 32. 32 and of course education
- 33. Copyright Napster pic by bixentro, cc 2.0 http://www.flickr.com/photos/bixentro/ Busy record store pic by gomattolson http://www.flickr.com/photos/gomattolson Empty record store pic by passetti http://www.flickr.com/photos/passetti/ First Frankenstein screenshot form : “Frankenstein” movie by Kenneth Branagh, 1994 Second Frankenstein screenshot form : “I, Frankenstein” movie by Stuart Beattie, 2014 Automation screenshot from: Startrek Next generation, TV show (1987–94)
- 34. Thanks