![Hypotheses Template
Based on user research, we know that [X] is an important user problem. If we do
[Y], we expect to see [this] change. This change should cause an increase in
[KPIs].](https://image.slidesharecdn.com/goketombuk-171106140534/85/GOKCE-TOMBUL-HOW-TO-BUILD-A-SUCCESSFUL-EXPERIMENTATION-PROGRAM-50-320.jpg)
![Hypotheses Template
Based on user research, we know that [X] is an important user problem. If we do
[Y], we expect to see [this] change. This change should cause an increase in
[KPIs].
Example: “Based on user research, we know that 40% of first-time visitors
are not sure if Udemy is the right place to learn a new skill. If we feature
successful student stories on our product pages, we expect to increase the
trust of our first-time visitors. This should cause lower bounce rates and
increased conversion rates."](https://image.slidesharecdn.com/goketombuk-171106140534/85/GOKCE-TOMBUL-HOW-TO-BUILD-A-SUCCESSFUL-EXPERIMENTATION-PROGRAM-51-320.jpg)
GOKCE TOMBUL - HOW TO BUILD A SUCCESSFUL EXPERIMENTATION PROGRAM
- 1. HOW TO BUILD A SUCCESSFUL EXPERIMENTATION PROGRAM Gokce Tombul
- 2. A little bit about me Gokce Tombul Sr. Director of Product, Growth & Discovery
- 3. A little bit about Udemy We are a marketplace for online courses. Instructors Students
- 5. Let’s talk about how to build a successful experimentation program.
- 6. "To invent you have to experiment, and if you know in advance that it's going to work, it's not an experiment.” - Jeff Bezos Image source: huffpost.com
- 7. - Limited resources - Few experiments per quarter - No clear process - Very little revenue contribution coming from experiments Before
- 8. - Limited resources - Few experiments per quarter - No clear process - Very little revenue contribution coming from experiments Before - 4 agile teams - 40 - 50 experiments per quarter - Established process - Significant revenue contribution Today
- 9. Our Approach Step 1: Set Clear Business Metrics
- 10. Our Approach Step 1: Set Clear Business Metrics Step 2: Ideation
- 11. Our Approach Step 1: Set Clear Business Metrics Step 2: Ideation Step 3: Prioritization
- 12. Our Approach Step 1: Set Clear Business Metrics Step 2: Ideation Step 3: Prioritization Step 4: Execution
- 13. 0
- 14. This is how I felt Image source: beyondanxietyanddepression.com
- 15. How did we actually get there? Step 1: Build the Right Process Step 2: Build the Right Team Step 3: Create the Right Culture
- 16. Step 1: Build the Right Process Image source: 123rf.com
- 17. Process: The Experimentation Loop Prioritize User Problems Form Hypotheses Gather Data Prioritize Hypotheses Design Build Learn & Improve Test
- 18. Process: The Experimentation Loop Prioritize User Problems Form Hypotheses Gather Data Prioritize Hypotheses Design Build Learn & Improve Test
- 19. Gather Data Rely on quantitative data to answer these questions: - How do people use your product? - Where do they get stuck?
- 20. How Do People Use Your Product: Pathway Analysis
- 21. Pathway Analysis Traffic Channel Landing Page Discovery Conversion
- 22. Pathway Analysis Traffic Channel Landing Page Discovery Conversion Note: Does not represent real Udemy numbers 30% Direct 40% SEO 20% Paid Acq
- 23. Pathway Analysis Traffic Channel Landing Page Discovery Conversion Note: Does not represent real Udemy numbers 30% Direct 40% SEO 20% Paid Acq 70% Product Page 30% Home Page
- 24. Pathway Analysis Traffic Channel Landing Page Discovery Conversion Note: Does not represent real Udemy numbers 30% Direct 40% SEO 20% Paid Acq 70% Product Page 30% Home Page 20% Search 10% Browse 2% Success Page
- 25. Pathway Analysis Traffic Channel Landing Page Discovery Conversion 30% Direct 40% SEO 20% Paid Acq 70% Product Page 30% Home Page 1% Success Page20% Search 10% Browse 2% Success Page Note: Does not represent real Udemy numbers
- 26. How Do People Use Your Product: Funnel Analysis Understand the leakiest parts of your funnel Product Landing Page 1,000,000 Watched the Preview Video 600,000 Created an Account 60,000 Purchased 1,200 60% 10% 2% Note: Does not represent real Udemy numbers
- 28. 3 Questions to Ask to Figure Out the Right Segment
- 29. 3 Questions to Ask to Figure Out the Right Segment 1) Which type of users might be using your product differently?
- 30. 3 Questions to Ask to Figure Out the Right Segment 1) Which type of users might be using your product differently? 2) Which one of these users actually behave differently in your product?
- 31. 3 Questions to Ask to Figure Out the Right Segment 1) Which type of users might be using your product differently? 2) Which one of these users actually behave differently in your product? 3) Is the segment that’s behaving differently big enough for you to care?
- 32. 3 Questions to Ask to Figure Out the Right Segment 1) Which type of users might be using your product differently? 2) Which one of these users actually behave differently in your product? 3) Is the segment that’s behaving differently big enough for you to care? For Udemy We ran the pathway analysis and funnel analysis for these user segments: - First time Udemy visitors vs returning visitors - Mobile users vs desktop users - Users coming from different traffic channels - Users coming to buy different category of courses
- 33. We decided to focus on First time Udemy visitors that landed on one of our product pages and didn’t take any action on the page.
- 34. Process: The Experimentation Loop Prioritize User Problems Form Hypotheses Gather Data Prioritize Hypotheses Design Build Learn & Improve Test
- 35. Quantitative data is helpful to understand how users behave
- 36. Quantitative data is helpful to understand how users behave but it’s not very helpful to understand “why”
- 37. When users showed exit intent, we asked a simple question: What is preventing you from purchasing this course right now? Qualitative User Research Image source: blog.informizely.com
- 38. Prioritization of User Problems 1) Start with an open ended survey question to your users. E.g - “What is preventing you from purchasing this course right now?”
- 39. Prioritization of User Problems 1) Start with an open ended survey question to your users. E.g - “What is preventing you from purchasing this course right now?” 2) Bucket the answers in meaningful categories and this time ask as a multi-choice question. E.g “I will not purchase because the price is not right.” or “I don’t trust Udemy yet”.
- 40. Top 3 Reasons For Udemy Were - I don’t know if Udemy is the right place to learn this skill. - I don’t understand how this course compares to the other courses. - I don’t know if this course fits my skill level.
- 41. Now, we are so much more focused The new focus: “We would like to increase Revenue Per Visitor for first time visitors who land on one of our product pages. And we will do that by solving these 3 problems for users” The old focus: “We would like to increase the Revenue Per Visitor.”
- 42. Process: The Experimentation Loop Prioritize User Problems Form Hypotheses Gather Data Prioritize Hypotheses Design Build Learn & Improve Test
- 43. Form a clear hypothesis that ties your idea to the problem you are trying to solve. Your hypothesis statement should answer these questions: Form Hypotheses
- 44. 1) What is the problem you are solving? Form Hypotheses Form a clear hypothesis that ties your idea to the problem you are trying to solve. Your hypothesis statement should answer these questions:
- 45. 1) What is the problem you are solving? 2) Which user segments have this problem? Form a clear hypothesis that ties your idea to the problem you are trying to solve. Your hypothesis statement should answer these questions: Form Hypotheses
- 46. 1) What is the problem you are solving? 2) Which user segments have this problem? 3) Is this a real problem? What evidence do we have to support that? Form a clear hypothesis that ties your idea to the problem you are trying to solve. Your hypothesis statement should answer these questions: Form Hypotheses
- 47. 1) What is the problem you are solving? 2) Which user segments have this problem? 3) Is this a real problem? What evidence do we have to support that? 4) How big is the problem? Form a clear hypothesis that ties your idea to the problem you are trying to solve. Your hypothesis statement should answer these questions: Form Hypotheses
- 48. Form a clear hypothesis that ties your idea to the problem you are trying to solve. Your hypothesis statement should answer these questions: Form Hypotheses 1) What is the problem you are solving? 2) Which user segments have this problem? 3) Is this a real problem? What evidence do we have to support that? 4) How big is the problem? 5) How are we going to solve the problem
- 49. Form a clear hypothesis that ties your idea to the problem you are trying to solve. Your hypothesis statement should answer these questions: Form Hypotheses 1) What is the problem you are solving? 2) Which user segments have this problem? 3) Is this a real problem? What evidence do we have to support that? 4) How big is the problem? 5) How are we going to solve the problem 6) How will we know if this experiment succeeds?
- 50. Hypotheses Template Based on user research, we know that [X] is an important user problem. If we do [Y], we expect to see [this] change. This change should cause an increase in [KPIs].
- 51. Hypotheses Template Based on user research, we know that [X] is an important user problem. If we do [Y], we expect to see [this] change. This change should cause an increase in [KPIs]. Example: “Based on user research, we know that 40% of first-time visitors are not sure if Udemy is the right place to learn a new skill. If we feature successful student stories on our product pages, we expect to increase the trust of our first-time visitors. This should cause lower bounce rates and increased conversion rates."
- 52. Process: The Experimentation Loop Prioritize User Problems Form Hypotheses Gather Data Prioritize Hypotheses Design Build Learn & Improve Test
- 53. Prioritization of Hypotheses 1) ICE Model 2) Build a Customized Criteria for Your Own Business
- 55. Customized Framework Above the Fold (Y/N) Addressing an issue discovered via user research? (Y/N) Did we have success in a similar experiment before? (Y/N) Confidence (1-5) Ease of Building (1-5) Total Score 1 1 1 5 5 20 0 1 1 4 3 11 1 0 0 3 5 10
- 56. Process: The Experimentation Loop Prioritize User Problems Form Hypotheses Gather Data Prioritize Hypotheses Design Build Learn & Improve Test
- 57. Process: The Experimentation Loop Prioritize User Problems Form Hypotheses Gather Data Prioritize Hypotheses Design Build Learn & Improve Test
- 58. Analyzing the Results Two key questions to answer: 1) Did the experiment solve the user problem? 2) Did solving the user problem move the needle?
- 59. Record and share your learnings and decide on the next steps
- 60. Your Success Depends On: 1) How strong your hypotheses are
- 61. 1) How strong your hypotheses are 2) How strong your prioritization framework is Your Success Depends On:
- 62. Your Success Depends On: 1) How strong your hypotheses are 2) How strong your prioritization framework is 3) How fast you can move through the cycle
- 63. Your Success Depends On: 1) How strong your hypotheses are 2) How strong your prioritization framework is 3) How fast you can move through the cycle 4) How much you are learning in each cycle
- 64. Step 2: Building the Right Team Image source: www.scrumalliance.org
- 65. The Roles on the Team - The Experiment Owner (Product Manager, Marketer, CRO Expert) - Engineers - User Experience Designer - Data Analyst - Data Scientist - User Researcher
- 66. Managing Dependencies Image source: agiletrainings.eu
- 68. To remove dependencies, create dedicated, cross-functional teams with clear decision makers. Image source: www.solstice.com
- 69. Qualities of a Good Experimenter
- 75. Step 3: Create the Right Culture Image source: www.halogensoftware.com
- 77. 8 Out of 10 Experiments Fail
- 78. Eliminate Fear of Failure Image source: saraohara.com
- 80. Experiment to Win Learn
- 81. Reward not only wins, but also failed experiments with important learnings
- 82. It Has Been a Winding Road Image source: www.feco.info
- 83. This Is How We Started Image source: www.fafsfoundation.org
- 84. This Is How We Felt Along The Way Image source: www.sportpsychologytoday.com
- 85. This Is How We Are Feeling Now Image source: www.ntd.tv