Lean Analytics Cycle
373 likes403,622 views
The document discusses the lean analytics cycle of metrics, hypothesis, experiment, and act. It provides examples of how Hello Bar used this process to improve their installation rate. They found a low installation rate in metrics, hypothesized that more options would increase installations, tested this in an experiment, and achieved a 40% increase. Through dozens of experiments, their rate increased by 89%. The document encourages analyzing metrics to find opportunities, forming hypotheses through research, rigorously testing hypotheses, and making data-driven decisions.
Lean Analytics Cycle
- 1. Lean Analytics Lean Product & Lean UX Silicon Valley - July 2014 Hiten Shah [email protected] Download the slides: h p://kiss.ly/leanprod2014
- 3. Continuous Learning & Improvement
- 4. Don’t know where to start?
- 5. Wondering if you’re working on the right things?
- 6. Or, you just want to hear me babble!
- 8. Lean Analytics helps you Optimize for Learning
- 9. Lean Analytics Cycle AVINASH KAUSHIK HTTP://KISS.LY/LEANAC
- 10. Metrics → Hypothesis → Experiment → Act
- 12. What is Hello Bar?
- 21. Metrics Figure out what to improve Hypothesis Make an educated guess Experiment Test your guess Act Decide what’s next
- 22. Our funnel showed a low installation rate Hello Bar Funnel Metrics Steps Visitors Homepage URL Entries Created First Bar Completed Registration Installed Totals 15068 1868 716 688 398 % prev step 12.40% 38.33% 96.09% 57.85% % drop off -87.60% -61.67% -3.91% -42.15% % of visitors 100% 12.40% 4.75% 4.57% 2.64%
- 23. We achieved a 40% increase for our installa2on rate with just this one first experiment. The Lean Analytics Cycle At Work
- 24. Original Hello Bar installation screen
- 25. Metrics Figure out what to improve Hypothesis Make an educated guess Experiment Test your guess Act Decide what’s next
- 26. Find out why people don’t install Hello Bar
- 27. Didn’t Meet My Expectations
- 28. I Had Trouble Installing It
- 29. I Don’t Have A Website
- 30. Not Ready To Install
- 31. Other
- 32. Hello Bar Experiment Hypothesis If more installa2on op2ons are given, then more people will add Hello Bar to their website due to the need for a WordPress plugin and ability to email the code to someone else.
- 33. Metrics Figure out what to improve Hypothesis Make an educated guess Experiment Test your guess Act Decide what’s next
- 34. Created a new flow to test
- 35. I can install code myself.
- 36. 41.67% responded with “WordPress Plugin”
- 37. 25% responded with “email my developer / web designer”
- 38. I need help.
- 39. Metrics Figure out what to improve Hypothesis Make an educated guess Experiment Test your guess Act Decide what’s next
- 40. Full funnel metrics for the experiment Variation Steps Visitors Homepage URL Entries Created First Bar Completed Registration Installed Totals 7456 906 342 329 276 % prev step 12.15% 37.75% 96.20% 83.89% % drop off -87.85% -62.25% -3.80% -16.11% % of visitors 100% 12.15% 4.59% 4.41% 3.70% Control Steps Visitors Homepage URL Entries Created First Bar Completed Registration Installed Totals 7469 925 353 341 197 % prev step 12.38% 38.16% 96.60% 57.77% % drop off -87.62% -61.84% -3.40% -42.23% % of visitors 100% 12.38% 4.73% 4.57% 2.64%
- 41. Resulted in a 30% installation rate increase HTTP://ISVALID.ORG Experiment resulted in a 40% improvement
- 42. ACer a dozen more experiments, our installa2on rate is now hovering between 5% and 7% That’s an addi;onal 89% increase!!! The Lean Analytics Cycle At Work
- 43. Metrics → Hypothesis → Experiment → Act How You Can Put It Into Practice
- 44. Metrics Figure out what to improve
- 45. “What gets measured, gets managed.” Peter Drucker
- 46. Baselines DATA DNA EBOOK HTTP://KISS.LY/OPTDATADNA
- 47. Funnels KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL
- 48. SaaS Funnel KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL
- 49. eCommerce Funnel KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL
- 51. Spreadsheets and Calculations HELLOBAR HTTP://HELLOBAR.COM Steps Visitors Homepage URL Entries Created First Bar Completed Registration Installed Created Second Bar Totals 15068 1868 716 688 398 193 % prev step 12.40% 38.33% 96.09% 57.85% 48.49% % drop off -87.60% -61.67% -3.91% -42.15% -51.51% % of visitors 100% 12.40% 4.75% 4.57% 2.64% 1.28%
- 52. Hypothesis Make an educated guess
- 53. “A solid test hypothesis is an informed solution to a real problem – not an arbitrary guess. The more research and data you have to base your hypothesis on, the be er it will be.” — Michael Aagaard, h p://kiss.ly/abthypo
- 54. Research Tools & Tac;cs
- 56. Jobs To Be Done #JTBD HTTP://JOBSTOBEDONE.ORG
- 63. Email a survey to people HTTP://SURVEYMONKEY.COM
- 64. Ask people why they canceled HTTP://SURVEYMONKEY.COM
- 65. Get feedback right when they cancel
- 68. Find out what words people use to describe your product HTTP://QUALAROO.COM
- 69. Learn about people’s expectations HTTP://INTERCOM.IO
- 70. Learn what frustrates people HTTP://INTERCOM.IO
- 71. More actionable reading… QUALITATIVE DATA HTTP://KISS.LY/QUALABT
- 72. Experiment Hypothesis Formula DATA DNA EBOOK HTTP://KISS.LY/OPTDATADNA
- 73. Experiment Test your guess
- 74. Only 1 out of 5 tests win. 4 out of 5 tests cost you money. $
- 75. 1) Which top metric are we trying to improve? 2) What can you anchor your metric against? 3) What is your hypothesis about why this metric is at x%? 4) What do we expect from this experiment? 5) What was the result and what did we learn from it? HTTP://KISS.LY/6QSAB Answer key questions for each experiment
- 77. Don’t copy “best practices” blindly
- 78. Learn what works best for you
- 79. Act Decide what’s next
- 80. Explore the results and make data informed decisions. THE ANALYTICS THAT MATTER TO FACEBOOK HTTP://KISS.LY/FBINFORM
- 81. KISSmetrics A/B Testing Report KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
- 83. We know that people love Google Analytics
- 84. KISSmetrics Experiment Hypothesis If people authen2cate with Google, then more people will sign up due to their exis2ng familiarity with Google Analy2cs and alignment with our winning headline.
- 85. KISSmetrics Homepage Test Control
- 86. KISSmetrics Homepage Test Variation
- 87. KISSmetrics Homepage Test Signup Conversion Rate KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
- 88. KISSmetrics Homepage Test Received Data KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
- 89. KISSmetrics Homepage New Control
- 90. Inspiration
- 91. KISSmetrics Homepage New Test Variation
- 92. KISSmetrics Homepage Test Signup Conversion Rate KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
- 93. KISSmetrics Homepage Test Received Data KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
- 95. Metrics Figure out what to improve Hypothesis Make an educated guess Experiment Test your guess Act Decide what’s next
- 96. Hiten Shah [email protected] @hnshah You will get all you want in life if you help enough other people get what they want. Zig Ziglar Download the slides: h p://kiss.ly/leanprod2014