Optimizely Customer Showcase
2 likes1,051 views
The document discusses A/B testing strategies and results from various organizations, including DonorsChoose.org and Stella & Dot. It highlights successful changes, such as adjusting donation amounts and the effectiveness of different taglines, alongside significant conversion rate increases. The importance of a structured testing process and collaboration among teams for optimal results is emphasized throughout.
Optimizely Customer Showcase
- 1. Mark Halvorson Head of Interac2ve Marke2ng, Atlassian @halv0012 Charlo=e Weiski=el Sr Director, Product DonorsChoose.org @noDornaught Jeremy Yoches Sr Product Manager, Stella & Dot Customer Showcase
- 4. A tes0ng team of one
- 7. Remove $500 and $1000 • Removing $500 and $1000 increased conversion by 58%.
- 8. Photo and tagline (mul0variate test) • Female teacher performed best with 3 of 4 taglines • “Join 3,500 people…” did slightly be=er than the original, but it wasn’t significant. 1. A teacher spends $40 each month on classroom essen2als. (original) 2. Make giving a monthly habit. 3. Join the 3,500 people who support a classroom every month. 4. Help a classroom every month.
- 9. Charge your card now or in a month? • Made the change to wait a month before we had Op2mizely • Removing men2on of the next month increased conversion +217%.
- 10. $50, $25 or no incen0ve • Conversion rate – $50 incen2ve: 1.8% – No incen2ve: 1.6% – $25 incen2ve: 1.3% • The no incen0ve group was the least likely to cancel a`er two months (8%) • The no incen2ve group had the highest average monthly signup amount of $26.
- 11. Tips from a lean non-‐profit • Just start tes2ng • Clear the path for the tester • Be a broken record • Look at the results holis2cally, not just A vs. B • Don’t overfill tes2ng road map
- 13. About Atlassian We help great teams build be=er so`ware, together. • Products include: JIRA, Confluence, Bitbucket, HipChat… • Founded in 2002 • Over 35,000 Customers • Over 700 employees globally • No sales people
- 14. A/B Tes0ng at Atlassian is cross func0onal Central Growth Hacking Team to support: • Email • Product Management • Customer PlaDorm • Customer Advocates • Internal Systems • Leads • Product Marke2ng • Support • Bitbucket • HipChat
- 15. High Level view of sta0s0cs. Blimp Dashboard
- 16. Examples
- 17. Reduce Fric0on
- 20. A Highlighted Example : Simplify B
- 21. Lather, rinse, repeat Our Process • Collect great ideas • Triage based on poten2al impact • Build-‐out good candidates • Schedule to avoid conflicts • Run experiments & analyze results.
- 22. Where do experiment ideas come from?
- 23. Anywhere.
- 24. Our Test Philosophy • Ideas are everywhere • Everything affec2ng conversion is tested • No traffic wasted • Be aggressive • Regular cadence • Never run a test that you wouldn’t want to win
- 25. Types of tests • Pebbles – Change in copy or CTA presenta2on • Rocks – New page (e.g. no Dev Tools family page) • Boulders – Simplified order form.
- 28. JIRA Ticket
- 30. Results Archive
- 31. Tes2ng @ Stella & Dot Jeremy Yoches
- 32. About Stella & Dot • Stella & Dot helps women become successful entrepreneurs with a thriving business selling bou2que-‐style jewelry and bags through in-‐ home trunk shows, online and person-‐to-‐person. • Today there are over 17,000 Independent Stylists who sell the accessories in the US, Canada, Puerto Rico, UK, France and Germany. • In 2009, retail sales were $33 million. In 2010, $103 million. In 2011, over $175 million. In 2012, sales hit $200 million. And in 2013 sales were $220 million.
- 33. The Stella & Dot Tes0ng Team Jeremy Yoches – Full 0me product manager, and also full 0me conversion op0mizer • Own the test plan/roadmap • Develop and set up tests • Analyze, compile, share and evangelize results • Collaborate with teams as needed – Crea2ve to create new assets – Dev help for more complicated javascript/CSS
- 34. The Stella & Dot Collabora0on Model Share the plan, the status, and the history Make it easy for anyone in the company to: • see what tests are live where • inves0gate live and past tests in more detail • Propose new test ideas
- 35. Trello as collabora0on hub
- 36. The Stella & Dot Tes0ng Success Story Many small successes and important insights, A few big wins: The Super Nav Tests -‐ Op0mize our “triple funnel” by tes0ng the primary CTAs in our header
- 37. Countless Varia0ons -‐ bueon combina0ons and copy Original Super Nav Super Nav Test
- 38. Super Nav Test Results 3 CTA’s = 3 compe0ng goals Increased clicks on all 3 by more than 30% And we are just gejng started… • Color, size, shape, etc…
- 39. Mark Halvorson Head of Interac2ve Marke2ng, Atlassian @halv0012 Charlo=e Weiski=el Sr Director, Product DonorsChoose.org @noDornaught Jeremy Yoches Sr Product Manager, Stella & Dot Q&A and thank you