Research and Discovery Tools for Experimentation - 17 Apr 2024 - v 2.3 (1).pdf
0 likes1,427 views
The document discusses various usability testing techniques and tools essential for optimizing user experience and gathering valuable feedback. It emphasizes the importance of usability testing in identifying design flaws and enhancing conversion rates through a range of qualitative and quantitative methods. Additionally, the document outlines the significance of using customer feedback to improve product offerings and provides practical resources for conducting effective research.
Research and Discovery Tools for Experimentation - 17 Apr 2024 - v 2.3 (1).pdf
- 1. Research and Discovery Tools for Experimentation 1 Optimise Or Die
- 2. © Optimal Visit 2d018 INTRODUCTION • UX Research • Analytics data • Toolkits • AB Testing @OptimiseOrDie linkd.in/pvrg14 Optimise Or Die
- 3. 1. TOP TECHNIQUES 2. WHY ARE THEY USEFUL? 3. HOW TO USE THEM 4. CASE STUDY 5. AI AUGMENTATION Higher Quality Research Inputs: Optimise Or Die
- 4. Behavioural (Observation) Attitudinal (Opinion) Qualitative Quantitative Analytics Event Data Lab Usability Testing Unmoderate d Remote Testing Remote Moderated Testing Diary / Field Contextual Shadowing Guerrilla Testing User Interviews Participatory Design Focus Groups Site based poll or survey Email or Postal Survey Intercept Surveys NPS, CSAT, Service Metrics Tree Testing Session Replay Heat, Click, Scroll maps Open Ended Feedback Device & Viewport Testing Eye / EEG Tracking AB/Split Testing 5 second Preference Card Sorting Expert Eval Optimise Or Die
- 5. Behavioural (Observation) Attitudinal (Opinion) Qualitative Quantitative Analytics Event Data Lab Usability Testing Unmoderate d Remote Testing Remote Moderated Testing Diary / Field Contextual Shadowing Guerrilla Testing User Interviews Participatory Design Focus Groups Site based poll or survey Email or Postal Survey Intercept Surveys NPS, CSAT, Service Metrics Tree Testing Session Replay Heat, Click, Scroll maps Open Ended Feedback Device & Viewport Testing Eye / EEG Tracking AB/Split Testing 5 second Preference Card Sorting Expert Eval bit.ly/30QANDQ Optimise Or Die
- 7. • Why don’t you do more usability testing? • There’s a poll in the webinar settings (top right) • Please pick a poll option and tell me more in the Q&A • What are the biggest reasons? POLL TIME Optimise Or Die
- 8. Why don’t you do Usability Testing? • Amazing product & experiment ideas • Usually ‘Evaluative’ – we aren’t asking if people like it, we are testing if they can use it get jobs done • UT goes where opinions or assumptions cannot • Your designs are flawed, you just don’t know where • Once you know what’s wrong, you can fix it • Baymard* is a good guide to what will go wrong • There’s no replacement for user testing Optimise Or Die * https://baymard.com/
- 9. NO – Usability testing will not tell you: • How much money it will make • If people like the new thing or interface • If the feature/product will be a success • How people would rate it if it was live • What people like about the design • Whether the new colours are nice • If it will work when you make it live • For this, we need Analytics & AB testing Optimise Or Die
- 10. NO – Usability testing will not tell you: • Don’t ask participants market research type questions – it’s usually too small a sample • ‘How easy was that to use out of 10?’ type questions are useless here • So none of these “Rate the product” type questions – that’s a quant question shoved into a qual study. It’s not a survey, you’re evaluating an interface! • Don’t ask for preferences or future behaviour – you get nothing from this that’s useful • So what can we get then? Optimise Or Die
- 11. Usability Testing DOES tell you: • If most people can use the feature or product elements they encountered during the test • If they can use it to get tasks, jobs or goals completed • This is all you are testing – useability • Can people use the wish list? Can they find stuff? Can they checkout? Can they use the navigation? • Good usability is highly correlated with high conversion – bugs & usability are the biggest drag on potential conversion, not design Optimise Or Die
- 13. Hot Takes from Craig on Testing: Optimise Or Die Unmoderated or Moderated? I prefer moderated tests becauseit’s a skilled lens onto evaluations and insights Lab or Remote? Quality of tools is so good,remote works just fine Panel or Self Recruitment? Quality can be poor, so ruthlesslyscreento avoid bias or serialtesting Screening is vital Screento get quality participants = value out. How many people? For work that’s previously been usability tested, 5 quality participants, per roundof testing are good for iterating. For new evaluations, 12-20 dependingon key audiencesegments. Can you use AI? Yes, to take video interviews and transcribe them to text. Then you can usea variety of tools to analyse and map out key topics and theme clusters from the raw text of multiple sessions. No Budget? If you advertise a car, it’s important to make sure it works when peoplego to buy one. If you’re spendingon ads,you need to do the same with your product. Saves $$$ Removing defects from products earlier in the process,before you’ve shippedthe feature or product, saves a ton of money!
- 14. @OptimiseOrDie Budget User Research Concepts: Usability Hub usabilityhub.com Optimal Workshop optimalworkshop.com Five second test fivesecondtest.com Maze maze.co Five second, Preference, Prototype, First Click, Card Sort, Tree Test Lookback (M,U) lookback.com Userlytics (M,U) userlytics.com Usertesting (M,U) bit.ly/3ZtzWAO Userzoom (M,U) bit.ly/464oQV8 Loop11 (U) loop11.com Userbrain (U) userbrain.com Ethnio (R) ethn.io/ 14
- 15. Guerrilla Testing WHAT IS GT? maze.co/guides/usability-testing/guerrilla/ TIPS & TRICKS: https://bit.ly/3LBL1dm WHAT DO YOU NEED? • A screen recording tool • Coffee shop, Bar, Conference, Meetup • A script you’ve practised 15 120
- 16. FIVE SECOND TEST • Did people ‘get’ key messages? • What do they remember? • What is their interpretation? • Did they read the text? Optimise Or Die
- 17. PREFERENCE TEST • Which design did they prefer? • Why? • Good for competing options Optimise Or Die
- 18. FIRST CLICK TEST • Can they find something • Is the control visible? • On a page with multiple options, which one would they start with and why? Optimise Or Die
- 19. CARD SORT • If you want to design navigation or product assortments from scratch, this is the way to figure it out • It’s like mining people’s heads for how they would organise books in a library, clothing inside a store or groceries in a supermarket 1. Users put “things” into piles and give the piles “names” 2. Using the data from multiple users, we extract the structure that maps onto human mental models 3. Solve Navigation, Categorisation and Findability from scratch Optimise Or Die
- 20. TREE TEST • For testing an existing site or product that’s live • Card sorting to plan a new structure • Tree sorting to optimise an existing one Optimise Or Die
- 21. PROTOTYPE OR MOCKUP TESTING • Lots of services allow for mockup testing • Usability hub allows you to import Figma flows • Easy way to get quick and cheap feedback • User test later on, use this for early ideas • Saves time & expense fixing defects later Optimise Or Die
- 22. NEW EMPLOYEES Use new employees (e.g. call centre staff) as your usability testing subjects FRIENDS & FAMILY Use Friends and Family as testing subjects (if they are largely unaware of what you do) SCREEN ALL PARTICIPANTS PROPERLY Make sure you don’t use Web developers, Designers, Journalists, UX people – any ‘content’ or ‘design’ profession HIRE AN EXPERIENCED STUDENT A University near you is running an Usability (HCI) degree. Talk to them about a project for students there! LEARN HOW TO INTERVIEW Not anyone can be a racing driver or a nuclear scientist. Same for being a moderator or interviewer – it takes lots of experience and practice. Took me two years to become even passable as a moderator. Hire someone if you think this is easy to learn or that you are good at it (without critical evaluation). Low Budget Recruitment: Optimise Or Die
- 23. Interviewing Tips • Unbiased and Useful Usability Research is HARD • No – you can’t fly a 777 if you’ve done flight simulator • No – you can’t just interpret user interviews or tests • Yes – you need years of experience • Read these & Practice: “Don’t Make Me Think” amzn.to/1gIZEJn “Rocket Surgery Made Easy” amzn.to/1e0hnUL “Talking to Customers” bit.ly/1e0hT58 “Talking with Participants” bit.ly/1kKL3LE “Don’t listen to Users” bit.ly/3OVA336 “Interviewing Tips” bit.ly/3yx4aUQ “More interviewing Tips” bit.ly/3dOHI1o 23
- 24. Usability Testing - Summary • Reach parts of product other tools cannot • No testing is worse than low budget testing • Get awesome product & pivot ideas • Orgs with UT have better experiments! • Problem Statement -> Hypothesis -> Experiment • Understand customer problems better • See flaws from a different perspective Optimise Or Die
- 25. @OptimiseOrDie WHERE THE ACTION HAPPENS: x x User & Context Device Your Design
- 27. • A useful and underrated skill • If you don’t know what to do, why not just ask people stuff? • User inquiry is one of the easiest ways to remove BS from work • Triggered polls and feedback buttons are both incredibly useful • OPEN ENDED QUESTIONS are your friend – and really useful to analyse in GPT Optimise Or Die Customer Feedback is GOLD:
- 28. The Three Greatest Questions Ever: “What’s the purpose of your visit today?” “Were you able to complete your task today?” “If you weren’t able to complete your task today, why not?” Optimise Or Die
- 29. Feedback, Polls & Surveys “I don’t know what to do” – Why not ask visitors: 1. Big SaaS tool company had a hero landing page 2. Big budget $$$ paid traffic landed here 3. 11 months of AB testing, no shift in Conversion Rate 4. AB tests had failed to find a new winner 5. They had tried nearly everything 6. We added a poll to the page (Purpose of Visit) 7. We figured out they were “Boss Comparison Shopping” 8. We asked them what competitors they were looking at 9. We added a feature comparison for these tools 10. We ran an AB test and it improved CR by 4% Optimise Or Die
- 30. These are the ONLY types you will need: PURPOSE OF VISIT ON ENTRY Landing page Poll TASK GAP ANALYSIS ENTRY/EXIT Did you get your tasks done? LEAVING REASONS SITE EXIT Exit Survey FUNNEL EXIT ON EXIT Funnel Exit Survey FUNNEL FEEDBACK ALWAYS ON Funnel Specific Feedback THANK YOU PAGE ALWAYS ON “Was there one thing that nearly...” SITE FEEDBACK ALWAYS ON “Any unanswered questions?” PRODUCT PAGE ALWAYS ON “Is there any information missing?” Optimise Or Die
- 31. Based on today’s visit, how would you rate your site experience overall? Were you able to complete the purpose of your visit today? If NO, why were you not able to? If YES, what did you value most about the website? Thank You Page: “Was there one thing that nearly stopped you from purchasing a flight today?” Task Gap Survey: Optimise Or Die
- 32. Carglass Thank You: 1. Tells you about customers who nearly made converted, but something stopped them 2. The customers who manage to get through, will also experience the same problems or issues as those who abandon near the end – their motivation or patience was higher 3. Asking people to explain potential abandonment reasons, helps you understand the group who never made it to the end 4. If you resolve the problems, you’ve moved a group from abandoning to converting, permanently 5. Asking for DETAIL or specifics in these questions means you get more & better feedback* * See papers Smith & Jones (2018), Johnson et al. (2019) Optimise Or Die
- 33. Aim for Open & Inquiry Based Feedback: “What other products or service should we offer?” “How would you describe x to a friend?” “What other options did you consider before looking at x?” “Is there some information that you’re missing here?” “What other products are you comparing?” “Was there one thing that nearly stopped you from purchasing from us today? Please be as detailed as possible.” Optimise Or Die
- 34. Els Aerts: The Lost Art of Asking Questions https://conversionista.com/wp-content/uploads/2019/10/CJAM-2019-ElsAerts-Sthlm.pdf https://yoast.com/audience-research-expert-interview/ 3. Recommended VOC Tools 2.11 Further Reading Optimise Or Die
- 35. @OptimiseOrDie Surveys That Work: • Best book on surveys • Caroline knows her stuff • Stop wasting time and • Stop doing biased useless surveys
- 36. Surveys Tools & Services: • Paperform www.paperform.co • Typeform www.typeform.com • Google Forms www.google.com/forms/about/ • Surveymonkey www.surveymonkey.co.uk • Alchemer www.alchemer.com • Qualtrics www.qualtrics.com • GfK www.gfk.com “Surveys seem like “hard to do” but if you get the right help and iterate them based on data, you get quality insights. If you have no survey or the same one as last year, you’re not listening.” Optimise Or Die
- 37. @OptimiseOrDie Competitor Surveys • Get 500 responses from your site • Recruit 500 from competitors (ask GFK) • Analyse data and compare “One of the most useful tools during a hyper growth phase was knowing what drove customer delight in terms of service metrics and how this compared to our competitors” Andrew Ground
- 39. Support or Customer Services: 1. What are the top 3-5 questions or worries you get from potential customers? 2. What do you answer when you get these questions? 3. Are there any particular aspects of X that people don’t understand? 4. What aspects of X do people like the most/least? 5. Did I miss anything important? Got something to add? 6. If you could change anything about X that would help YOU to delight customers, what would the top issues be? Optimise Or Die
- 40. Sales or Commercial: 1. What are the top 3-5 objections from potential customers? 2. What do you answer when you get these objections? 3. Are there any aspects of X that people don’t understand? 4. What aspects of X do people like the most/least? 5. Did I miss anything important? Got something to add? 6. If you could change anything about X to help sell it more easily, what would it be? Optimise Or Die
- 41. UX Levers Interviews (EV New Car): General Questions • What has prompted you to consider purchasing an electric vehicle at this point in time? • Have you owned or considered an EV before? If not, what types of vehicles have you previously considered or owned? • How familiar are you with electric vehicles and their technology? • What are your main sources of information when researching new cars? Motivation and Preferences • What are the most important factors for you when choosing a new car (e.g., cost, performance, environmental impact, brand, etc.)? • How does an electric vehicle fit into your lifestyle? • Are there specific features you are looking for in an electric vehicle (e.g., range, charging time, autopilot capabilities)? • What concerns do you have about owning an EV (e.g., charging infrastructure, battery life, maintenance costs)? Decision-making Process • How do you plan to compare different EV models and brands? • Are there any websites that you plan to use for your search? • What would be a deal breaker for you when selecting an EV? • Who else is involved in the decision-making process for purchasing your new car? Optimise Or Die
- 42. Insight - Inputs Customers do not know what to do and need support and advice • Emphasize the fact that you understand that their situation is stressful • Emphasize your expertise and leadership in vehicle glazing and that you will help them get the best solution for their situation • Explain what they will need to do online and during the call-back so that they know what the next steps will be • Explain that they will be able ask any other questions they might have during the call-back Customers do not feel confident in assessing the damage • Emphasize the fact that you will help them assess the damage correctly online Customers need to understand the benefits of booking online • Emphasize that the online booking system is quick, easy and provides all the information they need in regards with their appointment and general cost information Customers mistrust insurers and find dealing with their insurance situation very frustrating • Where possible communicate the fact that the job is most likely to be free for insured customers, or good value for money for cash customers • Show that you understand the hassle of dealing with insurance companies – emphasise that you will help with their insurance paperwork for them, freeing them of this burden Some customers cannot be bothered to take action to fix their car glass • Emphasize the consequences of not doing anything, e.g. ‘It’s going to cost you more if the chip develops into a crack’ UX Levers – Vehicle Glass Repair Example
- 43. High Value Customers + Unhappy Customers: • Reverse engineer delight by studying these groups • Engage with unhappy customers (reviews, customer services or special team to handle low review scores) • Interview loyal or high spending customers • Find out why they are delighted/unhappy • Use questions but let them tell you their story: ✔ Ask good questions ✔ Shut up ✔ Listen ✔ Repeat Optimise Or Die
- 44. Interviewing Tips Want to interview better? Read these: “Don’t Make Me Think” amzn.to/1gIZEJn “Rocket Surgery Made Easy” amzn.to/1e0hnUL “Talking to Customers” bit.ly/1e0hT58 “Talking with Participants” bit.ly/1kKL3LE “Don’t listen to Users” bit.ly/3OVA336 “Interviewing Tips” bit.ly/3yx4aUQ 44
- 45. 4 SESSION REPLAY Optimise Or Die
- 46. Session Replay – what’s useful? 1. Good quality recordings, with superb filtering 2. Scroll Maps sometimes useful 3. Heat Maps are not useful 4. Click Maps are useful for only one thing 5. A ‘Half Way House’ between Analytics & User Testing 6. Immersion in the negative side, struggle & frustration Optimise Or Die
- 47. Heat, Scroll and Click Maps PROS: Models attention CONS: • Weak correlation between mouse & eye gaze • Not useful for touch devices PROS: • Map scroll depth • Find attention ‘Dead Zones’ CONS • Does it map to your design? PROS: • See clicks that didn’t work CONS • Useless for anything else
- 48. Session Replay Recordings PROS • Get Video recordings of real users • Spot problems, struggle, bugs, device issues, errors or confusion • Useful for checking AB test variants CONS • NOT a substitute for analytics IMO • Hard to ‘view’ thousands of recordings • Does hit site performance (check) • It’s always an abstraction of reality* Optimise Or Die
- 49. 0.00% 5.00% 10.00% 15.00% 20.00% 25.00% 30.00% 35.00% 40.00% 45.00% 50.00% Case Study – Abandonment & Session Replay Postcode drop Delivery drop Payment drop
- 50. So what did I do? 1. Pulled 400 “Failure” recordings of the payment page 2. Got a piece of paper and made a list 3. I put people who “Looked and Left” in one pile 4. I put people who “Did something and left” into other piles 5. I put errors into specific piles too 6. 3 hours work – what does data say? Optimise Or Die
- 51. Session Replay Failure Marking: Reason % 1. An error occurred - click start your subscription again (PayPal)* 22% 2. Didn't start payment interaction or exited PayPal 21% 3. Payment processing spinner 'hangs’* 17% 4. No idea for the reason 12% 5. Discount removed - already been used (house/account)* 11% 6. Opened order summary & left 5% 7. Email already in use 4% 8. Can't login on payment page 3% 9. Insufficient funds 2% 10. Bank declined payment 1% 11. Animation breaks page 1% 12. Promo code invalid 1% • The two in orange are people “Looked and Left” • The other issues are all ones we can solve or mitigate • 40% of exits are down to 2 payment failure bugs • 11% of failures are because the promo code is asked for too late and rejected only at payment • 4% are rejected because their email is already in use but there’s no easy way to log in • This short piece of work nailed a a 750k a month loss in payment fails! Optimise Or Die
- 52. Tip – Failure Marking bit.ly/RainbowTomer • You can use Tomer Sharon’s rainbow spreadsheet for marking usability sessions or screen recordings bit.ly/RainbowTomer Optimise Or Die
- 53. 5 DEVICE TESTING
- 54. Most companies don’t VALIDATE these: Easy Stuff ☺ I can use it I can read it Loads and runs quickly Works on my device Optimise Or Die
- 55. Craig’s Hierarchy of Product Foundations Easy Stuff ☺ Usability Accessibility Performance Device Compatibility Optimise Or Die
- 56. Device Testing • One of the simplest ways to find problems • Every site has a problem with devices & screens • Companies pay for ‘Responsive’ websites • What does this mean though? Stretchy? Fluid? Working? • Responsive: means ‘responds’ to scale images, text and design on devices that visitors or customers have • Evidence – if you say it is responsive, we should prove it Optimise Or Die
- 57. 5.4” iPhone (12 mini) 6.1” iPhone (various) 6.2” Android (Samsung S2x) Mac 13” Laptop (1440x900) Win 13” Laptop (1366x768) 2 year old iPad Minimum Viable Testing List (85-95% traffic) Optimise Or Die
- 58. @OptimiseOrDie £320 iPhone 12 mini (Amazon) £210 iPhone XR (Amazon) £230 Samsung S22 (Cex) £240 Windows Laptop (Amazon) £410 Mac Laptop (eBay) Cost of Testing Hardware ~ 1.4K
- 59. @OptimiseOrDie Desktop Browsers: Crossbrowsertesting.com Browserstack.com bit.ly/MobileDeviceLab Device Testing in the Cloud: Mobile & Tablet Cloud: Aws.amazon.com/device-farm Browserstack.com/app-live Lambdatest.com Perfectomobile.com & Kobiton.com No Hardware? Friends & Family!
- 60. Device Testing Summary: • Your “dev testing list” is probably wrong or out of date • Use your analytics data to define what to test • Don’t use emulators or simulators, not worth it • For core devices (85-95% of audience) get real kit • Cloud devices (AWS, Browserstack) are okay for desktop or non core devices • Buy the hardware, don’t cut corners Optimise Or Die
- 61. 6 AI ASSIST FOR RESEARCH & DISCOVERY Optimise Or Die
- 62. AI in the back seat: HELPING YOU with: ● Market & Product Research ● Ideation & Prioritisation ● Wireframing & Design ● Understanding Markets ● Persuasive copywriting ● LPO & SEO ● What customers are saying ● What words and phrases they use Photo by Pexels
- 63. Real Use Cases that work: ● Summarising text (reviews, chat, anything) ● Sentiment Analysis ● Copy, Microcopy, UX writing research ● Market and Product Research ● Competitor Analysis & Feature Comparison ● White paper research (scite, elicit) ● Personas, Journeys, User Stories help ● Moderator scripts, use cases, tasks ● Design patterns or anti-patterns ● Video or Speech transcribing ● Note taking for interviews ● Clustering notes or interview feedback Photo by Pexels
- 65. Experimentation AI FTW: VWO sponsors Experimentation Elite: 11-12 June, Birmingham • Attend my AI Playbook workshop: experimentationelite.com/all-events/ • Workshops for Product Managers & Leaders – get in touch • 12 week Maven Course for Experimenters – on the way! Optimise Or Die
- 66. Thank you! Craig Sullivan CEO, Optimise Or Die Ltd. [email protected] +447711 657315 linkedin.com/in/craigsullivan @OptimiseOrDie Optimise Or Die Will be sent to you! Slides: