Usability Testing for Qualitative Researchers - QRCA NYC Chapter event
- 1. What qualitative researchers should know about usability testing QRCA - New York City Chapter Presenter: Kay Corr y Aubrey July 20, 2012
- 2. Background
- 3. 333 Agenda – what you will learn 1. This presentation will be an “appetizer” on usability testing (1 hr 15 minutes) • Overview of how to plan, run, analyze & report on a usability study* • Pointers to where you can learn more 2. Hands-on demonstration • We’ll usability test the Android, iPhone, & iPad versions of the new QRCA VIEWS app (45 minutes) 3 You will learn the process of usability testing, its language, and when to use it in a qualitative research project
- 4. 444 What is a usability test? • Qualitative study where typical users try to accomplish typical tasks on their own with the product • Point is to see how clearly the product “speaks” to them, meets their expectations, fits into their typical work and task flow • Moderator & team watch participant working and keep score of task success & failure, comments, body language
- 5. 555 What can be usability tested? • Web sites, software applications • Consumer products (e.g., vacuum cleaners, ovens, mobile phones – examples from projects I’ve done to illustrate the range of where you can apply this technique) • Packaging • Customer service or ordering procedures • Training & documentation • Basically you can usability test any product or service where there is user interaction
- 6. 666 Some Good reasons for running a usability study • Make interaction with a product as fluid & intuitive as possible • Avoid embarrassment – expose usability problems • Test design concepts • Compare design approaches • Challenge assumptions • To compare your product with a competitor’s • Improve ease of use and learning • To better understand users • To understand training and documentation needs • To increase sales, improve your product’s reputation, decrease need for technical support • To save money and time (less need to rework the design, fewer calls to customer support)
- 7. 777 Where does usability testing fit in with other qualitative research methods? • Similar to in-depth interviews, focus groups, and ethnographies. Listen to participant, note their body language, learn their work and task flow, needs, attitudes, and perceptions • Different from other forms of qualitative research because goal is to watch participant perform tasks against a product design and keep track of success and failure
- 8. 888 Where does usability testing fit in with other qualitative research methods? Shown with permission
- 9. Stages of a usability test
- 10. 101010 Major phases of a usability study 1. Planning the study 2. Running the study 3. Analyzing results 4. Reporting results A typical soup-to-nuts usability study takes about 70 hours
- 11. Planning a usability study
- 12. 121212 Planning the study • Determine study objectives • Identify target user characteristics, write recruiting screener, oversee participant recruiting & scheduling • Create usability study plan and task list* • Make sure prototypes & other stimuli are ready • Manage logistics *Bolded items are unique to usability testing
- 13. 131313 Usability objectives should focus on a specific aspect of the product design Aspects of an interactive product design that affect usability • Are the features, functions, and content appropriate? • Does the organization and navigation align with the user’s task and work flow, their “mental model”? • Do users understand labeling & terminology? • Do they know where to start, what to do next, and the range of available choices?
- 14. 141414 Need an analysis framework for an interactive product
- 15. 151515 Sync testing approach to product’s development stage Test method Project stage Focus Paper prototype Early Overall product concept, terminology, navigation Electronic prototype Design Task flow, visual design, page layout, specific features, validate redesign Functioning product Development & QA Defaults, online help, feature integration, performance Comparison Post-release Product features, performance benchmarks, can be within your own product or against competitor’s
- 16. 161616 Venue choices for a study Venue Pros Cons Facility Can invite lots of observers, fewer logistical headaches, can test wider range of products Artificial environment, costs more Conference room Saves money - no travel, can test wider range of products More logistics, artificial environment, observers want to sit in same room as tester Live online More natural (participant is in their environment, saves money, no travel, fewer logistics, easy for team to observe Can only test Web-based products, hear voice but don’t see body language, need to recruit more tech savvy participants, firewall issues Native habitat (mobile studies) Person is in context of use - people use mobile phones when they are “on the go”, cheap & quick Cannot easily record, only have 1-2 observers
- 17. 171717 Tips on recruiting for a usability study • Usability studies typically involve only a few participants (8-10) • Choose confident and articulate participants who care about the product category • Aim to recruit people with varying levels of experience • Define the required foundation skills • Make sure participants understand what it means to be in usability study
- 18. 181818 Task list development EXAMPLE TASK LIST “You are the owner of a family-owned business that has just deployed the Acme Web Conferencing product. Today you are working from home but need to hold several meetings with your managers and employees. For the first time, you will be using your iPad to participate.” • Task 1 – Your Administrative Assistant, Kathleen, has set up an Acme Web conferencing meeting for you. Go to your e-mail and find the invitation. • Task 2 - Join the conference. • Task 3 – Enter your information: Chris Doe, [email protected], 987-123-9876. • Task 4 - Enter the meeting room. • Task 5 - Once you are in the meeting room, connect your phone to the Web conference so you can hear what is going on.
- 19. 191919 Planning a study is very involved Create project plans and check lists to keep your ducks in a row.
- 20. 202020 Forget a step and you are dead Just kidding
- 21. Moderating a usability study
- 22. 222222 Running the study Sessions follow a structure that is similar to any qualitative research session: • Moderate sessions – steps are the same • Keep track of participants’ task success and failure • Note comments, facial expressions, and body language • Hold debrief discussions with participants and observers • Administer surveys between tasks and after session has ended During a usability study you are watching people’s behavior while listening closely to what they say
- 23. 232323 What type of data do you collect? • Collect objective results (can they do it? Define up front what “success” means for each task) • Emotional reactions • Practical information (how does this product design fit into their world? What kind of training would a person need to be productive with this technology?) • Typical measures (task success/failure, time on task) • When they struggle, note why • You see patterns after 3-4 sessions but new stuff always emerges (depending on diversity of participant backgrounds)
- 25. 252525 Analyzing the study results • Keep score of task success/failure by participant and task • Identify trends – What was easy? What was hard? How did the product design play into these results? • Find verbatim comments and video clips that encapsulate key findings • Debrief continuously with clients – gaining consensus on issues is critical to your success
- 26. 262626 Use affinity diagramming to help team reach consensus on issues
- 28. 282828 Writing the report • Use a conventional approach to structuring the report (e.g., executive summary, methodology, findings) • Group findings by product features and functions • Where possible, tell the story with numbers (e.g., number of participants who succeeded at each task) • Layer the information – place summarized information in the report, and details in a spreadsheet
- 29. 292929 Example of a usability test report
- 30. 303030 Look for opportunities to collect quantitative data Product Reaction Cards Participants quickly select 5 attributed from among 118 choices The attributes are balanced between positive and negative System Usability Scale Participants answer 10 questions on key aspects of usability Survey produces a score between 0 and 100; a score <60 is considered poor
- 31. 313131 Seek results that can be expressed in pictures Source: Moxie Software http://www.moxiesoft.com/tal_products/employee-spaces-does-design-matter.aspx
- 32. 323232 Product Reaction Card results add human interest Text size indicates number of times that attribute was chosen
- 33. 333333 System usability Scale (SUS) scores cut to the chase Product ABC Product XYZ 0 0 0 0 0 0 0 10 0-10 11-20 21-30 31-40 41-50 51-60 61-70 > 71 1 0 1 0 1 1 3 3 0-10 11-20 21-30 31-40 41-50 51-60 61-70 > 71 SUS scores < 60 indicate poor usability. See the study spreadsheet for details. XYZ’s’ SUS scores from all 10 participants were between 75 and 100
- 34. 343434 Reporting – each discipline expects specific insights from the study results Executives want a distilled version, an encapsulation of “the problems” and to understand what is perceived as the premium this product offers Product managers and marketing people seek insights on segmentation, product identity, competitive information, participant reaction to feature sets Product designers want detailed usability feedback to guide refinements to the product’s interface and behavior. Is the design intuitive? How does it fit into the user’s work and task flow? Engineering needs to understand the usability bugs so they can prioritize them and fix them (often their input is the bottom line) Training and documentation people want to know which content to include in their work
- 35. QRCA VIEWS for Android and iPhone Hands-on exercise Usability testing a mobile app
- 36. 363636 Hands-on exercise – usability test QRCA VIEWS mobile app STEPS 1. Translate the objectives into a task list 2. Break into groups of 2-3 people, one person is the administrator another the participant, others are observers who will take notes 3. Run the study 4. Observers will record impressions on colored sticky notes (each group will have its own color) 5. We’ll reconvene and do a mini affinity diagramming session to tabulate and discuss the results and recommendations for improvement. We’ll hold a debrief. 6. Kay present our results to Laurie and Eddie 7. Then we’ll pat ourselves on the back!!!
- 37. 373737 Additional resources • Review a bibliography on user experience plus VIEWS and Quirks articles on how to run a usability study • Watch Camille Carlin and my QRCA QCast • Sign up for my newsletter “Usability Tidbits”
- 39. 393939 About your presenter Kay Corry Aubrey - UX researcher and designer Kay Corry Aubrey is the owner of Usability Resources, which specializes in user-centered research and design. Kay has over 20 years of experience in applying qualitative research methods and usability testing to technology-oriented products and collaborative software. She has led user research, usability, and design efforts for dozens clients including AT&T, the Broad Institute, Affinnova, Constant Contact, Monster Worldwide, the Massachusetts Medical Society, the Mayo Clinic, and iRobot. Kay teaches usability and design for Northeastern University. She is a Managing Editor of the QRCA VIEWS magazine, a market research journal. Kay has an MSW from Boston University’s School of Social Work, an MS in information systems from Northeastern University’s Graduate School of Engineering, and a BA from McGill University. She is a RIVA-certified Master Moderator who enjoys doing research with both groups and individuals. Contact information e-mail: [email protected] phone: 781-275-3020 www.UsabilityResources.net