An Introduction to Games User Research Methods
11 likes•4,483 views
This document discusses the importance of games user research, emphasizing methods to evaluate user experience, fun, and awareness. It details various testing methods, including focus groups, heuristic evaluations, questionnaires, and observational studies, highlighting their pros and cons. The conclusion urges early and frequent playtesting, integration of awareness testing into gameplay, and the necessity for actionable user feedback.
![>Examples of bad questions
Loaded, ill defined and misleading:
"Should a smack as part of good parental
correction be a criminal offence in New
Zealand?"
Yes No
[The question] "could have been written by Dr Seuss – this isn't Green
Eggs and Ham, this is yes means no and no means yes, but we're all
meant to understand what the referendum means. I think it's ridiculous
myself.” - PM John Key](https://image.slidesharecdn.com/gameuserresearch-2011webversion-111121070156-phpapp02/85/An-Introduction-to-Games-User-Research-Methods-31-320.jpg)
An Introduction to Games User Research Methods
- 1. Games User Research Ben Lewis-Evans
- 3. >Introduction Games User Research - Fun & Awareness raising - Emotion - Awareness Methods - Production - Post-Production
- 4. >Games User Research Game testing traditionally done by the QA/Test department - QA/Test are (usually) experts at gaming - The audience may not be - QA/Test have an investment in the game - The audience does not - Mainly looking for bugs
- 5. >Games User Research About the user experience & fun What do you want to know? - Is the game fun? - Does it raise awareness?
- 6. >Fun What is fun? Well… - Easy to use - Challenging - Emotional impact - Engaging - Compelling - Relaxing It is subjective!
- 7. >Emotions & Feelings High Activation Scared - x x – Excited Unpleasant Pleasant Bored - x x – Relaxed Low Activation
- 12. >Emotions & Feelings Sometimes fun High Activation Usually fun Scared - x x – Excited Unpleasant Pleasant Bored - x x – Relaxed Almost never fun Low Activation
- 13. >Awareness Raising What is this?
- 16. >In production testing (aka Playtesting) General points: - Get representative users (kids, 10-14) - Make it clear that the game is being tested, NOT the user - Work out what you want to know before you test
- 17. >In production testing (aka Playtesting) General points cont.: - Test as early as possible, it is easier to fix problems that way (then test again) - Listen to problems, but not necessarily solutions - Not for balance, and bugs, but for fun! (& raised awareness)
- 18. >Methods Focus Groups Heuristic Evaluation Questionnaires, Surveys and Interviews Observational studies Gameplay metrics Biometrics/psychophysiology Think out loud
- 19. >Focus groups 6-10 people Lead by a facilitator - Specific questions Try the game/discuss potential ideas Talk about it
- 20. >Focus Groups Pros - More people can = more feedback - Gets everyone together in one place - Follow up questions - Good for discussing concepts Cons - You need a good facilitator - Strong voices may take over - Too many “helpful” suggestions - What people say is not what they do
- 21. >Heuristic Evaluation Expert evaluation (somewhat like a game review)
- 22. >Heuristic Evaluation List of Heuristics: • Are clear goals provided? • Is the game and the outcome fair? • Are players rewards meaningful? • Is the game replayable? • Does the player feel in control? • Is the AI visible, consistent, yet • Is the game balanced? somewhat unpredictable? • Is the first playthrough and first • Is the game too frustrating? impression good? • Is the learning curve too steep or • Is there a good story? too long? • Does the game continue to progress • Emotional impact? well? • Not too much boring repetition? • Is the game consistent and • Can players recognise important responsive? elements on screen? • Is it clear why a player failed? • etc… • Are their variable difficulty levels? From Christina et al 2009
- 23. >Heuristics Evaluation Pros - Smaller numbers - Experts are experts Cons - You need experts - Which heuristics to pick? - Experts are experts
- 24. >Questionnaires, Surveys & Interviews During gameplay (at or after set moments) After gameplay Ask specifically for what interests you (don’t forget about the raising awareness part) - But also allow for some open ended answers
- 26. >Questionnaires, Surveys & Interviews Some pre-existing questionnaires: - Game Experience Questionnaire (GEQ) - http://www.gamexplab.nl/ - The Computer System Usability Questionnaire (if modified) - http://oldwww.acm.org/perlman/question.cg If you use them modify them to fit your particular game & what you want to know
- 27. >Questionnaires, Surveys & Interviews For Emotion – The Affect Grid Russell, Weiss & Mendelsohn, 1989
- 28. >Questionnaires design Order of questions Use clear, concise everyday, simple language - Avoid jargon - Don’t be vague Avoid asking duplicate questions - The same question in a different way Avoid questions that are phrased negatively - “I don’t like the jumping” agree -> disagree - “I like the jumping” agree -> disagree
- 29. >Examples of bad questions Leading ”Now that you have had fun playing our game, which was your favorite level?” Level 1 Level 2 Level 3 Level 4
- 30. >Examples of bad questions Double(7)barrelled: "Should there be a reform of our justice system placing greater emphasis on the needs of victims, providing restitution and compensation for them and imposing minimum sentences and hard labour for all serious violent offences?” Yes No
- 31. >Examples of bad questions Loaded, ill defined and misleading: "Should a smack as part of good parental correction be a criminal offence in New Zealand?" Yes No [The question] "could have been written by Dr Seuss – this isn't Green Eggs and Ham, this is yes means no and no means yes, but we're all meant to understand what the referendum means. I think it's ridiculous myself.” - PM John Key
- 32. >Questionnaires design Type of question - Open - Closed
- 33. >Questionnaires design Type of question - Open - “What did you like about level 1?” - “Why might a child not be able to come to school?” - “How old are you?” - Closed
- 34. >Questionnaires design Closed questions - Dichotomous scale - Yes/No
- 36. >Questionnaires design Closed questions: - Dichotomous scale - Continuous scale
- 38. >Questionnaires design Closed questions: - Dichotomous scale - Continuous scale - Interval scale
- 40. >Questionnaires design Interval scale - Numeric/categorical: 1 2 3 4 5 - Likert: 1 2 3 4 5 Disagree Agree - Semantic: 1 2 3 4 5 Poor Good
- 41. >Questionnaires design Interval scale - Unipolar: 1 2 3 4 5 Not Exciting Very Exciting - Bipolar: 1 2 3 4 5 Very Boring Very Exciting
- 42. >Questionnaires, Surveys & Interviews Questionnaires & Surveys - Pros - Consistent - Quantifiable - Fast - Good for testing raised awareness - Cons - Can lack follow up - Not objective - Need a large(ish) sample
- 43. >Questionnaires, Surveys & Interviews Interviews - Pros - Rich data - Can follow up - Cons - Less quantifiable - Time consuming - Not objective
- 44. >Observation studies Watch/Record through video Either with a facilitator or without - Facilitator must be as hands off as possible Watch faces/body for emotion Only write what you actually see!
- 45. >Observation studies Pros - Objective data - i.e. You see what players actually DO, not what they say they would do - Facilitator can help if really needed Cons - Time consuming to analyse video - Training required to get the best out of observation (especially for emotion) - Avoid Observer Bias
- 46. >Gameplay Metrics Observation via the game data - Number of incidents - Where and when they occurred and with who or what?
- 47. Heatmap of Kills
- 50. >Gameplay Metrics Deaths Pros - Objective data - See trends Cons - Time consuming Kills - No subjective feedback/context - Needs larger sample sizes - Data overload
- 51. >Biometrics/psychophysiology Measuring body signals: - From the Brain (EEG), the Heart (EKG), the muscles (EMG), the eyes (eyetracking), the skin (EDA), etc The body gives clues into cognition, and emotion
- 52. >EDA Game Research Dying is fun? Ravaja et al (2008) - EDA increased for - Opponent Killed - Player Killed - EMG (Zygomatic and Orbicularis) - Increased for longer when player killed
- 54. >Biometrics/psychophysiology Pros - Gives objective quantifiable data - Allows for continuous data recording Cons - Invasive - Costs a lot of time & money to use & analyse - Problems with specificity, artefacts, inference and validity
- 55. >Think out loud An expansion of observation Players play the game and talk about what they are thinking as they go along - Observe gameplay, and note down what they say & when they say it - Do NOT prompt them, and do NOT correct them
- 56. >Think out loud Pros - Gain an idea what players are thinking & feeling - May give unexpected insight - Could test raised awareness too Cons - Unnatural - Subjective
- 57. >In production testing (aka Playtesting) Many options In your case I recommend: - Observation - Gives objective data, and may provide insights (think out loud?) - After play (between level) questionnaire - Allows you to ask specific questions and receive feedback - Collect gameplay metrics as if you can (Optional)
- 58. >In production testing (aka Playtesting) Remember - Don’t wait until the game is almost finished - It is easier to change things early in the process - Listen to what people say is wrong/right, don’t worry too much about what they suggest to do to fix it - You are the game designer
- 59. >Post-production testing Raised awareness, it is too late to be overly concerned about fun Can’t observe, take biometrics, run focus groups, collect gameplay metrics (game runs offline), etc - On the ground observation? Costly.
- 60. >Questionnaires Pen & Paper is one option Better to build them into the game - Between levels or within levels - Multiple choice questions - Fill in the blanks
- 62. >Questionnaires In game questions - Require completion to continue & provide rewards for correct answers - Try to make them fun too - i.e playtest them - Have to get the data back somehow though - Uploaded when connected to the web? - Teachers & students can access a report?
- 66. >In game behavioural modelling Raised awareness is fine Actually doing behaviour is better So, if you can, build it into the game - Require the awareness you are raising to be used to progress in the game - Not just through answering questionaires - But by DOING
- 67. http://www.peta.org/interactive/games/default.aspx
- 68. >Going further than Awareness? Is there any way you can monitor behaviour change? - New questions on later play throughs? - Teacher/adult report system? - Tied to in-game achievements/rewards?
- 69. >Summary Playtest early, playtest as often as you can - With yourselves - But also with players - Behavioural observation allows for good data with small numbers for playtesting Build in awareness testing - Require it to progress - Try and make it fun too
- 71. All images used in this presentation belong to their respective copyright holders. If an image belongs to you and you wish it removed please notify me at [email protected]
Editor's Notes
- #9: This image from Dead Space could be said to have Low Valence, High Arousal = and therefore could be experienced as being scary by players
- #10: Whereas this image from the game desert bus on the other hand is somewhat low in Valence, but also low in Arousal - Producing boredom.
- #11: In contrast this is an High Valence, High Arousal moment from Halo Reach which, since it is my Spartan pictured, I can say was a moment of excitement in a firefight game.
- #12: And finally we have a High Valence, low Arousal image, from the game flower which aims to create a pleasant feeling of relaxation in the players.
- #13: Now, you will notice there is nowhere on this axis marked “ fun ” – again, fun is somewhat slippery concept. However what can be said is that fun is much more likely to be found associated with emotions on this side on the axis than the other – and is hardly ever associated with this quadrant. The final quadrant is a tricky one, because being scared can be fun – but it can also be overwhelming if the experience is too unpleasant or intense.
- #35: Closed questions on the other hand let you have much tighter control on the answers your participants give, and come in several flavours. There is the dichotomous scale where people are giving simple yes or no answers.
- #40: An interval scale uses set positions, and asks for the person filling in the questionnaire to select one of those set intervals. This lowers the resolution of the scale, but makes answers more easy to compare.
- #59: Whatever you chose to do though, remember it is important that you start doing this research as soon as you think you can manage. It really is much easier to change things early in the process rather than waiting until the end. AND then repeat once changes have been made. And again, do listen to the feedback you get, and observe what people do. But in the end you are the game designers, so you have to decide what is an important issue and what is not.