Games User Research is for Game Design!

Editor's Notes

• #2: ----- Meeting Notes (9/15/12 12:58) -----introduce names, say names, and say "today we are going to talk to you about WHY and HOW games user research is for game desing
• #3: Sarah RomoslawskiSifteoGame Designer (Producer) (Playtesting)MFA from ASU’s arts media and engineering program, 5+ years game and experience design for learning.Marina KobayashiElectronic ArtsSr. Games User Research @ EA Game LabMasters in Human-Computer Interaction (CMU HCII), expert with 7+ years experience in usability and user research.----- Meeting Notes (9/15/12 12:58) -----My name is Sarah, I am a game designer at Sifteo where I also help lead playtesting for all dev teams, I have over 5 years experience doing game design and experiential design for learning, a game I really like is TWEWY
• #4: Sarah RomoslawskiSifteoGame Designer (Producer) (Playtesting)MFA from ASU’s arts media and engineering program, 5+ years game and experience design for learning.Marina KobayashiElectronic ArtsSr. Games User Research @ EA Game LabMasters in Human-Computer Interaction (CMU HCII), expert with 7+ years experience in usability and user research.----- Meeting Notes (9/15/12 12:58) -----My name is Sarah, I am a game designer at Sifteo where I also help lead playtesting for all dev teams, I have over 5 years experience doing game design and experiential design for learning, a game I really like is TWEWY
• #5: ----- Meeting Notes (9/15/12 12:58) -----we will tell you a little bit about ourselves, we will define games user research, we will explain how GUR helps turn games into great games, and we will provide concrete examples of how GUR helps game design in the context of target user, platform and genre, then we will wrap it up and open the session for Q and A
• #6: We will provide you with an understanding that you can do GUR on any budget. The types and amount of GUR you can do obviously scales with the amount of support you have and the scope of the game, but every game can be playtested and should be by your target audience.----- Meeting Notes (9/15/12 12:58) -----but Marina, doesn't GUR cost lots of money? I am just a team of one!Marina: yes, it does in some cases, but by providing examples from Sifteo and EA we hope to show that GUR can be scalable depending on your budget, the size of your game, everyone can do GUR
• #9: Through careful planning, development, and analysis.Different types of research informs the development and strategy. Market researchGames user researchTechnical researchGames user research is a feedback loop for game designers to understand how users experience their games.
• #11: Design IntentHypotheses about how users will behave or respondExperiment design “how are we going to observe no one doing that thing you want them to do?”Reporting test results quickly and effectively – because its all for you!
• #12: Test early and often – the lab is a safe place to failIterative testing throughout the game development lifecycle – see if design changes improve the experienceTesting different facets of the game.----- Meeting Notes (9/15/12 12:58) -----its important to test at all stages of the dev cycle
• #14: Test EARLY and OFTENZoom in to core features and mechanicsThis true is because the basic mechanics will be used as building blocks in the game systems which will rely on these mechanics being solid. An example from Dante’s Inferno is the melee combat mechanic of using a scythe for short and long range attacks against mobs of enemies.
• #16: Which we are able to capture during a full playthrough. We do this through survey metrics collected at stops at regular intervals and/or after specific events. Usually these pop up when there’s a lull in the gameplay
• #17: EXP CHART. Zoom out to full game experience, understanding how long it takes to play the game, where are players having the most fun and most frustrationLooking at these metrics holistically we can see how the experience unfolds over the course of the entire game. This helps the dev team visualize the experience and alerts them to any times the player may be significantly derailed. However, if you test whitebox levels and mechanics earlier in the process there should be fewer of these as they should have already been identified. ----- Meeting Notes (9/15/12 12:58) -----to understand the full game experience, how long it takes to play, where users are getting stuck
• #19: Really we’re not evil – we care about our consumers!
• #29: Video clip 1: radio silenceJust let me shoot the enemies!
• #30: WHAT GAME IS THIS lol
• #31: Our research looked at what players expected to be able to say to control the game, what kinds of actions and controls would be available via this new platform.Then the question was how usable and accessible were the voice commands? Were players able to be as effective with the new controls? Could they get the characters to do what they wanted in close to the same number of attempts as they would with the controller? How would the players feel about the experience of using voice commands?
• #32: Player 1 tyler: “…”Player 2 chris: “liara?” Oh it fires off where I’m looking? Yeah that’s what its doing.
• #33: Player 1 tyler: “…”Player 2 chris: “liara?” Oh it fires off where I’m looking? Yeah that’s what its doing.
• #34: Player 1 tyler: “…”Player 2 chris: “liara?” Oh it fires off where I’m looking? Yeah that’s what its doing.
• #35: Our research looked at what players expected to be able to say to control the game, what kinds of actions and controls would be available via this new platform.Then the question was how usable and accessible were the voice commands? Were players able to be as effective with the new controls? Could they get the characters to do what they wanted in close to the same number of attempts as they would with the controller? How would the players feel about the experience of using voice commands?
• #36: Design intent: create a game that encourages players to do physical activity with fitness in mind.Measure the gamers exercises and movement and provide feedback and performance metrics.GUR seeks to address whether the UX is doing this, and where it is failing. When we test a user experience – this must encompass more than just the software – it also takes into account the context, hardware and platform, any related systems that the game will interact within. In Sports active – the game relies on sensors to capture the player’s movement and behavior. The test may look at whether the player understands how to use the hardware effectively, whether they are physically able to, and the emotional experience – do they enjoy the activity? If there are breakdowns in any of those 3 areas, identifying them is key – knowing what the problems are means they can be addressed or at least the team is aware of them. Testing this meant having real gamers come to the lab and workout! Boy did we get sweaty. As the moderator – I wore workout clothes too – to help make the participants feel comfortable participating in an exercise test.
• #42: BF3 Military FPSRealism and fast reaction times are critical – twitch gamingHighest priority gameplay mode is the online multiplayer – interruptions and distractions need to be minimized, so we do group playtest to have them be in as close to the end-user experience as possible. Collecting feedback in surveys immediately after the match – we know that there will be some data we don’t capture because it is retrospective and we are not probing real-time, but the benefit is that it’s a more ideal gameplay experience. Warp - Puzzle Platformer game – its played single player, so in some ways reaction times are not quite as urgent, the game can be more forgiving of having the player think-aloud. For puzzle games our goal is to understand how players are solving the puzzles – their cognition! This doesn’t mean we didn’t DO 1:1 think-aloud usability tests for BF3… we did! But it depended on what the goal was – getting feedback on the UX/menus/HUD etc we did 1:1 think-aloud usability and maximized efficiency – having 1 consumer participant recruited at a time, and having internal players act as “server fodder”. Also minimal time spent in-game – maximizing tie spent on the focus areas for the test.
• #43: BF3 Military FPSRealism and fast reaction times are critical – twitch gamingHighest priority gameplay mode is the online multiplayer – interruptions and distractions need to be minimized, so we do group playtest to have them be in as close to the end-user experience as possible. Collecting feedback in surveys immediately after the match – we know that there will be some data we don’t capture because it is retrospective and we are not probing real-time, but the benefit is that it’s a more ideal gameplay experience. Warp - Puzzle Platformer game – its played single player, so in some ways reaction times are not quite as urgent, the game can be more forgiving of having the player think-aloud. For puzzle games our goal is to understand how players are solving the puzzles – their cognition! This doesn’t mean we didn’t DO 1:1 think-aloud usability tests for BF3… we did! But it depended on what the goal was – getting feedback on the UX/menus/HUD etc we did 1:1 think-aloud usability and maximized efficiency – having 1 consumer participant recruited at a time, and having internal players act as “server fodder”. Also minimal time spent in-game – maximizing tie spent on the focus areas for the test.
• #44: We dedicated a lot of resources to testing Battlefield 3 – doing all these methods to get feedback on the game throughout the development lifecycle.
• #45: BF3 Military FPSRealism and fast reaction times are critical – twitch gamingHighest priority gameplay mode is the online multiplayer – interruptions and distractions need to be minimized, so we do group playtest to have them be in as close to the end-user experience as possible. Collecting feedback in surveys immediately after the match – we know that there will be some data we don’t capture because it is retrospective and we are not probing real-time, but the benefit is that it’s a more ideal gameplay experience. Warp - Puzzle Platformer game – its played single player, so in some ways reaction times are not quite as urgent, the game can be more forgiving of having the player think-aloud. For puzzle games our goal is to understand how players are solving the puzzles – their cognition! This doesn’t mean we didn’t DO 1:1 think-aloud usability tests for BF3… we did! But it depended on what the goal was – getting feedback on the UX/menus/HUD etc we did 1:1 think-aloud usability and maximized efficiency – having 1 consumer participant recruited at a time, and having internal players act as “server fodder”. Also minimal time spent in-game – maximizing tie spent on the focus areas for the test.
• #47: BF3 Military FPSRealism and fast reaction times are critical – twitch gamingHighest priority gameplay mode is the online multiplayer – interruptions and distractions need to be minimized, so we do group playtest to have them be in as close to the end-user experience as possible. Collecting feedback in surveys immediately after the match – we know that there will be some data we don’t capture because it is retrospective and we are not probing real-time, but the benefit is that it’s a more ideal gameplay experience. Warp - Puzzle Platformer game – its played single player, so in some ways reaction times are not quite as urgent, the game can be more forgiving of having the player think-aloud. For puzzle games our goal is to understand how players are solving the puzzles – their cognition! This doesn’t mean we didn’t DO 1:1 think-aloud usability tests for BF3… we did! But it depended on what the goal was – getting feedback on the UX/menus/HUD etc we did 1:1 think-aloud usability and maximized efficiency – having 1 consumer participant recruited at a time, and having internal players act as “server fodder”. Also minimal time spent in-game – maximizing tie spent on the focus areas for the test.
• #49: BF3 Military FPSRealism and fast reaction times are critical – twitch gamingHighest priority gameplay mode is the online multiplayer – interruptions and distractions need to be minimized, so we do group playtest to have them be in as close to the end-user experience as possible. Collecting feedback in surveys immediately after the match – we know that there will be some data we don’t capture because it is retrospective and we are not probing real-time, but the benefit is that it’s a more ideal gameplay experience. Warp - Puzzle Platformer game – its played single player, so in some ways reaction times are not quite as urgent, the game can be more forgiving of having the player think-aloud. For puzzle games our goal is to understand how players are solving the puzzles – their cognition! This doesn’t mean we didn’t DO 1:1 think-aloud usability tests for BF3… we did! But it depended on what the goal was – getting feedback on the UX/menus/HUD etc we did 1:1 think-aloud usability and maximized efficiency – having 1 consumer participant recruited at a time, and having internal players act as “server fodder”. Also minimal time spent in-game – maximizing tie spent on the focus areas for the test.
• #50: BF3 Military FPSRealism and fast reaction times are critical – twitch gamingHighest priority gameplay mode is the online multiplayer – interruptions and distractions need to be minimized, so we do group playtest to have them be in as close to the end-user experience as possible. Collecting feedback in surveys immediately after the match – we know that there will be some data we don’t capture because it is retrospective and we are not probing real-time, but the benefit is that it’s a more ideal gameplay experience. Warp - Puzzle Platformer game – its played single player, so in some ways reaction times are not quite as urgent, the game can be more forgiving of having the player think-aloud. For puzzle games our goal is to understand how players are solving the puzzles – their cognition! This doesn’t mean we didn’t DO 1:1 think-aloud usability tests for BF3… we did! But it depended on what the goal was – getting feedback on the UX/menus/HUD etc we did 1:1 think-aloud usability and maximized efficiency – having 1 consumer participant recruited at a time, and having internal players act as “server fodder”. Also minimal time spent in-game – maximizing tie spent on the focus areas for the test.
• #51: BF3 Military FPSRealism and fast reaction times are critical – twitch gamingHighest priority gameplay mode is the online multiplayer – interruptions and distractions need to be minimized, so we do group playtest to have them be in as close to the end-user experience as possible. Collecting feedback in surveys immediately after the match – we know that there will be some data we don’t capture because it is retrospective and we are not probing real-time, but the benefit is that it’s a more ideal gameplay experience. Warp - Puzzle Platformer game – its played single player, so in some ways reaction times are not quite as urgent, the game can be more forgiving of having the player think-aloud. For puzzle games our goal is to understand how players are solving the puzzles – their cognition! This doesn’t mean we didn’t DO 1:1 think-aloud usability tests for BF3… we did! But it depended on what the goal was – getting feedback on the UX/menus/HUD etc we did 1:1 think-aloud usability and maximized efficiency – having 1 consumer participant recruited at a time, and having internal players act as “server fodder”. Also minimal time spent in-game – maximizing tie spent on the focus areas for the test.
• #52: BF3 Military FPSRealism and fast reaction times are critical – twitch gamingHighest priority gameplay mode is the online multiplayer – interruptions and distractions need to be minimized, so we do group playtest to have them be in as close to the end-user experience as possible. Collecting feedback in surveys immediately after the match – we know that there will be some data we don’t capture because it is retrospective and we are not probing real-time, but the benefit is that it’s a more ideal gameplay experience. Warp - Puzzle Platformer game – its played single player, so in some ways reaction times are not quite as urgent, the game can be more forgiving of having the player think-aloud. For puzzle games our goal is to understand how players are solving the puzzles – their cognition! This doesn’t mean we didn’t DO 1:1 think-aloud usability tests for BF3… we did! But it depended on what the goal was – getting feedback on the UX/menus/HUD etc we did 1:1 think-aloud usability and maximized efficiency – having 1 consumer participant recruited at a time, and having internal players act as “server fodder”. Also minimal time spent in-game – maximizing tie spent on the focus areas for the test.
• #53: BF3 Military FPSRealism and fast reaction times are critical – twitch gamingHighest priority gameplay mode is the online multiplayer – interruptions and distractions need to be minimized, so we do group playtest to have them be in as close to the end-user experience as possible. Collecting feedback in surveys immediately after the match – we know that there will be some data we don’t capture because it is retrospective and we are not probing real-time, but the benefit is that it’s a more ideal gameplay experience. Warp - Puzzle Platformer game – its played single player, so in some ways reaction times are not quite as urgent, the game can be more forgiving of having the player think-aloud. For puzzle games our goal is to understand how players are solving the puzzles – their cognition! This doesn’t mean we didn’t DO 1:1 think-aloud usability tests for BF3… we did! But it depended on what the goal was – getting feedback on the UX/menus/HUD etc we did 1:1 think-aloud usability and maximized efficiency – having 1 consumer participant recruited at a time, and having internal players act as “server fodder”. Also minimal time spent in-game – maximizing tie spent on the focus areas for the test.
• #54: BF3 Military FPSRealism and fast reaction times are critical – twitch gamingHighest priority gameplay mode is the online multiplayer – interruptions and distractions need to be minimized, so we do group playtest to have them be in as close to the end-user experience as possible. Collecting feedback in surveys immediately after the match – we know that there will be some data we don’t capture because it is retrospective and we are not probing real-time, but the benefit is that it’s a more ideal gameplay experience. Warp - Puzzle Platformer game – its played single player, so in some ways reaction times are not quite as urgent, the game can be more forgiving of having the player think-aloud. For puzzle games our goal is to understand how players are solving the puzzles – their cognition! This doesn’t mean we didn’t DO 1:1 think-aloud usability tests for BF3… we did! But it depended on what the goal was – getting feedback on the UX/menus/HUD etc we did 1:1 think-aloud usability and maximized efficiency – having 1 consumer participant recruited at a time, and having internal players act as “server fodder”. Also minimal time spent in-game – maximizing tie spent on the focus areas for the test.
• #55: BF3 Military FPSRealism and fast reaction times are critical – twitch gamingHighest priority gameplay mode is the online multiplayer – interruptions and distractions need to be minimized, so we do group playtest to have them be in as close to the end-user experience as possible. Collecting feedback in surveys immediately after the match – we know that there will be some data we don’t capture because it is retrospective and we are not probing real-time, but the benefit is that it’s a more ideal gameplay experience. Warp - Puzzle Platformer game – its played single player, so in some ways reaction times are not quite as urgent, the game can be more forgiving of having the player think-aloud. For puzzle games our goal is to understand how players are solving the puzzles – their cognition! This doesn’t mean we didn’t DO 1:1 think-aloud usability tests for BF3… we did! But it depended on what the goal was – getting feedback on the UX/menus/HUD etc we did 1:1 think-aloud usability and maximized efficiency – having 1 consumer participant recruited at a time, and having internal players act as “server fodder”. Also minimal time spent in-game – maximizing tie spent on the focus areas for the test.
• #57: What other methods are there?We’re looking at
• #58: What other methods are there?Diary study
• #59: What other methods are there?Diary study
• #60: What other methods are there?Diary study
• #62: What other methods are there?Diary study
• #63: What other methods are there?Diary study
• #64: What other methods are there?Diary study
• #65: What other methods are there?Diary study
• #68: Gamers lie to get into studies and get the incentivesCareful screening is important for making sure gamers have the experience we are looking for
• #69: Gamers lie to get into studies and get the incentivesCareful screening is important for making sure gamers have the experience we are looking for
• #70: Gamers lie to get into studies and get the incentivesCareful screening is important for making sure gamers have the experience we are looking for
• #71: Gamers lie to get into studies and get the incentivesCareful screening is important for making sure gamers have the experience we are looking for
• #72: Gamers lie to get into studies and get the incentivesCareful screening is important for making sure gamers have the experience we are looking for
• #73: Tell me why you like Skyrim – well I like the characters and the graphics. Oh which characters did you like? Ummm the king? I can’t remember his name right now, but I thought he was cool. Ok, how about the graphics? What did you like about them?They were cool – they looked good. I liked the style.How about the gameplay? Yeah it was cool. I liked it.
• #76: So why does getting the right players matter?When we’re looking for players to test a co-op campaign
• #78: So why does getting the right players matter?When we’re looking for players to test a co-op campaign
• #79: Because we’re looking for qualified responses.
• #81: Targeted to ages 7-12, the sifteo cube gaming platform is hands on. Games are played by manipulating a set of cubes, putting them together (we call that neighboring), shaking, tilting, flipping – Since the cubes are a spatial and tangible interface, the platform really lends itself to puzzle games including word games, logic games, and maze adventure games. The system has audio as well that comes out of what we call the master cube. So now that you have a better understanding of the platform, we can dig a little deeper into what playtesting this platform is like.
• #85: Finally, the platform AND the games were being built at the same time. As our LEAD QA put it, it is like buiding an overpass and driving on it at the same time.
• #87: So what do we do at the PlayLab?
• #91: Cube buddies is a puzzle game. There are characters in the game and you engage with their faces by putting them back together after they have been scrambled. Faces are put back together by neighboring cubes and thus allowing the game to swap parts. So basically, if the blue/green guy and the orange guy were just neighbored, there eyes would have just swapped.
• #92: Learned through playtesting
• #93: Lots of things are wrong here, especially the eyes. Zorg’s eyes are on Suli’s head and Suli’s eyes are on Zorg’s head.
• #94: The player would start by putting the cubes together to swap the eyes.
• #96: Kids understood that the eyes were upside down, but making the leap to swap the same type of part was HUGE for kids. Why weren’t we teaching this? Kids could obviously tell that the eyes were wrong, they just had no idea what to do with it.
• #98: We had to single out this puzzle solving skill. This was tough to pinpoint just in level design but easy to pinpoint while playtesting. And playtesting resulted in an entire puzzle level around the core skill – swapping eyes.
• #101: Tilt the cubes to rearrange letters only on the cube, move the cubes around and neighbor to spell a target word
• #102: Why present word caravan in the context of platform?
• #106: Through playtesting we realized that word games are extra tricky. Players came to the game with models of other word games in their mind. They wanted to swipe the cubes like they were an iPad and they wanted to be able to swap out the letters, which they couldn’t. We had to find a way to teach OUR model of a word game.
• #107: SO how do we teach them our model and our rules, when they have something engrained in their mind of how word games work, AND how do we do that in a way that made the game too easy for older kids.
• #109: It helps a little, but if you are not an avid reader, you wont even know what patterns to look for, 7-9year olds may only just be learning compound words, phonemic patterns, prefixes and suffixes
• #110: This certainly helped in early levels of the game, but we were still finding that kids got stuck in later levels
• #114: We worked with our designers, creating that feedback loop – to carefully to make hypothesis on hinting, and initially, we still weren’t getting results
• #116: Developer watched each iteration Turn around within a day Sometimes reprogrammed hint mechanism on the fly after watching kids play, seeing first hand that the kids did not notice the icons encouraged the developer to take action on the critical items
• #118: Sandwich Kingdom is a
• #119: For this game, it was really important to work with the game designer to come up with a script of specific tasks we wanted to the player to do.
• #120: Are the users having fun playing as a girl character? Do they feel emersed in the world? What causes them not to be? Will the platform be a distraction?
• #121: But you have to start somewhere, the best way to start writing a script for your playtest is to play that version of the build
• #122: And figure out where you yourself are getting confused… this works great in early stages of the game, later one, you might know too much, so put pressure on the developer to discuss what the critical paths are to ensure the player makes it through their quest
• #123: One feature that is a great example are the dangerous pits. In the final implementation of the game, which you see here, we have a lot of modalities teaching us about the pits, we have an NPC telling us, and art that we will stumble upon if we have pearl walk through the temple a little further
• #124: So the purpose of the pit is that if you walk over it, it sends you back to the beginning, but early in the game…
• #125: We didn’t have NPCs to tell the player to be aware and the art for the pit was just some hash marks. Testing something like this early, even before the feature was done, allowed us to understand if the purpose feature was clear to the player or if it was just annoying. By testing this we learned that the player enjoyed avoiding the pits and saw it as a challenging mechanic rather than just a frustrating obstacle, so we continued with the feature and built it out.
• #127: And depending on your genre, you will need to ask different questions.
• #129: First off, the boring stuff – forms forms forms…
• #130: Make sure kids know you aren’t testing them, you are testing the game… you are only there to observe. Let them know that
• #132: Having siblings play together naturally is a great way to learn if your game is usable or not. It is especially helpful if you can see one child teach the other child how to play. If they are explaining a feature that doesn’t even exist or if they have trouble explaining a feature, then you know you may not be teaching core skills of the game properly
• #134: Its tough to ask kids “was this hard”
• #135: But how do we know how the entire game works???
• #136: We were developing the hw and the sw at the same time. Saving game metrics wasn’t a sensible priority for the team so we had to find another way to get quantitative data
• #147: Adults can be easier to come by, but you have to keep in mind that adults play differently than kids, so use your target audience as much as possible but recognize what value you can get from having a different set of eyes
• #148: Longitudinal testing allows you to see major issues, but sometimes it just isn’t possible to fix them
• #149: - Doing Games User Research is better than nothing! Get your feedback loop on!Think about what your design is about – what do you want the player to experience?Work closely with the Games User Researcher or persons working on the test to make sure they understand what feedback you need – what your priorities are? What information are you lacking?Games user research can help you improve the Game Designthe platform experience, Experience when used by your target audience, and collecting the most actionable feedback – regardless of development phase, gameplay, genre…
• #150: - Doing Games User Research is better than nothing! Get your feedback loop on!Think about what your design is about – what do you want the player to experience?Work closely with the Games User Researcher or persons working on the test to make sure they understand what feedback you need – what your priorities are? What information are you lacking?Games user research can help you improve the Game Designthe platform experience, Experience when used by your target audience, and collecting the most actionable feedback – regardless of development phase, gameplay, genre…
• #151: - Doing Games User Research is better than nothing! Get your feedback loop on!Think about what your design is about – what do you want the player to experience?Work closely with the Games User Researcher or persons working on the test to make sure they understand what feedback you need – what your priorities are? What information are you lacking?Games user research can help you improve the Game Designthe platform experience, Experience when used by your target audience, and collecting the most actionable feedback – regardless of development phase, gameplay, genre…