Rita Bush - How to Use Games to Mitigate Cognitive Bias in Analysis

INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)
How to Use Games
to Mitigate Cognitive Bias in Analysis
Dr. Rita M. Bush

Coast Guard
Central Intelligence Agency
Army
Navy
Air Force
National Reconnaissance Office
National Geospatial-Intelligence
Agency
National Security Agency
Defense Intelligence Agency
Department of State
Department of Energy
Department of the Treasury
Department of Homeland Security
Federal Bureau of Investigation
Drug Enforcement Administration
Marine Corps
Office of the Director of National Intelligence
3

IARPA Mission and Method
• Bring the best minds to bear on our problems
– Full and open competition to the greatest possible extent
– World-class, rotational, Program Managers
• Define and execute research programs that:
– Have goals that are clear, measureable, ambitious and credible
– Employ independent and rigorous Test & Evaluation
– Involve IC partners from inception to finish
– Run from three to five years
4
To envision and lead high risk/high payoff research
that delivers innovative technology
for future overwhelming intelligence advantage.

Sirius Goal
Use serious games to train analysts to
recognize and mitigate their cognitive
biases
5

Why Games?
• Serious Games provide
– Experiential learning
– Freedom to fail
– Repetition, repetition, repetition
• Game/simulation based training is
common in military, business,
medicine, etc.
• New analysts have likely grown up
playing videogames
6

Cognitive Bias
• Decision making relies on simple, fast, heuristics
– “rules of thumb”
• Heuristics gone astray = Cognitive Bias
• Affects all decision-makers: medical
professionals, law enforcement officers,
intelligence analysts… you, me…
• Examples in the IC
– The Commission on the Intelligence Capabilities of the United
States Regarding Weapons of Mass Destruction (2005)
– Lessons Not Learned: Postmortems on Intelligence Failures,
1950-2004, Kent Center for Analytic Tradecraft (2005)
7

Biases Examined by Sirius Program
8

Why Conduct this Research?
• Cognitive biases are very hard to
overcome
• Studies of game-based learning treat the
game as a “black box”
• Limited research on game-based
learning effects over time
• No other research using games to
mitigate cognitive biases
• Very few successful examples of
developing valid tests of cognitive bias
9

INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA) 10
SUNY Albany “CYCLES”
Leidos “Missing”
Example Screen Shots

Assessment of Biases in Cognition
(ABC) Instrument
• Developed by ETS, an industry leader in
assessment development and validation
• Web-based assessment measuring susceptibility
to cognitive biases
• Extensive literature review available
11

Independent Validation & Verification (IV&V)
Research Design (Johns Hopkins APL)
12
IARPA Supplied
Training Video
IARPA Supplied
Training Video
Best & Final
Versions of
Performer Games
ETS/MITRE
Pretest
Participants
ETS/MITRE
Immediate
Posttest
ETS/MITRE Delayed
Online Posttest
Phase 1: 8 weeks
Phase 2: 12 weeks
Demographics
ETS/MITRE
Standardized
Measures of
Cognitive Bias
and Bias
Knowledge
RandomAssignment
ETS/MITRE
Standardized
Measures of
Cognitive Bias
and Bias
Knowledge
ETS/MITRE
Standardized
Measures of
Cognitive Bias
and Bias
Knowledge

If yes, how much will the training effect
decay,
8-12 weeks after final game-play?
Do games work as training tools?
Sirius Answers 3 Questions
Do games work as well as a video?

Sirius Phase 1 game 8-week posttest scores
stayed the same or improved!
The video shows the expected
decay after 8 weeks
All games showed
significant improvement in
bias mitigation scores

Phase 2 Student Study Results
15
All games & video improved bias mitigation from pre-test to post-test across biases – larger improvements
for Missing & Cycles. Game bias mitigation dropped at 12 weeks but still improved over pre-test

Active vs Passive Learning
(N = 305)
Playing the game was better than watching the game, which was better than watching the video
16

Myth-Busting
• Myth: Games only work well for gamers, or
younger analysts, or males
– “Non-gamers” learned just as much
– No age or gender differences
• Myth: High fidelity (close match to physical or
organizational features of the job) is
important
– Simple graphics and game style worked just as
well (or better)
– “Cognitive fidelity” did matter
17

Point of Contact
• Sirius Program Manager
– Dr. Rita Bush
– rita.bush@iarpa.gov
• For more information, see:
– www.iarpa.gov
18

Thank You!
19

Rita Bush - How to Use Games to Mitigate Cognitive Bias in Analysis

More Related Content

What's hot (8)

Similar to Rita Bush - How to Use Games to Mitigate Cognitive Bias in Analysis (20)

More from SeriousGamesAssoc (20)

Recently uploaded (20)

Rita Bush - How to Use Games to Mitigate Cognitive Bias in Analysis

Editor's Notes