D. Mayo: Replication Research Under an Error Statistical Philosophy

SPP
D.
Mayo
1

Replication Research Under an Error Statistical Philosophy
Deborah Mayo
Around a year ago on my blog:
“There are some ironic twists in the way psychology is
dealing with its replication crisis that may well threaten even
the most sincere efforts to put the field on firmer scientific
footing”
Philosopher’s talk: I see a rich source of problems that cry out
for ministrations of philosophers of science and of statistics

SPP
D.
Mayo
2

Three main philosophical tasks:
#1 Clarify concepts and presuppositions
#2 Reveal inconsistencies, puzzles, tensions (“ironies”)
#3 Solve problems, improve on methodology
• Philosophers usually stop with the first two, but I think
going on to solve problems is important.
This presentation is ‘programmatic’- what might replication
research under an error statistical philosophy be?
My interest grew thanks to Caitlin Parker whose MA thesis was
on the topic

SPP
D.
Mayo
3

Example of a conceptual clarification (#1)
Editors of a journal, Basic and Applied Social Psychology,
announced they are banning statistical hypothesis testing
because it is “invalid”
It’s invalid because it does not supply “the probability of the
null hypothesis, given the finding” (the posterior probability of
H0) (2015 Trafimow and Marks)
• Since the methodology of testing explicitly rejects the mode
of inference they don’t supply, it would be incorrect to claim
the methods were invalid.
• Simple conceptual job that philosophers are good at

SPP
D.
Mayo
4

Example of revealing inconsistencies and tensions (#2)
Critic: It’s too easy to satisfy standard significance thresholds
You: Why do replicationists find it so hard to achieve
significance thresholds?
Critic: Obviously the initial studies were guilty of p-hacking,
cherry-picking, significance seeking, QRPs
You: So, the replication researchers want methods that pick up
on and block these biasing selection effects.
Critic: Actually the “reforms” recommend methods where
selection effects and data dredging make no difference

SPP
D.
Mayo
5

Whether this can be resolved or not is separate.
• We are constantly hearing of how the “reward structure”
leads to taking advantage of researcher flexibility
• As philosophers, we can at least show how to hold their
feet to the fire, and warn of the perils of accounts that bury
the finagling
The philosopher is the curmudgeon (takes chutzpah!)
I’ll give examples of
#1 clarifying terms
#2 inconsistencies
#3 proposed solutions (though I won’t always number them)
.

SPP
D.
Mayo
6

Demarcation: Bad Methodology/Bad Statistics
• A lot of the recent attention grew out of the case of Diederik
Stapel, the social psychologist who fabricated his data.
• Kahneman
in
2012
“I
see
a
train-‐wreck
looming,”
setting

up
a
“daisy
chain”
of
replication.

• The Stapel investigators: 2012 Tilberg Report, “Flawed
Science” do a good job of characterizing pseudoscience.
• Philosophers tend to have cold feet when it comes to saying
anything general about science versus pseudoscience.

SPP
D.
Mayo
7

Items in their list of “dirty laundry” include:
“An experiment fails to yield the expected statistically
significant results. The experimenters try and try again
until they find something (multiple testing, multiple
modeling, post-data search of endpoint or subgroups),
and the only experiment subsequently reported is the
one that did yield the expected results.”
… continuing an experiment until it works as desired, or
excluding unwelcome experimental subjects or results,
inevitably tends to confirm the researcher’s research
hypotheses, and essentially render the hypotheses
immune to the facts”. (Report, 48)
--they walked into a “culture of verification bias”

SPP
D.
Mayo
8

Bad Statistics
Severity Requirement: If data x0 agree with a hypothesis
H, but the test procedure had little or no capability, i.e., little
or no probability of finding flaws with H (even if H is
incorrect), then x0 provide poor evidence for H.
Such a test we would say fails a minimal requirement for a
stringent or severe test.
• This seems utterly uncontroversial.

SPP
D.
Mayo
9

• Methods that scrutinize a test’s capabilities, according to
their severity, I call error statistical.
• Existing error probabilities (confidence levels, significance
levels) may but need not provide severity assessments.
• New name: frequentist, sampling theory, Fisherian,
Neyman-Pearsonian—are too associated with hard line
views and personality conflicts (“It’s the methods, stupid”)
(example of new solutions #3)

SPP
D.
Mayo
10

Are philosophies about science relevant?
One of the final recommendations in the Report is this:
In the training program for PhD students, the relevant
basic principles of philosophy of science, methodology,
ethics and statistics that enable the responsible practice
of science must be covered. (p. 57)

SPP
D.
Mayo
11

A critic might protest:
“There’s nothing philosophical about my criticism of
significance tests: a small p-value is invariably, and
erroneously, interpreted as giving a small probability to the null
hypothesis that the observed difference is mere chance.”
Really? P-values are not intended to be used this way;
presupposing they should be stems from a conception of the role
of probability in statistical inference—this conception is
philosophical.
(of course criticizing them because they might be misinterpreted
is just silly)

SPP
D.
Mayo
12

Two
main
views
of
the
role
of
probability
in
inference

Probabilism.
To
provide
a
post-‐data
assignment
of
degree

of
probability,
confirmation,
support
or
belief
in
a

hypothesis,
absolute
or
comparative,
given
data
x0.

Performance.
To
ensure
long-‐run
reliability
of
methods,

coverage
probabilities,
control
the relative frequency of
erroneous inferences in a long-run series of trials.

What happened to the goal of scrutinizing bad science by the
severity criterion?

SPP
D.
Mayo
13

• Neither “probabilism” nor “performance” directly captures
it.
• Good long-run performance is a necessary not a sufficient
condition for avoiding insevere tests.

• The problems with selective reporting, multiple testing,
stopping when the data look good are not problems about
long-runs—
• It’s that we cannot say about the case at hand that it has
done a good job of avoiding the sources of
misinterpretation.

SPP
D.
Mayo
14

• Probabilism
says
H
is
not
justified
unless
it’s
true
or

probable
(made
firmer).

• Error
statistics
(probativism)
says
H
is
not
justified

unless
something
(a
good
job)
has
been
done
to
probe

ways
we
can
be
wrong
about
H.

• If
it’s
assumed
probabilism
is
required
for
inference,

error
probabilities
could
be
relevant
only
by

misinterpretation.
False!

• Error
probabilities
have
a
crucial
role
in
appraising
well-‐
testedness
(new
philosophy
for
probability
#3)

• Both
H
and
not-‐H
be
can
be
poorly
tested,
so
a
severe
testing

assessment
violates
probability

SPP
D.
Mayo
15

Understanding
the
Replication
Crisis
Requires

Understanding
How
it
Intermingles
with
PhilStat

Controversies

• It’s not that I’m keen to defend many common uses of
significance tests
• It’s just that the criticisms (in psychology and elsewhere)
are based on serious misunderstandings of the nature and
role of these methods; consequently so are many “reforms”
• How can you be clear the reforms are better if you might be
mistaken about existing methods?

SPP
D.
Mayo
16

Criticisms
concern
a
kind
of
Fisherian
Significance
Test
(i) Sample
space:
Let
the
sample
be
X
=
(X1,
…,Xn),
be
n
iid

(independent
and
identically
distributed)
outcomes
from
a

Normal
distribution
with
standard
deviation

σ

(ii)
A
null
hypothesis
H0:
µ
=

0

(Δ: µΤ − µC = 0)

(iii)
Test
statistic:
A
function
of
the
sample,
d(X)
reflecting
the

difference
between
the
data
x0
=
(x1,
…,xn),
and
H0:

The
larger
d(x0)
the
further
the
outcome
from
what’s

expected
under
H0,
with
respect
to
the
particular
question.

(iv)
Sampling
distribution
of
test
statistic:
d(X)

SPP
D.
Mayo
17

The
p-‐value
is
the
probability
of
a
difference
larger
than
d(x0),

under
the
assumption
that
H0
is
true:

p(x0)=Pr(d(X)
>
d(x0);
H0).

If p(x0)
is
sufficiently
small,
there’s
an
indication
of

discrepancy
from
the
null.

(Even
Fisher
had
implicit
alternatives,
by
the
way)

SPP
D.
Mayo
18

P-‐value
reasoning:
from
high
capacity
to
curb

enthusiasm

If
the
hypothesis
H0
is
correct
then,
with
high
probability,
1-‐p,

the
data
would
not
be
statistically
significant
at
level
p.

x0
is
statistically
significant
at
level
p.

____________________________

Thus,
x0
indicates
a
discrepancy
from
H0.

That
merely
indicates
some
discrepancy!

SPP
D.
Mayo
19

A genuine experimental effect is needed
“[W]e need, not an isolated record, but a reliable method of
procedure. In relation to the test of significance, we may say
that a phenomenon is experimentally demonstrable when we
know how to conduct an experiment which will rarely fail to
give us a statistically significant result.” (Fisher 1935, 14)
(low P-value ≠> H: statistical effect)
“[A]ccording
to
Fisher,
rejecting
the
null
hypothesis
is
not

equivalent
to
accepting
the
efficacy
of
the
cause
in

question.
The
latter...requires
obtaining
more
significant

results
when
the
experiment,
or
an
improvement
of
it,
is

repeated
at
other
laboratories
or
under
other

conditions.”
(Gigerentzer
1989,
95-‐6)
(H ≠> H*)

SPP
D.
Mayo
20

Still,
simple
Fisherian
Tests
have
Important
Uses

• Testing
assumptions

• Fraudbusting
and
forensics:
Finding
Data
too
good
to
be

true
(Simonsohn)

• Finding
if
data
are
consistent
with
a
model

Gelman and Shalizi (meeting of minds between a Bayesian and
an error statistician)
“What we are advocating, then, is what Cox and Hinkley (1974)
call ‘pure significance testing’, in which certain of the model’s
implications are compared directly to the data, rather than
entering into a contest with some alternative model.” (p.20)

SPP
D.
Mayo
21

Fallacy
of
Rejection:
H
–
>
H*
:
Erroneously
take
statistical

significance
as
evidence
of
research
hypothesis
H*

The
fallacy
is
explicated
by
severity:
flaws
in
alternative
H*
have

not
been
probed
by
the
test,
the
inference
from
a
statistically

significant
result
to
H*
fails
to
pass
with
severity

Merely refuting the null hypothesis is too weak to
corroborate substantive H*, “we have to have ‘Popperian
risk’, ‘severe test’ [as in Mayo], or what philosopher Wesley
Salmon called ‘a highly improbable coincidence.’” (Meehl
and Waller 2002, 184)

(Meehl
was
wrong
to
blame
Fisher)

SPP
D.
Mayo
22

NHST
are
pseudostatistical:

Why
do
psychologists
speak
of
NHSTs
–tests
that
supposedly

allow
moving
from
statistical
to
substantive?

So
defined,
they
exist
only
as
abuses
of
tests:
they
exist as
something you’re never supposed to do

Psychologists
tend
to
ignore
Neyman-‐Pearson
(N-‐P)
tests:
N-‐P

supplemented
Fisher’s
tests
with
explicit
alternatives

SPP
D.
Mayo
23

Neyman-‐Pearson
(N-‐P)
Tests:
A
null
and
alternative

hypotheses
H0,
H1
that
exhaust
the
parameter
space

So
the
fallacy
of
rejection
H
–
>
H*
is
impossible

(rejecting
the
null
only
indicates
statistical
alternatives)

Scotches
criticisms
that
P-‐values
are
only
under
the
null

Example:
Test
T+:

sampling
distribution
of
d(x)
under
null

and
alternatives.
H0:
µ
≤
µ0

vs.

H1:
µ
>
µ0

if
d(x0)
>

cα,
"reject"
H0,

if
d(x0)
<

cα,
"do
not
reject”
or
“accept"
H0,

e.g.
cα=1.96
for
α=.025

SPP
D.
Mayo
24

The
sampling
distribution
yields
Error
Probabilities

Probability
of
a
Type
I
error
=
P(d(X)
>

cα;
H0)
≤

α.

Probability
of
a
Type
II
error:
=
P(d(X)
<
cα;
H0)
=
ß(µ1),
for

any
µ1
>
µ0.

The
complement
of
the
Type
II
error
probability=
power

against
(µ1)

POW(µ1)=
P(d(X)
>
cα;
µ1)

Even
without
“best”
tests,
there
are
“good”
tests

SPP
D.
Mayo
25

N-‐P
test
in
terms
of
the
P-‐value:
reject
H0
iff
P-‐value
<
.025

• Even
N-‐P
report
the
attained
significance
level
or
P-‐value

(Lehmann)

• “reject/do
not
reject”
uninterpreted
parts
of
the

mathematical
apparatus

Reject
could
be:
“Declare
statistically
significant
at
the
p-‐level”

• “The
tests…
must
be
used
with
discretion
and

understanding”
(N-‐P,
1928,
p.
58)

(“it’s
the
methods,
stupid”)

SPP
D.
Mayo
26

Why
Inductive
behavior?

N-‐P
justify
tests
(and
confidence
intervals)
by
performance,

control
of
long-‐run
error
coverage
probabilities

They
called
this
inductive
behavior,
why?

• They
were
reaching
conclusions
beyond
the
data

(inductive)

• If
inductive
inference
is
probabilist,
then
they
needed
a

new
term.

In
Popperian
spirit,
they
(mostly
Neyman)
called
it

inductive
behavior-‐-‐
adjust
how
we’d
act
rather
than
beliefs

(I’m
not
knocking
performance,
but
error
probabilities
also

serve
for
particular
inferences—evidential)

SPP
D.
Mayo
27

N-‐P
tests
can
still
commit
a
type
of
fallacy
of
rejection:

Infer
a
discrepancy
beyond
what’s
warranted:

––especially
with n sufficiently large:
large
n
problem.

• Severity
tells
us:
an
α-‐significant
difference
is
indicative
of
less

of
a
discrepancy
from
the
null
if
it
results
from
larger
(n1)

rather
than
a
smaller
(n2)
sample
size
(n1
>
n2
)
What’s
more
indicative
of
a
large
effect
(fire),
a
fire
alarm
that

goes
off
with
burnt
toast
or
one
so
insensitive
that
it
doesn’t

go
off
unless
the
house
is
fully
ablaze?
[The
larger
sample
size

is
like
the
one
that
goes
off
with
burnt
toast.)

SPP
D.
Mayo
28

Fallacy
of
Non-‐Significant
results:
Insensitive
tests

• Negative
results
may
not
warrant
0
discrepancy

from
the
null,
but
we
can
use
severity
to
rule
out

discrepancies
that,
with
high
probability,
would
have

resulted
in
a
larger
difference
than
observed

Similar
to
Cohen’s
power
analysis
but
sensitive
to
the

outcome—P-‐value
distribution
(#3)

• I
hear
some
replicationists
say
negative
results
are

uninformative:
not
so
(#2
ironies)

No
point
in
running
replication
research
if
your

account
views
negative
results
as
uninformative

SPP
D.
Mayo
29

Error
statistics
gives
evidential
interpretation
to
tests

(#3)

Use
results
to
infer
discrepancies
from
a
null
that
are
well
ruled-‐
out,
and
those
which
are
not

I’d
never
just
report
a
P-‐value

Mayo
(1996);

Mayo
and
Cox
(2010):
Frequentist

Principle
of

Evidence:
FEV

Mayo
and
Spanos
(2006):
SEV

SPP
D.
Mayo
30

One-‐sided
Test
T+:

H0:
µ
<
µ0

vs.

H1:
µ
>
µ0

d(x)
is
statistically
significant
(set
lower
bounds)

(i)
If
the
test
had
high
capacity
to
warn
us
(by

producing
a
less
significant
result)
if
µ
≤
µ0
+
γ.
then

d(x)
is
a
good
indication
of
µ
>
µ0
+
γ.

(ii)
If
the
test
had
little
(or
even
moderate)
capacity

(e.g.
<
.5)
to
produce
a
less
significant
result
even
if
µ
≤

µ0
+
γ,
then
d(x)
is
a
poor
indication
of
µ
>
µ0
+
γ

(If
an
even
more
impressive
result
is
probable,
due
to

guppies,
it’s
not
a
good
indication
of
a
great
whale)

SPP
D.
Mayo
31

d(x)
is
not
statistically
significant
(set
upper
bounds)

(i)If
the
test
had
a
high
probability
of
producing
a

more
statistically
significant
difference
if
µ
>
µ0
+
γ,

then
d(x)
is
a
good
indication
that
µ
≤
µ0
+
γ.

(ii)
If
the
test
had
a
low
probability
of
a
more

statistically
significant
difference
if
µ
>
µ0
+
γ,
then
d(x)

is
poor
indication
that
µ
≤
µ0
+
γ.
(too
insensitive
to

rule
out
discrepancy
γ)

If
you
set
an
overly
stringent
significance
level
in
order
to

block
rejecting
a
null,
we
can
determine
the

discrepancies
you
can’t
detect
(e.g.,
risks
of
concern)

SPP
D.
Mayo
32

Confidence
Intervals
also
require
supplementing

Duality
between
tests
and
intervals:
values
within
the
(1
-‐
α)

CI
are
non-‐rejectable
at
the
α
level

• Still
too
dichotomous:
in
/out,
plausible/not
plausible

(Permit
fallacies
of
rejection/non-‐rejection).

• Justified
in
terms
of
long-‐run
coverage
(performance).

• All
members
of
the
CI
treated
on
par.

• Fixed
confidence
level
(SEV
needs
several
benchmarks).

• Estimation
is
important
but
we
need
tests
for

distinguishing
real
and
spurious
effects,
and
checking

assumptions
of
statistical
models.

SPP
D.
Mayo
33

The
evidential
interpretation
is
crucial
but
error

probabilities
can
be
violated
by
selection
effects
(also

violated
model
assumptions)

One
function
of
severity
is
to
identify
which
selection
effects

are
problematic
(not
all
are)
(#3).

Biasing
selection
effects:
when
data
or
hypotheses
are

selected
or
generated
(or
a
test
criterion
is
specified),
in

such
a
way
that
the
minimal
severity
requirement
is

violated,
seriously
altered
or
incapable
of
being
assessed.

SPP
D.
Mayo
34

Nominal vs actual significance levels
Suppose
that
twenty
sets
of
differences
have
been

examined,
that
one
difference
seems
large
enough
to
test

and
that
this
difference
turns
out
to
be
‘significant
at
the
5

percent
level.’
….The
actual
level
of
significance
is
not
5

percent,
but
64
percent!
(Selvin,
1970,
p.
104)

• They
were
clear
on
the
fallacy:
blurring
the
“computed”

or
“nominal”
significance
level,
and
the
“actual”
level

• There
are
many
more
ways
you
can
be
wrong
with

hunting
(different
sample
space)

SPP
D.
Mayo
35

This is a genuine example of an invalid or unsound method

You report: Such
results
would
be
difficult
to
achieve
under

the
assumption
of
H0
When
in
fact
such
results
are
common
under
the

assumption
of
H0
(formally): You say Pr(P-value < Pobs; H0) ~ α (small)

but in fact Pr(P-value < Pobs; H0) = high, if not guaranteed
• Nowadays,
we’re
likely
to
see
the
tests
blamed
for

permitting
such
misuses
(instead
of
the
testers).

• Worse
are
those
accounts
where
the
abuse
vanishes!

SPP
D.
Mayo
36

What
defies
scientific
sense?

On
some
views,
biasing
selection
effects
are
irrelevant….

Stephen
Goodman
(epidemiologist):

Two
problems
that
plague
frequentist
inference:
multiple

comparisons
and
multiple
looks,
or,
as
they
are
more

commonly
called,
data
dredging
and
peeking
at
the
data.

The
frequentist
solution
to
both
problems
involves

adjusting
the
P-‐value…But
adjusting
the
measure
of

evidence
because
of
considerations
that
have
nothing
to
do

with
the
data
defies
scientific
sense,
belies
the
claim
of

‘objectivity’
that
is
often
made
for
the
P-‐value.”
(1999,
p.

1010).

SPP
D.
Mayo
37

Likelihood
Principle
(LP)

The
vanishing
act
takes
us
to
the
pivot
point
around
which

much
debate
in
philosophy
of
statistics
revolves:

In probabilisms, the import of the data is via the ratios of
likelihoods of hypotheses:
P(x0;H1)/P(x0;H0)

Different
forms:
posterior
probabilities,
Bayes
factor

(inference
is
comparative,
data
favors
this
over
that–is
that

even
inference?)

SPP
D.
Mayo
38

All
error
probabilities
violate
the
LP
(even
without

selection
effects):

“Sampling
distributions,
significance
levels,
power,
all
depend

on
something
more
[than
the
likelihood
function]–something

that
is
irrelevant
in
Bayesian
inference–namely
the
sample

space”.
(Lindley
1971,
p.
436)

The
information
is
just
a
matter
of
our
“intentions”

“The
LP
implies…the
irrelevance
of
predesignation,
of

whether
a
hypothesis
was
thought
of
before
hand
or
was

introduced
to
explain
known
effects
(Rosenkrantz,
1977,

122)

SPP
D.
Mayo
39

Many current Reforms are Probabilist
Probabilist reforms to replace tests (and CIs) with likelihood
ratios, Bayes factors, HPD intervals, or just lower the P-value
(so that the maximal likely alternative gets .95 posterior)
while ignoring biasing selection effects, will fail.

The same p-hacked hypothesis can occur in Bayes factors;
optional stopping can exclude true nulls from HPD intervals.
With one big difference: Your direct basis for criticism and
possible adjustments has just vanished.
(lots of #2 inconsistencies)

SPP
D.
Mayo
40

How
might
probabilists
block
intuitively
unwarranted

inferences?
(Consider
first
subjective)
When we hear there’s statistical evidence of some unbelievable
claim (distinguishing shades of grey and being politically
moderate, ovulation and voting preferences), some probabilists
claim—you see, if our beliefs were mixed into the interpretation
of the evidence, we wouldn’t be fooled
We know these things are unbelievable, a subjective Bayesian
might say
That could work in some cases (though it still wouldn’t show
what researchers had done wrong)—battle of beliefs.

SPP
D.
Mayo
41

It wouldn’t help with our most important problem:
• How to distinguish the warrant for a single hypothesis H
with different methods (e.g., one has biasing selection
effects, another, registered results and precautions)?
So now you’ve got two sources of flexibility, priors and biasing
selection effects (which can no longer be criticized).
Besides, researchers really do believe their hypotheses.

SPP
D.
Mayo
42

Diederik Stapel says he always read the research literature
extensively to generate his hypotheses.
“So that it was believable and could be argued that this
was the only logical thing you would find.” (E.g., eating
meat causes aggression.)
(In “The Mind of a Con Man,” NY Times, April 26,
2013[4])

SPP
D.
Mayo
43

Conventional
Bayesians

The most popular probabilisms these days are “non-subjective”
(reference, default) or conventional designed
to
prevent
prior

beliefs
from
influencing
the
posteriors:

“The
priors
are
not
to
be
considered
expressions
of

uncertainty,
ignorance,
or
degree
of
belief.
Conventional

priors
may
not
even
be
probabilities…
.”
(Cox
and
Mayo

2010,
p.
299)

How
might
they
avoid
too-‐easy
rejections
of
a
null?

SPP
D.
Mayo
44

Cult
of
the
Holy
Spike

Give
a
spike
prior
of
.5
to
H0
the
remaining
.5
probability
being

spread
out
over
the
alternative
parameter
space,
Jeffreys.

This
“spiked
concentration
of
belief
in
the
null”
is
at
odds
with

the
prevailing
view
“we
know
all
nulls
are
false”
(#2)

Bottom line: By convenient choices of priors and alternatives
statistically significant differences can be evidence for the null

The
conflict
often
considers
the
two
sided
test

H0:
µ
=
0
versus
H1:
µ
≠
0

SPP
D.
Mayo
45

Posterior
Probabilities
in
H0

n
(sample
size)

____________________________

p

z

n=50

n=100

n=1000

.10

1.645

.65

.72

.89

.05

1.960

.52

.60

.82

.01

2.576

.22

.27

.53

.001

3.291

.034

.045

.124

If
n
=
1000,
a
result
statistically
significant
at
the
.05
level

leads
to
a
posterior
to
the
null
of
.82!

From
Berger
and
Sellke
(1987)
based
on
a
Jeffreys
pror

SPP
D.
Mayo
46

• With
a
z
=
1.96
difference,
the
95%
CI
(2-‐sided)
or
the
.975

CI
one
sided
excludes
the
null
(0)
from
the
interval

• Severity reasoning: Were H0 true, the probability of getting
d(x) < dobs is high (~.975), so SEV
(µ
>
0) ∼ .975
• But they give P(H0 | z = 1.96 ) = .82
• Error statistical critique: there’s a high probability that they
give posterior probability of .82 to H0:µ = 0 erroneously
• The onus is on probabilists to show a high posterior for H
constitutes having passed a good test.

SPP
D.
Mayo
47

Informal
and
Quasi-‐Formal
Severity
:
H
-‐>
H*

• Error
statisticians
avoid
the
fallacy
of
going
directly
from

statistical
to
research
hypothesis
H*

• Can
we
say
nothing
about
this
link?

• I
think
we
can
and
must,
and
informal
severity

assessments
are
relevant
(#3)

I
will
not
discuss
straw
man
studies
(“chump
effects”).

This is believable: Men react more negatively to success of
their partners than to their failures (compared to women)?
Studies have shown:
H: partner’s success lowers self-esteem in men

SPP
D.
Mayo
48

Macho
Men

H*: partner’s success lowers self-esteem in men

I
have
no
doubts
that
certain
types
of
men
feel
threatened

by
the
success
of
their
female
partners,
wives
or
girlfriends

I’ve
even
known
a
few.

Can
this
be
studied
in
the
lab?
Ratliff
and
Oishi
(2013)
did:

.

H*:
“men’s
implicit
self-‐esteem
is
lower
when
a
partner

succeeds
than
when
a
partner
fails.”

Not so for women
Their example does a good job, given the standards in place.

SPP
D.
Mayo
49

Treatments: Subjects are randomly assigned to five

“treatments”:
think,
write
about
a
time
your
partner

succeeded,
failed,
succeeded
when
you
failed
(partner

beats
me),
failed
when
you
succeeded
(I
beat
partner),

and
a
typical
day
(control).

Effects:
a
measure
of
“self-‐esteem”

Explicit:
“How
do
you
feel
about
yourself?”

Implicit:
a test of word associations with “me” versus “other”.
None showed statistical significance in explicit self-esteem, so
consider just implicit measures

SPP
D.
Mayo
50

Some null hypotheses: The average self-esteem score is no
different (these are statistical hypotheses)
a) when partner succeeds (rather than failing)
b) when partner beats (surpasses) me or I beat her
c) control: when she succeeds, fails, or it’s a regular day
There are at least double this, given self-esteem could be
“explicit” or “implicit” (others too, e.g., the area of success)

Only
null
(a)
was
rejected
statistically!

Should
they
have
taken
the
research
hypothesis
as

disconfirmed
by
negative
cases?

Or
as
casting
doubt
on
their
test?

SPP
D.
Mayo
51

Or
should
they
just
focus
on
the
null
hypotheses
that

were
rejected,
in
particular
null
(a),
for
implicit
self-‐esteem.

They
opt
for
the
third.

It’s not that they should have regarded their research
hypothesis H* as disconfirmed much less falsified.

This is precisely the nub of the problem! I’m saying the
hypothesis that the study isn’t well-run needs to be considered
• Is the artificial writing assignment sufficiently relevant to
the phenomenon of interest? (look at proxy variables)
• Is the measure of implicit self esteem (word associations) a
valid measure of the effect? (measurements of effects)

SPP
D.
Mayo
52

Take,
null
hypothesis
b):
The average self-esteem score is no
different when partner beats (surpasses) me or I beat her

Clearly
they
expected
“she
beat
me
in
X”
to
have
a
greater

negative
impact
on
self-‐esteem
than
“she
succeeded
at
X”.

Still,
they
could
view
it
as
lending
“some
support
to
the
idea

that
men
interpret
‘my
partner
is
successful’
as
‘my
partner

is
more
successful
than
me”
(p.
698),

….as
do
the
authors.

That
is,
any
success
of
hers
is
always
construed
by
Macho
man

as,
she
beat
me.

SPP
D.
Mayo
53

Bending
over
Backwards

For
the
stringent
self-‐critic,
this
skirts
too
close
to
viewing

the
data
through
the
theory,
a
kind
of
“self-‐sealing
fallacy”.

I want to be clear that this is not a criticism of them given
existing standards
“I'm talking about a specific, extra type of integrity...bending
over backwards to show how you're maybe wrong, that you
ought to have when acting as a scientist.”
(R. Feynman 1974)

I’m
describing
what’s
needed
to
show
“sincerely
trying
to

find
flaws”
under
the
austere
account
I
recommend

The
most
interesting
information
was
never
reported!

Perhaps
it
was
never
even
looked
at:
what
they
wrote
about.

SPP
D.
Mayo
54

Conclusion: Replication Research in Psychology Under an
Error Statistical Philosophy
Replication problems can’t be solved without correctly
understanding their sources

Biggest
sources
of
problems
in
replication
crises

(a) Stat
H
-‐>research
H*
and
(b)
biasing
selection
effects:

Reasons for (a): focus on P-values and Fisherian tests ignoring
N-P tests (and the illicit NHST that goes directly H–> H*)

SPP
D.
Mayo
55

Another reason, false dilemma:
probabilism or long-run performance
plus assuming that N-P can only give the latter
I argue for a third use of probability: Rather than report on
believability researchers need to report the properties of the
methods they used:
What was their capacity to have identified, avoided,
admitted bias?
What’s
wanted
is
not
a
high
posterior
probability
in
H

(however
construed)
but
a
high
probability
the
procedure

would
have
unearthed
flaws
in
H
(reinterpretation
of
N-‐P

methods)

SPP
D.
Mayo
56

What’s
replicable?
Discrepancies
that
are
severely
warranted

Reasons
for
(b)
[embracing
accounts
that
formally
ignore

selection
effects]:
accepting
probabilisms
that
embrace
the

likelihood
principle
LP

There’s
no
point
in
raising
thresholds
for
significance
if

your
methodology
does
not
pick
up
on
biasing
selection

effects.

SPP
D.
Mayo
57

Informal assessments of probativeness are needed to scrutinize
statistical inferences in relation to research hypotheses H –> H*
One
hypothesis
must
always
be:
our
results
point
to
the

inability
of
our
study
to
severely
probe
the
phenomenon
of

interest
(problem
with
proxy
variables,
measurements,
etc.)

The scientific status of an inquiry is questionable if it cannot or
will not distinguish the correctness of inferences from problems
stemming from a poorly run study
If ordinary research reports adopted the Feynman “bending over
backwards” scrutiny, the interpretation of replication efforts
would be more informative (or perhaps not needed)

SPP
D.
Mayo
58

REFERENCES

Baggerly,
K.
A.,
Coombes,
K.
R.
&
Neeley,
E.
S.
(2008).
“Run
Batch
Effects

Potentially
Compromise
the
Usefulness
of
Genomic
Signatures
for
Ovarian

Cancer.”
Journal
of
Clinical
Oncology.
26(7):
1186-‐1187.

Bartless,
T.
(2012).
“Daniel
Kahneman
Sees
‘Train-‐Wreck
Looming’
for
Social

Psychology”.
Chronicle
of
Higher
Education
Blog
(Oct.
4,
2012)
article

w/links
to
email
D.
Kahneman
sent
to
several
social
psychologists.

http://chronicle.com/blogs/percolator/daniel-‐kahneman-‐sees-‐train-‐
wreck-‐looming-‐for-‐social-‐psychology/31338.

Berger,
J.
O.
(2006).
“The
Case
for
Objective
Bayesian
Analysis.”
Bayesian

Analysis
1
(3):
385–402.

Berger,
J.
O.
&
Sellke,
T.
(1987).
“Testing
a
Point
Null
Hypothesis:
The

Irreconcilability
of
P
Values
and
Evidence
(with
Discussion).”
Journal
of
the

American
Statistical
Association
82
(397)
(March
1):
112–122.

Bhattacharjee,
Y.
(2013).
“The
Mind
of
a
Con
Man”.
The
New
York
Times

Magazine
(4/28/2013),
p.
44.

Cohen,
J.
1988.
Statistical
Power
Analysis
for
the
Behavioral
Sciences.
2nd
ed.

Hillsdale,
NJ:
Erlbaum.

SPP
D.
Mayo
59

Coombes,
K.
R.,
Wang,
J.
&
Baggerly,
K.
A.
(2007).
“Microrrays:
retracing
steps.”

Nature
Medicine.
13(11):1276-‐7.

Cox,
D.
R.
&
D.
V.
Hinkley.
(1974).
Theoretical
Statistics.
London:
Chapman
and

Hall.

Cox,
D.
R.
&
Mayo,
D.
G.
(2010).
“Objectivity
and
Conditionality
in
Frequentist

Inference.”
In
Error
and
Inference:
Recent
Exchanges
on
Experimental

Reasoning,
Reliability,
and
the
Objectivity
and
Rationality
of
Science,
edited

by
Deborah
G.
Mayo
and
Aris
Spanos,
276–304.
Cambridge:
Cambridge

University
Press.

Diaconis,
P.
(1978).
“Statistical
Problems
in
ESP
Research”.
Science
201
(4351):

131-‐136.
(Letters
in
response
can
be
found
in
the
Dec.
15,
1978
issue
pp.

1145-‐6.)

Dienes,
Z.
(2011)
“Bayesian
versus
Orthodox
Statistics:
Which
Side
Are
You
On?”

Perspectives
on
Psychological
Science
6(3):
274-‐290.

Feynman,
R.

(1974).
“Cargo
Cult
Science.”
Caltech
Commencement
Speech.

Fisher,
R.
A.
(1947).
The
Design
of
Experiments,
4th
ed.
Edinburgh:
Oliver
and

Boyd.

SPP
D.
Mayo
60

Gelman,
A.
(2011).
“Induction
and
Deduction
in
Bayesian
Data
Analysis.”
Edited

by
Deborah
G.
Mayo,
Aris
Spanos,
and
Kent
W.
Staley.
Rationality,
Markets

and
Morals:
Studies
at
the
Intersection
of
Philosophy
and
Economics
2

(Special
Topic:
Statistical
Science
and
Philosophy
of
Science):
67–78.

Gelman,
A.
&
Shalizi,
C.
(2013).
“Philosophy
and
the
Practice
of
Bayesian

Statistics.”
British
Journal
of
Mathematical
and
Statistical
Psychology
66
(1):

8–38.

Gigerenzer,
G.
(2000).
“The
Superego,
the
Ego,
and
the
Id
in
Statistical

Reasoning.
“
Adaptive
Thinking,
Rationality
in
the
Real
World,
OUP.

Goodman,
S.
N.
(1999).
Toward
evidence-‐based
medical
statistics.
2:
The
Bayes

factor.”
Annals
of
Internal
Medicine,
130:1005
–1013.

Howson,
C.
&
Urbach,
P.
(1993).
Scientific
Reasoning:
The
Bayesian
Approach.

2nd
ed.
La
Salle,
IL:
Open
Court.

Johansson
T.
(2010)
“Hail
the
impossible:
p-‐values,
evidence,
and
likelihood.”

Scandinavian
Journal
of
Psychology
52:113-‐125.

Kruschke,
J.
K.
(2010).
“What
to
believe:
Bayesian
methods
for
data
analysis”.

Trends
in
Cognitive
Science,
14(7):
297-‐300.

Lehmann,
E.
L.
(1993).
“The
Fisher,
Neyman-‐Pearson
Theories
of
Testing

SPP
D.
Mayo
61

Hypotheses:
One
Theory
or
Two?”
Journal
of
the
American
Statistical

Association
88
(424):
1242–1249.

Levelt
Committee,
Noort
Committee,
Drenth
Committee.
(2012).
“Flawed

science:
The
fraudulent
research
practices
of
social
psychologist
Diederik

Stapel”.
Stapel
Investigation:
Joint
Tilburg/Groningen/Amsterdam

investigation
of
the
publications
by
Mr.
Stapel.

https://www.commissielevelt.nl/

Lindley,
D.
V.
(1971).
“The
Estimation
of
Many
Parameters.”
In
Foundations
of

Statistical
Inference,
edited
by
V.
P.
Godambe
and
D.
A.
Sprott,
435–455.

Toronto:
Holt,
Rinehart
and
Winston.

Mayo,
D.
G.
(1996).
Error
and
the
Growth
of
Experimental
Knowledge.
Science
and

Its
Conceptual
Foundation.
Chicago:
University
of
Chicago
Press.

Mayo,
D.
G.
&
Cox,
D.
R.
(2010).
"Frequentist
Statistics
as
a
Theory
of
Inductive

Inference"
in
Error
and
Inference:
Recent
Exchanges
on
Experimental

Reasoning,
Reliability
and
the
Objectivity
and
Rationality
of
Science
(D.

Mayo
and
A.
Spanos
eds.),
Cambridge:
Cambridge
University
Press:
1-‐27.

This
paper
appeared
in
The
Second
Erich
L.
Lehmann
Symposium:

Optimality,
2006,
Lecture
Notes-‐Monograph
Series,
Volume
49,
Institute
of

Mathematical
Statistics,
pp.
247-‐275.

SPP
D.
Mayo
62

Mayo,
D.
G.,
and
A.
Spanos.
(2006).
“Severe
Testing
as
a
Basic
Concept
in
a

Neyman–Pearson
Philosophy
of
Induction.”
British
Journal
for
the

Philosophy
of
Science
57
(2)
(June
1):
323–357.

Mayo,
D.
G.,
and
A.
Spanos.

(2011).
“Error
Statistics.”
In
Philosophy
of
Statistics,

edited
by
Prasanta
S.
Bandyopadhyay
and
Malcom
R.
Forster,
7:152–198.

Handbook
of
the
Philosophy
of
Science.
The
Netherlands:
Elsevier.

Meehl,
P.
E.
&
Waller,
N.
G.
(2002).
“The
Path
Analysis
Controversy:
A
New

Statistical
Approach
to
Strong
Appraisal
of
Verisimilitude.”
Psychological

Methods
7(3):
283–300.

Morrison,
D.
E.
&
Henkel,
R.
E.
(eEds).
(1970).
The
Significance
Test
Controversy:

A
Reader.
Chicago:
Aldine
De
Gruyter.

Micheel,
C.
M.,
Nass,
S.
J.
&
Omenn
G.
S.
(Eds)
Committee
on
the
Review
of
Omics-‐
Based
Tests
for
Predicting
Patient
Outcomes
in
Clinical
Trials;
Board
on

Health
Care
Services;
Board
on
Health
Sciences
Policy;
Institute
of
Medicine

(2012).
Evolution
of
Translational
Omics:
Lessons
Learned
and
the
Path

Forward.
Nat.
Acad.
Press.

Neyman,
J.
(1957).
“‘Inductive
Behavior’”
as
a
Basic
Concept
of
Science.”
Revue

de
l'Institut
International
de
Statistique/Review
of
the
International

Statistical
Institute,
25
(1/3):
7-‐22.

SPP
D.
Mayo
63

Neyman,
J.
&
Pearson,
E.
S.
(1928).
“On
the
Use
and
Interpretation
of
Certain

Test
Criteria
for
Purposes
of
Statistical
Inference.
Part
I,”
Biometrica
20A:

175-‐240
(reprinted
in
Joint
Statistical
Papers,
University
of
California
Press,

Berkeley,
1967,
pp.
1-‐66.)

Popper,
K.
(1962).
Conjectures
and
Refutations:
The
Growth
of
Scientific

Knowledge.
New
York:
Basic
Books.

Potti,
A.,
Dressman
H.
K.,
Bild,
A.,
Riedel,
R.
F.,
Chan,
G.,
Sayer,
R.,
Cragun,
J.,

Cottrill,
H.,
Kelley,
M.
J.,
Petersen,
R.,
Harpole,
D.,
Marks,
J.,
Berchuck,
A.,

Ginsburg,
G.
S.,
Febbo,
P.,
Lancaster,
J.

&
Nevins,
J.
R.

(2006).
“Genomic

signatures
to
guide
the
use
of
chemotherapeutics.”
Nature
Medicine.
Nov

12(11):1294-‐300.
Epub
2006
Oct
22.

Potti,
A.
&
Nevins,
J.
R.
(2007)
“Reply
to
Coombes,
Wang
&
Baggerly.”
Nature

Medicine
Nov
13(11):1277-‐8.

Ratliff,
K.
A.
&
Oishi,
S.
(2013).
“Gender
Differences
in
Implicit
Self-‐Esteem

Following
a
Romantic
Partner’s
Success
or
Failure”.

Journal
of
Personality

and
Social
Psychology
105(4):
688–702.

Rosenkrantz,
R.
(1977).
Inference,
Method
and
Decision:
Towards
a
Bayesian

Philosophy
of
Science.
Dordrecht,
The
Netherlands:
D.
Reidel.

SPP
D.
Mayo
64

Savage,
L.
J.
(1962).
The
Foundations
of
Statistical
Inference:
A
Discussion.

London:
Methuen.

Savage,
L.
J.
(1964).
“The
Foundations
of
Statistics
Reconsidered.”
In
Studies
in

Subjective
Probability,
H.
Kyburg
&
H.
Smokler
(eds.),
173-‐188.
New
York:

John
Wiley
&
Sons.

Selvin,
H.
(1970).
“A
Critique
of
Tests
of
Significance
in
Survey
Research.”
In
The

Significance
Test
Controversy,
edited
by
D.
Morrison
and
R.
Henkel,
94-‐106.

Chicago:
Aldine
De
Gruyter.

Trafimow,
D.
&
Marks
M.
(2015).
“Editorial”.
Basic
and
Applied
Social
Psychology,

37(1),
pp.
1-‐2.

Wagenmakers,
E.-‐J.
(2007).
“A
Practical
Solution
to
the
Pervasive
Problems
of
P

Values”.
Psychonomic
Bulletin
&
Review
14
(5),
779-‐804.

D. Mayo: Replication Research Under an Error Statistical Philosophy

More Related Content

What's hot

Similar to D. Mayo: Replication Research Under an Error Statistical Philosophy

More from jemille6

Recently uploaded

D. Mayo: Replication Research Under an Error Statistical Philosophy