Frontiers of Test Validity Theory Measurement, Causation,

and Meaning - 1st Edition

Visit the link below to download the full version of this book:

https://medipdf.com/product/frontiers-of-test-validity-theory-measurement-causat
ion-and-meaning-1st-edition/

Click Download Now

This page intentionally left blank
Frontiers of Test Validity
Theory
Measurement, Causation,
and Meaning

Keith A. Markus
and
Denny Borsboom

Routledge
i Taylor & Francis G roup

NEW YORK A N D LO N D O N
First published 2013
by Routledge
711 Third Avenue, New York, NY 10017
Simultaneously published in the UK
by Routledge
27 Church Road, Hove, East Sussex BN3 2FA
Routledge is an imprint of the Taylor & Francis Group, an informa business
© 2013 Psychology Press
The right of Keith A. Markus and Denny Borsboom to be identified
as authors of this work has been asserted by them in accordance with
sections 77 and 78 of the Copyright, Designs and Patents Act 1988.
All rights reserved. No part of this book may be reprinted or reproduced
or utilised in any form or by any electronic, mechanical, or other means,
now known or hereafter invented, including photocopying and recording,
or in any information storage or retrieval system, without permission in
writing from the publishers.
Trademark notice: Product or corporate names may be trademarks or
registered trademarks, and are used only for identification and
explanation without intent to infringe.
Library of Congress Cataloging in Publication Data
Markus, Keith A.
Frontiers in test validity theory : measurement, causation and meaning /
Keith A. Markus and Denny Borsboom.—1 Edition.
pages cm
Includes bibliographical references and index.
1. Psychometrics. I. Borsboom, Denny. II. Title.
BF39.M277 2013
150.28'7—dc23
2012047896

ISBN: 978–1–84169–219–7 (hbk)

ISBN: 978–1–84169–220–3 (pbk)
ISBN: 978–0–203–50120–7 (ebk)

Typeset in Sabon
by Swales & Willis Ltd, Exeter, Devon
Contents

About the Authors xii

Preface xiii
Acknowledgements xvii

1 Introduction: Surveying the Field of Test Validity Theory 1

1.1. Terminology 1
1.2. The Development of Test Validity Theory 5
1.3. Measurement, Causation, Meaning, and Test Validity 15
1.4. Further Reading 18

PART I:
2 Philosophical Theories of Measurement 19
2.1. The Classical Theory of Measurement 21
2.2. Addition and Concatenation 25
2.3. The Representational Theory of Measurement:
Stevens’ Version 27
2.4. The Representational Theory of Measurement: The
Axiomatic Approach 32
2.5. The Latent Variable Perspective 36
2.6. Chapter Conclusion 41
2.7. Further Reading 43

3 Psychometric Models 44
3.1. Psychometric Models 45
3.2. Test Scores, Constructs, and Validity 52
3.3. The Relation Between Validity and Other Psychometric
Concepts 56
3.4. Chapter Conclusion 64
3.5. Further Reading 65
viii Contents
4 Open Issues in Measurement Theory and Psychometrics 66
4.1. Measurement and the Structure of Psychological
Attributes 67
4.2. Measurement and Probabilistic Models 71
4.3. A Possible Reconciliation Between Theories of
Measurement 76
4.4. Is Measurement Necessary in Psychological Testing? 77
4.5. Chapter Conclusion 79
4.6. Further Reading 80

PART II:
5 Test Scores as Samples: Behavior Domain Theory 81
5.1. Conceptualizing Domain Scores 86
5.2. The Item Writing Argument for Behavior Domain
Theory 90
5.3. The Distinctness Argument Against Domain Score
Causation 92
5.4. Reconciling Behavior Domains and Causal Theories of
Measurement 96
5.5. Chapter Conclusion 103
5.6. Further Reading 104

6 Causality in Measurement 105

6.1. Causal Structures 108
6.2. Implications for Validity Theory 117
6.3. Direct, Indirect, and Mixed Models 121
6.4. Other Structures 125
6.5. Chapter Conclusion 136
6.6. Further Reading 136

7 Causation, Correlation, and Reflective Measurement

Models 137
7.1. Reflective Measurement With and Without Causation 140
7.2. Statistical Unidimensionality versus Causal
Unidimensionality 148
7.3. Three General Theories of Causation 153
7.4. Reflective Measurement Models with Different Causal
Interpretations 159
7.5. Chapter Conclusion 171
7.6. Further Reading 171
 Contents ix
8 Problems in Causation and Validity: Formative
Measurement, Networks, and Individual Differences 173
8.1. Formative Measurement Models and Causation 174
8.2. Network Measurement Models 183
8.3. Causation and Individual Differences 184
8.4. Validating Causal Measurement Models 190
8.5. Chapter Conclusion 193
8.6. Further Reading 195

PART III:
9 Interpreting Test Responses: Validity, Values, and
Evaluation 196
9.1. Messick’s Unified View of Validity 197
9.2. Historical Context and Recent Literature 199
9.3. Shadish, Cook, and Campbell’s Critique 200
9.4. Scriven’s Critique 205
9.5. Searl’s Naturalistic-Fallacy Fallacy 207
9.6. Putnam’s Critique of the Fact/Value Dichotomy 210
9.7. Conceptualizing Test Validity and its Consequences 216
9.8. Chapter Conclusion 219
9.9. Further Reading 220

10 A Model of Test Score Interpretation 221

10.1. Focal Example: Tuna Cleaner Work Sample 224
10.2. Interpretation and Entailment: Construct Theories
as Inference Engines 225
10.3. Interpretations of Test Scores 229
10.4. Chapter Conclusion 245
10.5. Further Reading 246

11 Open Questions About Test Score Meaning 247

11.1. The Psychologist’s Fallacy and Test Score
Interpretation 248
11.2. Meaning’s in the Heads of Test Takers? 253
11.3. Heteroglossia: Speaking in Diverse Tongues 258
11.4. Test Takers and Their Beliefs 264
11.5. Validity Without Determinate Meaning 269
11.6. Chapter Conclusion 272
11.7. Further Reading 272
x Contents
PART IV:
12 An Integrative View of Test Validity 273
12.1. An Integrative Model of Test Validation 274
12.2. Truth, Justification, Knowledge, and Action 280
12.3. Kinds of Causality and Types of Evidence 283
12.4. Justification Through Alignment 289
12.5. Knowledge, Value, and Action 291
12.6. Chapter Conclusion 292

13 Epilogue as Dialog: The Future of Test Validity Theory 294

13.1. Test Validation Methods 294
13.2. Test Validity Theory and Meaning 297
13.3. The Yardstick by Which Validation is Judged (If There
is One) 300
13.4. Validity and Response Processes 304
13.5. Attributes 308
13.6. Tests and Interpretations 312
13.7. Chapter Conclusion 316

Notes 317
References 319
Author Index 336
Subject Index 338
Example Index 340
About the Authors

Keith A. Markus is a Professor in the Department of Psychology at John

Jay College of Criminal Justice of The City University of New York
(CUNY). He received his PhD in Psychology from the CUNY Graduate
School. His research interests focus on causal models and causal infer-
ence, test validity, statistical inference, and applications of latent variable
modeling. He has served as Associate Editor of Psychological Methods
and of Structural Equation Modeling: A Multidisciplinary Journal. Dr.
Markus teaches courses in psychometrics, structural equation modeling,
program evaluation, and other topics in quantitative methods.
Denny Borsboom is a Professor in the Department of Psychology at the
University of Amsterdam where he received his PhD in Psychology in
2003. His research has focused on the conceptual background of psycho-
metric models, validity theory, and network modeling. He has published
on these issues in journals like Behavioral and Brain Sciences, Psychologi-
cal Review, Psychometrika, Psychological Methods, and Annual Review
of Clinical Psychology. In addition, he authored the monograph Measur-
ing the Mind: Conceptual Issues in Contemporary Psychometrics, which
was published by Cambridge University Press. Dr. Borsboom teaches
courses in Research Methods, Network Analysis, and Psychometrics. He
is a member of the Psychometric Society Board of Trustees.
Preface

There is much to capture the imagination in test validity theory. Unfor-

tunately, the challenges and unsolved puzzles of test validity often do not
come across as readily as they do in other areas of testing. In this book
we seek to throw Pandora’s Box wide open and shine a light on the fan-
tastic trove of complex and interconnected problems at the frontiers of
test validity theory. In so doing, we hope to attract renewed attention and
stimulate further advances in this critical domain of test theory. Argu-
ably, there is no other domain as critical to effective test design, construc-
tion, interpretation and use. If scores derived from a test do not inform
us about the desired construct, then it makes little difference how reli-
able, how precise, or how free from biases the scores may be. However,
validity theory is not a topic of intense research focus, in either practical
testing, scientific research, or research into psychometric methods. New
breakthroughs in test validity theory do not flood related journals. Cer-
tainly, the rate of progress does not compare to progress in reliability
estimation methods or new statistical models for test item scores. For a
field that commonly holds validity to be the central psychometric issue in
testing, this is a rather surprising situation.
The explanation for this situation may in part rest with the fact that
test validity is a very unusual specialization within a very unusual sub-
field. Most students attracted to behavioral science disciplines such as
psychology, sociology, education, or political science are not attracted to
these disciplines by their love of mathematics. Test theory, then, repre-
sents an unusual sub-field within behavioral science disciplines because it
represents the most mathematical sub-field for most such disciplines. In
contrast to test theory as a whole, test validity represents the least math-
ematical specialization within the most mathematical sub-field of less
mathematical disciplines. So, the path to a specialization in test validity
involves an unusual winding route with relatively little traffic to follow.
In this book, the reader will find both a guided tour of existing work and
also reports of some new expeditions into uncharted territory.
Students seeking entry into the problems and puzzles of test valid-
ity theory will find the book useful, particularly students who have
xiv Preface
completed a first course in testing or measurement. Test developers and
test users seeking additional conceptual tools to put to use in validating
their tests or testing programs will also find it useful. Additionally, mem-
bers of the same audience seeking to sharpen their test validity report-
ing practices will find it useful. Finally, the book breaks sufficient new
ground that we also hope that test validity specialists will find useful
contributions to existing theory between its covers.
We have tried to present the material in a way that makes it accessible
to a broad audience and does not assume a great deal of prior familiarity
with the material. It is not possible, however, to embed within a book on
test validity theory a comprehensive introduction to test theory or com-
prehensive discussion of the relevant statistical or philosophical back-
ground. As such, necessary background is often introduced in small doses
without any attempt to provide encyclopedic coverage. We present math-
ematical and statistical material in conceptual overview, without present-
ing the details of areas like axiomatic measurement theory, item response
theory, or structural equation models. We assume no background in rel-
evant philosophical material. Nonetheless, such material receives tightly
circumscribed introduction without any attempt to present broader con-
texts beyond the purposes at hand. If we have succeeded in our goals,
readers who bring considerable background knowledge to the book will
find that the material in the book engages with that background without
oversimplification or distortion. Readers who come to the book with less
background will find the treatment readable and understandable.
The book explores each of the above frontiers of test validity theory in
turn: measurement, causation and then meaning. The treatment is progres-
sive in the sense that one need not scratch too deeply below the surface of
current testing practices to encounter problems related to measurement.
If one digs a little deeper, questions related to causation begin to emerge.
Questions of meaning emerge at even deeper levels beneath the surface of
everyday practice. However, the treatment is also progressive in the sense
that later chapters often assume material introduced in earlier chapters.
Readers with less background will therefore find it advantageous to read
the chapters in order. In contrast, readers with more background will
find that chapters of specific interest to them stand alone. Throughout
the book, however, we draw not only from the literature of test theory,
but also from broader theoretical and philosophical sources that directly
inform issues involved in test theory. To facilitate reading chapters in
isolation, each of the remaining chapters begins with a box containing a
brief summary of material assumed from prior chapters.
The book devotes a subsection containing three or four chapters to
each of these three frontiers, yielding ten core chapters. Each core chapter
includes boxed examples threading a single central example through the
various issues explored in the chapter. We have attempted to choose a
range of examples that reflect the breadth of testing across various dis-
 Preface xv
ciplines including psychology, education, and others. Each core chapter
also includes a list of further reading at the end, which includes key ref-
erences for that chapter, and suggestions for readers who want to fur-
ther explore issues taken up in that chapter. In one form or another, the
last chapter in each section takes up open questions that remain fertile
areas for further research and further contributions to test validity theory.
Taken together, each of the three main sections presents the reader with
a range of theoretical options and unpacks each of those options in some
detail. However, each of these three frontiers could easily support a full-
length book treatment in and of itself. As such, none receive a compre-
hensive treatment.
In chapter 1, we first tackle the formidable challenge of constructing
a systematic terminology for use in the remainder of the book. We then
review how test validity reached its current state of development and
then introduce the three central problems of measurement, causation and
meaning. In chapter 2 we canvass central ideas in measurement theory
and relate them to testing. In chapter 3 we contrast various statistical
models of observed scores in relation to their assumptions. In chapter 4
we explore persistent difficulties in sorting out the roles of axioms and of
latent variables in understanding measurement in behavioral sciences. A
picture emerges from the three chapters in Part One of measurement, con-
strued more narrowly, and testing, construed more broadly, as theoreti-
cally diverse fields in which different testing applications call for different
theoretical approaches.
In chapter 5 we explore the viability of a non-causal approach to such
models based on domain sampling and also the prospects for reconcil-
ing domain sampling with a causal theory of measurement. In chapter
6 we contrast various causal models relating observed scores to latent
variables, or to the construct measured by the test. In chapters 7 and
8 we canvass several broad approaches to understanding causation and
what they add to a non-causal statistical interpretation of test score mod-
els. The picture that emerges from the chapters that comprise Part Two
arrays various causal and non-causal interpretations of statistical models
of test item scores as contrasting alternatives open to the test developer
that involve a series of trade-offs between evidence required to support
them and the conclusions that they in turn support.
In chapter 9 we consider the role of values, in the sense of what is
considered to have worth or merit, in test validation. In chapter 10 we
develop an approach to modeling test score interpretation. In chapter 11
we probe questions along the frontier of test validity theory and mean-
ing. Part Three comprises a set of forays into the terra incognita of what
stands as the least well-understood frontier of test validity theory, but also
a frontier that at some level encompasses all else and allows for no exter-
nal vantage point. Exploration of this frontier remains too preliminary
for a clear overall picture to emerge. Nonetheless, current approaches to
xvi Preface
test validation in terms of validity arguments offer a useful guide to this
largely unexplored region.
In chapter 12 we integrate the material from the three main sections
and relate this material to the practical task of test validation. Written as
a dialog between the authors, chapter 13 offers both a retrospective look
at the material covered in the earlier chapters and also a prospective look
at the broader context of that material. Together constituting Section
Four, these chapters allow us to address the interrelationships between
the issues explored in the previous sections and the broader relationships
with test validity theory and test validation practice. In the end, however,
a single book can merely begin a process of exploration that we expect to
see continue for decades to come. If nothing else, we hope that the chap-
ters that follow will put to rest any perception that test validity theory in
its current form represents a nearly completed project with little room
remaining for new contributions or advances. Nothing could fall further
from the truth.