Most existing solutions to the current replication crisis in science address only the factors stemming from specific poor research practices. We introduce a novel mechanism that leverages the experts' predictive abilities to analyze the root causes of replication failures. It is backed by the principle that the most accurate predictor is the most…

The year 2022 is the 20th anniversary of Joseph Schafer and John Graham's paper titled "Missing data: Our view of the state of the art," currently the most highly cited paper in the history of "Psychological Methods." Much has changed since 2002, as missing data methodologies have continually evolved and improved; the range of…

The design and psychometric methodology of the National Assessment of Educational Progress (NAEP) is constantly evolving to meet the changing interests and demands stemming from a rapidly shifting educational landscape. NAEP has been built on strong research foundations that include conducting extensive evaluations and comparisons before new…

The TOEFL iBT test was introduced in 2005 to better reflect the language demands of real-life academic tasks than did previous versions of the test. The task-based design of the test was intended to support the interpretation of its scores as a trustworthy measure of international students' ability to use English in an academic environment. Until…

It is imperative that counselors understand how to critically evaluate assessments before using them to make clinical decisions. This evaluation can be conducted through integrating the 5 sources of validity. Each source of validity is discussed, along with methods to appraise psychometric quality, throughout this special issue.

Fully updated to reflect the most recent changes in the field, the Second Edition of "Propensity Score Analysis" provides an accessible, systematic review of the origins, history, and statistical foundations of propensity score analysis, illustrating how it can be used for solving evaluation and causal-inference problems. With a strong…

Advances in validity theory and alacrity in validation practice have suffered because the term "validity" has been used to refer to two incompatible concerns: (1) the degree of support for specified interpretations of test scores (i.e. intended score meaning) and (2) the degree of support for specified applications (i.e. intended test…

This article is intended to assist teachers and researchers in designing studies that examine the efficacy of a particular intervention or strategy with students with sensory disabilities. Ten research designs that can establish causal inference (the ability to attribute any effects to the intervention) with and without randomization are discussed.

The scoring process is critical in the validation of tests that rely on constructed responses. Documenting that readers carry out the scoring in ways consistent with the construct and measurement goals is an important aspect of score validity. In this article, rater cognition is approached as a source of support for a validity argument for scores…

Many states now possess the data and statistical methods that can produce teacher value-added scores and link them to preparation programs. It is important to understand the limitations of these measures and the inferences that they do and do not support. These limitations fall into three categories. First, value-added measures (VAM) provide…

This Methodological Brief introduces the reader to the regression discontinuity design (RDD), which is a method that when used correctly can yield estimates of research treatment effects that are equivalent to those obtained through randomized control trials and can therefore be used to infer causality. However, RDD does not require the random…

The argument-based approach to validation involves two steps; specification of the proposed interpretations and uses of the test scores as an interpretive argument, and the evaluation of the plausibility of the proposed interpretive argument. More ambitious interpretations and uses tend to involve an extended network of inferences and assumptions…

The 1999 "Standards for Educational and Psychological Testing" defines validity as the degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests. Although quite explicit, there are ways in which this definition lacks precision, consistency, and clarity. The history of validity has taught us…

This paper outlines a procedure for language textbook analysis from the perspective of second language acquisition (SLA) principles as a preliminary procedure to evaluation for selection. The aim is to provide a tool that allows comparison of the potential of textbooks for supporting students' language learning. To this end, ten general principles…

This study aims to examine a) whether vocabulary knowledge, captured in the Vocabulary Levels Test (VLT), is related to the performance on the five types of reading comprehension items tested in TOEFL, i.e., Guessing Vocabulary, Main Idea, Inference, Reference, and Stated Detail; and b) whether EFL learners with different levels of vocabulary…