Research in the social and behavioral sciences relies on a wide range of experimental and quasi-experimental designs to estimate the causal effects of specific programs, policies, and events. In this paper we highlight measurement issues relevant to evaluating the validity of causal estimation and generalization. These issues impact all four…

Researchers across many fields have called for greater attention to heterogeneity of treatment effects--shifting focus from the average effect to variation in effects between different treatments, studies, or subgroups. True heterogeneity is important, but many reports of heterogeneity have proved to be false, non-replicable, or exaggerated. In…

Inference is considered an important factor in comprehension models and has been described as a causal factor in predicting comprehension. To date, specific tests for inference are rare and often rely on specific thematic texts. This reliance on thematic inference may raise some concerns as inference is related to prior text-specific knowledge.…

The digital transition in educational testing has introduced many new opportunities for technology to enhance large-scale assessments. These include the potential to collect and use log data on test-taker response processes routinely, and on a large scale. Process data has long been recognised as a valuable source of validation evidence in…

The evaluation of learning in mathematics is a worldwide problem, therefore, new methods are required to assess the understanding of mathematical concepts. In this paper, we propose to use the Item Response Theory to analyze the understanding level of undergraduate students about the real function mathematical concept. The Bayesian approach was…

The State of Washington requires school districts to file court petitions on students with excessive unexcused absences. The "Washington Assessment of Risks and Needs of Students" (WARNS), a self-report screening instrument developed for use by high school and juvenile court personnel in such situations, purports to measure six facets of…

In this study, we developed a Reading for Meaning special education teacher observation rubric that detailed the elements of evidence-based comprehension instruction and tested its psychometric properties using many-facet Rasch measurement. We collected video observations of classroom instruction from 10 special education teachers across 3 states…

Cognitive Diagnostic Assessment (CDA) is an alternative assessment which can give a clear picture of pupils' learning process and cognitive structures to education stakeholders so that appropriate instructional strategies can be designed to tailored pupils' needs. Coincide with this function, the Ordered Multiple-Choice (OMC) items were…

The University of Reading Learning Gain project is a three-year longitudinal project to test and evaluate a range of available methodologies and to draw conclusions on what might be the right combination of instruments for the measurement of Learning Gain in higher education. This paper analyses the validity of a measure of critical thinking…

Making inferences and reasoning with new scientific information is critical for successful performance in biology coursework. Thus, identifying students who are weak in these skills could allow the early provision of additional support and course placement recommendations to help students develop their reasoning abilities, leading to better…

This paper aims to present a critical analysis of the Year 1 Phonics Screening Check (PSC), with special focus on the relationship between the UK Department for Education's policy-making and the evidence considered in the process of developing and evaluating the PSC. The reports from the in-house Standards and Testing Agency and from commissioned…

In this study, we developed a Reading for Meaning special education teacher observation rubric that details the elements of evidence-based comprehension instruction and tested its psychometric properties using many-faceted Rasch measurement (MFRM). Video observations of classroom instruction from 10 special education teachers across three states…

The computer-administered Multiple-Choice Online Causal Comprehension Assessment (MOCCA) for Grades 3 to 5 has an innovative, 40-item multiple-choice structure in which each distractor corresponds to a comprehension process upon which poor comprehenders have been shown to rely. This structure requires revised thinking about measurement issues…

The computer-administered Multiple-Choice Online Causal Comprehension Assessment (MOCCA) for Grades 3 to 5 has an innovative, 40-item multiple-choice structure in which each distractor corresponds to a comprehension process upon which poor comprehenders have been shown to rely. This structure requires revised thinking about measurement issues…

We investigated the dimensionality of inference making in samples of 4- to 9-year-olds (Ns = 416-783) to determine if local and global coherence inferences could be distinguished. In addition, we examined the validity of our experimenter-developed inference measure by comparing with three additional measures of listening comprehension. Multitrait,…