Measuring opportunities to learn (OTL) is crucial for evaluating education quality and equity, but obtaining accurate and comprehensive OTL data at a large scale remains challenging. We attempt to address this issue by investigating measurement concerns in data collection and sampling. With the primary goal of estimating group-level OTLs for large…

While it is easy to assume that university students who wait until the last minute to complete surveys for their class research requirements provide low-quality data, this issue has not been empirically examined. The goal of the present study was to examine the relation between student research procrastination and two important data quality…

Thanks to COVID-19, schools were closed and tests were canceled. The result is that we may not see test-taking data typically seen before. For some analyses, sample sizes may not meet the minimum requirement. For others, the sample of test-takers may be different from previous years. In some situation, there may be no data at all. What do we do in…

Technical documentation for educational tests focuses primarily on properties of individual scores at single points in time. Reliability, standard errors of measurement, item parameter estimates, fit statistics, and linking constants are standard technical features that external stakeholders use to evaluate items and individual scale scores.…

Many educational summative and formative assessments have been transferred to a remote online setting because of the pandemic. Educational professionals and stakeholders have shown interest in learning how this change in the test mode influenced test takers; that is, whether test-taking experiences in a remote test setting were different from…

In this digital ITEMS module, Dr. Sue Lottridge, Amy Burkhardt, and Dr. Michelle Boyer provide an overview of automated scoring. Automated scoring is the use of computer algorithms to score unconstrained open-ended test items by mimicking human scoring. The use of automated scoring is increasing in educational assessment programs because it allows…

Within classrooms, psychologists and teachers use direct behavior observation methods, systematic behavior observations (SBOs) and direct behavior ratings (DBRs), to gather information about students' behaviors for the purposes of making decisions related to diagnosis and classroom management or behavioral feedback respectively. Observers use SBOs…

In this ITEMS module, we provide a two-part introduction to the topic of reliability from the perspective of "classical test theory" (CTT). In the first part, which is directed primarily at beginning learners, we review and build on the content presented in the original didactic ITEMS article by Traub and Rowley (1991). Specifically, we…

As the standards movement continues into its third decade, there remains a need for alignment methodologies that can be broadly applied to study instruction and policy. This article reports on a series of development efforts meant to revise the Surveys of Enacted Curriculum (SEC) surveys and methods to study the implementation of new college- and…

Data mining methods for classification and regression are becoming increasingly popular in various scientific fields. However, these methods have not been explored much in educational measurement. This module first provides a review, which should be accessible to a wide audience in education measurement, of some of these methods. The module then…

Simulation studies are fundamental to psychometric discourse and play a crucial role in operational and academic research. Yet, resources for psychometricians interested in conducting simulations are scarce. This Instructional Topics in Educational Measurement Series (ITEMS) module is meant to address this deficiency by providing a comprehensive…

This article has three goals. The first goal is to clarify the role that the consequences of test score use play in validity judgments by reviewing the role that modern writers on validity have ascribed for consequences in supporting validity judgments. The second goal is to summarize current views on who is responsible for collecting evidence of…

"Scaling" is the process of constructing a score scale that associates numbers or other ordered indicators with the performance of examinees. Scaling typically is conducted to aid users in interpreting test results. This module describes different types of raw scores and scale scores, illustrates how to incorporate various sources of…

Discusses some practical issues in linking educational assessments, focusing on the importance of clarity of purpose when assessments are linked. Also stresses the importance of the design used to collect data for linking. Uses linking studies from a variety of situations to illustrate these points. (SLD)

Outlines a series of security steps that might be taken by researchers or organizations that are contemplating Web-based tests and performance assessments. Focuses on what can be done to avoid the loss, compromising, or modification of data collected by or stored through the Internet. (SLD)