This study presents several multivariate Generalizability theory designs for analyzing automatic item-generated (AIG) based test forms. The study used real data to illustrate the analysis procedure and discuss practical considerations. We collected the data from two groups of students, each group receiving a different form generated by AIG. A…

Clustered data structures are common in many areas of educational and psychological research (e.g., students clustered in schools, patients clustered by clinician). In the course of conducting research, questions are often administered to obtain scores reflecting latent constructs. Multilevel measurement models (MLMMs) allow for modeling…

Data visualization is a core tenet of communicating measurement research and outcomes. Measurement professionals utilize data visualization in various phases of research, including exploration and communication. However, data visualization has not received enough attention in the measurement field. While it is true that many measurement graphics…

Technical difficulties occasionally lead to missing item scores and hence to incomplete data on computerized tests. It is not straightforward to report scores to the examinees whose data are incomplete due to technical difficulties. Such reporting essentially involves imputation of missing scores. In this paper, a simulation study based on data…

Country differences in response styles (RS) may jeopardize cross-country comparability of Likert-type scales. When adjusting for rather than investigating RS is the primary goal, it seems advantageous to impose minimal assumptions on RS structures and leverage information from multiple scales for RS measurement. Using PISA 2015 background…

In this ITEMS module, we provide a didactic overview of the specification, estimation, evaluation, and interpretation steps for diagnostic measurement/classification models (DCMs), which are a promising psychometric modeling approach. These models can provide detailed skill- or attribute-specific feedback to respondents along multiple latent…

I study student characteristics and academic performance at the University of California, where consideration of an applicant's ethnicity has been banned since 1996 and SAT scores were used in admitting students to the university until fall 2021. I show the following: (1) SAT scores were more important than high school grades in predicting…

In this digital ITEMS module, Dr. Jeffrey Harring and Ms. Tessa Johnson introduce the linear mixed effects (LME) model as a flexible general framework for simultaneously modeling continuous repeated measures data with a scientifically defensible function that adequately summarizes both individual change as well as the average response. The module…

Drawing upon his experience, more than 60 years ago, as a psychometric support person to a very special teacher of brain damaged children, the author of this article reflects on the productive use of educational assessments and data from them to educate - assessment in the service of learning. Findings from the Gordon Commission on the Future of…

In this digital ITEMS module, Nikole Gregg and Dr. Brian Leventhal discuss strategies to ensure data visualizations achieve graphical excellence. Data visualizations are commonly used by measurement professionals to communicate results to examinees, the public, educators, and other stakeholders. To do so effectively, it is important that these…

Discusses regression toward the mean and the confusion this can cause in the interpretation of educational data. The regression effect has two causes, imperfect measurement of the variables and imperfect correlation between the variables. Uses the example of reading achievement test scores in a Hawaiian private school to illustrate the discussion.…

This essay argues that reliability should be reconceptualized in a way that reflects the importance of the theoretical expectations of the test specialist and the learning and problem solving of the test takers. It is time to characterize clearly the substantive theoretical framework supporting reliability studies and the technical evaluation of…

MicroCAT version 3.0--an integrated test development, administration, and analysis system--is reviewed in this first article of a series on testing software. A framework for comparing testing software is presented. The strength of this package lies in the development, banking, and administration of items composed of text and graphics. (SLD)

Two easy-to-use microcomputer programs, the Student Problem Package and the Test Analysis Package, both by D. L. Harnisch et al. (1985), are described. These programs efficiently analyze test data for teachers. (SLD)

The purpose of this article was to model United States Medical Licensing Examination (USMLE) Step 2 passing rates using the Cox Proportional Hazards Model, best known for its application in analyzing clinical trial data. The number of months it took to pass the computer-based Step 2 examination was treated as the dependent variable in the model.…