This paper follows a line of logical argumentation to claim that what Samuel Messick conceptualized about construct validation has probably been misunderstood by some educational policy makers, practicing educators, and classroom teachers. It argues that, while Messick's unified theory of test validation aimed at (a) warning educational…

Evaluators use their eyes to see what is there, whether it is intended or not. But they use their test instruments to measure what is intended, whether it is there or not. Evaluators have been broadening their repertoire of instruments for years: curriculum-embedded tests, observer checklists, audiotape recorders, videotape recorders, unobtrusive…

Changes to state tests impact the ability of State Education Agencies (SEAs) to monitor change in performance over time. The purpose of this study was to evaluate the Standardized Performance Growth Index (PGIz), a proposed statistical model for measuring change in student and school performance, across transitions in tests. The PGIz is a…

The commentary and overview explored how curriculum and assessment reforms are being used by a small university and small country to improve the quality of education and gain international recognition. Although the reforms are potentially beneficial to the students, university, and country, there are dilemmatic factors that may either enhance or…

This paper provides a history of ETS's role in developing assessment instruments and psychometric procedures for measuring change in large-scale national assessments funded by the Longitudinal Studies branch of the National Center for Education Statistics. It documents the innovations developed during more than 30 years of working with…

In "Unique Characteristics of Diagnostic Classification Models: A Comprehensive Review of the Current State-of-the-Art," Rupp and Templin (2008) undertake the ambitious task of providing a thorough portrait of the current state of diagnostic classification models (DCM). In this commentary, the author applauds Rupp and Templin for their…

The purpose of this study is to compare the teacher retention rates in public elementary and middle schools in Georgia that met or did not meet the academic performance component of Adequate Yearly Progress. The teacher retention rates were expected to be higher in schools that met the academic performance component of AYP and lower in the schools…

In this commentary, the author addresses what is referred to as the deterministic input, noisy "and" gate (DINA) model. The author mentions concerns with how this model has been formulated and presented. In particular, the author points out that there is a lack of recognition of the confounding of profiles that generally arises and then discusses…

Rupp and Templin (2008) do a good job at describing the ever expanding landscape of Diagnostic Classification Models (DCM). In many ways, their review article clearly points to some of the questions that need to be answered before DCMs can become part of the psychometric practitioners toolkit. Apart from the issues mentioned in this article that…

In this commentary, the authors discuss some of the issues regarding the use of diagnostic classification models that practitioners should keep in mind. In the authors experience, these issues are not as well known as they should be. The authors then provide recommendations on diagnostic scoring.

The gold standard in making causal inference on program effects is a randomized trial. Most randomization designs in education randomize classrooms or schools rather than individual students. Such "clustered randomization" designs have one principal drawback: They tend to have limited statistical power or precision. This study aims to…

In educational testing, subscores may be provided based on a portion of the items from a larger test. One consideration in evaluation of such subscores is their ability to predict a criterion score. Two limitations on prediction exist. The first, which is well known, is that the coefficient of determination for linear prediction of the criterion…

On a general level, the objective of diagnostic classifications models (DCMs) lies in a classification of individuals regarding multiple latent skills. In this article, the authors show that this objective can be achieved by multidimensional adaptive testing (MAT) as well. The authors discuss whether or not the restricted applicability of DCMs can…

Michigan's superintendent of public instruction is convinced that a state testing program can provide the kind of information educators need to improve instructional planning. (Author)

It is argued that, by design, norm-referenced tests (NRT) and criterion-referenced tests (CRT) are conceived with different frames of reference. They are not totally exclusive of each other, but they do direct attention to different uses and references for information and decision making. Their combined contributions allow a more detailed and…