The framework and procedures of generalizability theory are introduced and illustrated in this instructional module that uses a hypothetical scenario involving writing proficiency. Generalizability analyses are useful for understanding the relative importance of various sources of error and for designing efficient measurement procedures. (SLD)

The monograph by D. Rogosa and G. Ghandour represents a body of cohesive and comprehensive research that can be the basis of a new measurement theory combining features of generalizability theory and strong true-score theory. Principles, approaches, arguments, and conclusions are reviewed; and critical comments are offered. (SLD)

Issues raised with the statistical models developed are discussed point by point, restating the emphasis on finite observation time, and reiterating the criticism of traditional psychometric methods. It is noted that the language and technical formulation of psychometrics can be extremely awkward in dealing with biased estimates. (SLD)

Investigates whether training raters to interpret the scoring dimensions on a rubric would increase reliability. Compares two groups of kindergarten and first-grade teachers: one group with training, one without. Finds that training increases raters' abilities to reliably interpret scoring items. (SC)

A measurement approach called generalizability theory (G-theory) is an important alternative to the more familiar classical measurement theory that yields less useful coefficients such as alpha or the KR-20 coefficient. G-theory is a theory about the dependability of behavioral measurements that allows the simultaneous estimation of multiple…

Psychometric theory has been one of psychology's stronger foundations and a major contributor to the recognition of psychology as a scientific discipline. Basic principles of psychometric theory led to the development of respected intelligence tests and large and comprehensive testing and assessment programs. This paper synthesizes major…

The Stages of Concerns Questionnaire (SoCQ) is an established instrument which has been used primarily with inservice teachers involved in innovations. While it focuses on inservice teachers' concerns about an innovation, the instrument is based on theoretical constructs developed by the work of Frances Fuller and others during the 1960s with…

The purpose for which a test is used and the examinees' stage of learning are two anchor points that are incorporated into a suggested framework for conducting construct validation studies for tests of students with English as a second language (ESL) or English as a foreign language (EFL). The framework includes the use of generalizability theory,…

The central thesis of this article is that the process of eliminating validity threats depends fundamentally on no-difference findings, a fact that has not been made explicit by researchers. The implications of this neglect are explored using examples from a number of different substantive areas such as psychology, health, and medicine.…

This study examined whether two variations on the typical Angoff group standard-setting process would produce sufficiently consistent results to recommend their use. The results imply that judgments gathered after an initial traditional group-process session can provide an efficient alternative mechanism for setting cutting scores using the Angoff…

This study examined the impact of rater agreement on decisions concerning the alignment between the Golden State Examination (GSE) in High School Mathematics and the University of California (UC) "Statement on Competencies in Mathematics." UC faculty and high school mathematics teachers (n = 20) rated the mathematics items of the GSE…

The peer review system in professional publications of scientific information is discussed in four articles. Topics discussed include the effects of information technology on scholarly communication channels, the publish or perish syndrome, the incidence of misrepresentation in scientific research, reasons for increasing difficulties in…

It is proposed that criteria for evaluating the quality of performance should be defined, at least tentatively, during the initial design of a performance assessment. Six characteristics of sound criteria are (1) significance; (2) fidelity; (3) generalizability; (4) developmental appropriateness; (5) accessibility; and (6) utility. (SLD)

Evidence is presented on the generalizability and convergent validity of performance assessments using data from six studies of student achievement that sampled a wide range of measurement facets and methods. Results at individual and school levels indicate that task-sampling variability is the major source of measurement error. (SLD)

Examined the generalizability of written-response scores on the English 30 diploma examination administered to Alberta 12th-grade students. Student scores differed as a function of rater, but this variance component was small across two tasks and two administrations; score generalizability was high using a two-rater system; and scale variability…