The Angoff method requires experts to view every item on the test and make a probability judgment. This can be time consuming when there are large numbers of items on the test. In this study, a G-theory framework was used to determine if a subset of items can be used to make generalizable cut-score recommendations. Angoff ratings (i.e.,…

Studied three procedures for estimating the standard errors of school passing rates using a generalizability theory model and considered the effects of student sample size. Results show that procedures differ in terms of assumptions about the populations from which students were sampled, and student sample size was found to have a large effect on…

The problem of predicting universe scores for samples of examinees based on their responses to samples of items is treated. The measurement model categorizes items according to the cells of a table of test specifications, and the linear function derived for minimizing error variance in prediction uses responses to these categories. (Author/JKS)

The zeitgeist for reform in education precipitated a number of changes in assessment. Among these are performance assessments, sometimes linked to "high stakes" accountability decisions. In some instances, the trustworthiness of these decisions is based on variance components and error variances derived through generalizability theory.…

Khuri's and Satterthwaite's methods of obtaining confidence intervals of variance components are compared. The article discusses that Khuri's method may be applied to obtain confidence intervals for the variance components and other linear functions of the expected mean squares used in generalizability theory. (Author/JAZ)

Presents solutions for the problem of maximizing the generalizability coefficient under a budget constraint. Shows that the Cauchy-Schwarz inequality can be applied to derive optimal continuous solutions for the number of conditions of each facet. Illustrates the formal similarity between optimization problems in survey sampling and…

Evidence is presented on the generalizability and convergent validity of performance assessments using data from six studies of student achievement that sampled a wide range of measurement facets and methods. Results at individual and school levels indicate that task-sampling variability is the major source of measurement error. (SLD)

The percentage of students at/above a cut point (PAAC) is one of the most common measures used for reporting school-level performance relative to a proficiency standard (L. Cronbach, N. Bradburn, and D. Horvitz, 1994). The two purposes of this study were to introduce procedures for estimating standard errors for school PAACs under a…

The New Standards Project conducted a pilot test of a series of performance-based assessment tasks in mathematics and English language arts at Grades 4 and 8 in the spring of 1993. This paper reports the results of a series of generalizability analyses conducted for a subset of the 1993 pilot study data in mathematics. Generalizability analyses…

An empirical investigation of methodological issues associated with evaluating treatment effect in single-subject research (SSR) designs is presented. This investigation: (1) conducted a generalizability (G) study to identify the sources of systematic and random measurement error (SRME); (2) used an analytic approach based on G theory to integrate…

In this paper, performance assessments are cast within a sampling framework. A performance assessment score is viewed as a sample of student performance drawn from a complex universe defined by a combination of all possible tasks, occasions, raters, and measurement methods. Using generalizability theory, the authors present evidence bearing on the…