This study demonstrates the advantages of using a constrained optimization algorithm to explore the optimal number of prompts, modes of discourse, and raters for achieving an acceptable level of reliability during a direct writing assessment. Writing samples elicited from 50 college students were rated by 3 graduate students and the scores…

The application of mathematical programming techniques is extended to the construction of measurement instruments in generalizability theory. Key concepts in generalizability theory are explained and a description is given of: (1) the one-facet crossed design; (2) the two-facet crossed design; and (3) a two-facet nested design. The optimization of…

Although student evaluation data provide a reasonable basis for making decisions about instructors when generalizing across courses and students, when the course is the object of measurement (OM), data are less generalizable. This finding may be due to the type of evaluation items used or to academic discipline differences in the type of courses…

As schools move toward performance assessment, there is increasing discussion of using these assessments for accountability purposes. When used for making decisions, performance assessments must meet high standards of validity and reliability. One major source of unreliability in performance assessments is interrater disagreement. In this paper,…

The purpose of this study was to investigate the impact of local item dependence (LID) in passage-based testlets on the test score reliability of an English as a Foreign Language (EFL) reading comprehension test from the perspective of generalizability (G) theory. Definitions and causes of LID in passage-based testlets are reviewed within the…

This study examined the generalizability and dependability of a performance-based assessment in algebra. Four forms of a five-item test were constructed using different subsets of eight items based on attributes from task analysis. Subjects included 142 "algebra 2" students from 2 high schools in the Midwestern United States and 148 11th…

Results of a study of the reliability of student ratings of academic advising from 15 postsecondary institutions show that generalizability theory is an appropriate and accurate way to estimate reliability. Advantages and disadvantages of the generalizability theory approach in comparison with coefficient alpha are discussed. (SLD)

This commentary focuses on the application of D. Rogosa and G. Ghandour's work to observational research on classroom processes. Rogosa and Ghandour have shown that the short length of an observation is typically the dominant source of error. Investigators should conduct observations for as long as possible. (SLD)

This study tests the generalizability of the factor pattern, structural parameters, and latent mean structure of a multidimensional, hierarchical model of physical self-concept in adolescents across gender and grade. A children's version of the Physical Self-Perception Profile (C-PSPP) was administered to seventh-, eighth- and ninth-grade high…

Although the rubric has emerged as one of the most popular assessment tools in progressive educational programs, there is an unfortunate dearth of information in the literature quantifying the actual effectiveness of the rubric as an assessment tool "in the hands of the students." This study focuses on the validity and reliability of the rubric as…

A total of 2,000 essays written by 1,000 students was submitted to generalizability analyses for domain-referenced tests. Each student had written one essay on each of two prompts representing two models of discourse. Each essay was read by six readers and judged on a scale of from 1 to 4. No reader read essays from both prompts. Reader agreement…

Two new indices of stability of content-referenced standard-setting results are presented, relating variability of judges' decisions to the variability of candidate scores and to the reliability of the test. These indices are used to indicate whether scores resulting from a standard-setting study are of sufficient precision. (Author/LMO)

An investigation of the Conners' Teacher Rating Scale-Revised hyperactivity scale found that the referents for teacher ratings should be determined, teachers' ratings should be made more objective, standardization across teachers should be demonstrated before norms are preferred, and the rating scale should be validated via observations or other…

In behavior research using cognitive and affective measures, there is often incongruity between the statistical analysis employed and the intended inference. This paper argues that incorporating items as levels of a random facet via generalizability theory allows the statistical examination of the inferential question in the desired universe of…

This study examines the score dependability of writing and speaking assessments from the Test of English as a Foreign Language (TOEFL) from the perspectives of univariate and multivariate generalizability theory (G-theory) and presents the findings of three separate G-theory studies. For writing, the focus was on evaluating the impact on…