Used generalizability theory to assess the variation in school scores across very short test forms that measured mathematics scores in grades 4 and 8. More than 25,000 students took each form of the 3 tests for each grade. Results demonstrate the lack of comparability in school scores across short, nonparallel tests forms and the importance of…

Generalizability (G) theory was used with a sample of 37 third-grade students to assess the variability in words correct per minute (WCPM) scores caused by student skill and passage variability. Reliability-like coefficients and the SEM based on a specific number of assessments using different combinations of passages demonstrated how manipulating…

In this article, the author echoes his co-author's and colleague's pleasure (Hill, this issue) at the thoughtfulness and far-ranging nature of the comments to their initial attempts at test validation for the mathematical knowledge for teaching (MKT) measures using the validity argument approach. Because of the large number of commentaries they…

We contend that generalizability (G) theory allows the design of psychometric approaches to testing English-language learners (ELLs) that are consistent with current thinking in linguistics. We used G theory to estimate the amount of measurement error due to code (language or dialect). Fourth- and fifth-grade ELLs, native speakers of…

Generalizability theory is used to examine the sources of variability present in a teacher and course evaluation instrument. Two studies were conducted. In the first study, four different forms commonly used by one specific college of a large midwestern university were examined using responses of 915 students. The analysis of variance performed on…

Generalizability theory (g-theory) was used, as an alternative to classical test theory, to evaluate measurement error in a behaviorally anchored role-play measure, highlighting the usefulness of this theory in instrument development. G-theory partitions an observed score into the universe score and error scores associated with separate sources of…

Increasing social awareness of the need for accurate and fair testing has combined with technological developments to produce dramatic advances in psychometric models, methods, and procedures. Three of the most important new directions within the measurement field concern the following areas: (1) item response theory; (2) item bias or differential…

Three strategies for augmenting the interpretation of significance test results are illustrated. Determining the most suitable indices to use in evaluating empirical results is a matter of considerable debate among researchers. Researchers increasingly recognize that significance tests are very limited in their potential to inform the…

This study used generalizability theory and other statistical concepts to assess the application of the Angoff method to setting cutoff scores on two professional certification tests. A panel of ten judges gave pre- and post-feedback Angoff probability ratings of items of two forms of a professional certification test, and another panel of nine…

This document describes the development and assessment of a methodology for generating on-the-job-training (OJT) task proficiency assessment instruments. The Task Evaluation Form (TEF) development procedures were derived to address previously identified deficiencies in the evaluation of OJT task proficiency. The TEF development procedures allow…

Methods for examining the viability of assumptions underlying generalized linear models are considered. By appealing to the likelihood, a natural generalization of the raw residual plot for normal theory models is derived and is applied to investigating potential misspecification of the linear predictor. A smooth version of the plot is also…

Generalizability theory, which subsumes classical measurement theory as a special case, provides a general model for estimating the reliability of observational rating data by estimating the variance components of the measurement design. Research data from the "Heart Smart" health intervention program were analyzed as a heuristic tool.…

Examined the extent to which tutor ratings remained stable in the long term by evaluating 291 ratings of 140 tutors at Maastricht University in the Netherlands between 1992 and 1995. The results indicated that, if the aggregated score and overall judgement are used to interpret the precision of individual scores, four and two occasions,…

Objective criteria for evaluating the Eysenck Personality Inventory's internal structure are discussed. An approach based on targeted rotations and the test's scoring key is proposed as a means of providing common criteria. Data from earlier structure and test results for 195 undergraduates support the utility of 3 criteria developed. (SLD)

Oral retellings are strongly recommended as a way to measure reading comprehension for second language learners (Bernhardt, 1985, 1990, 1991). However, the reliability of such ratings is a matter of concern for a variety of reasons (Aiken, 1996; Cooper, 1981; Saal, Downey, & Lahey, 1980). The purpose of this study was to establish reliable rating…