Math curriculum-based measurement (CBM) is an essential tool for multi-tiered systems of support decision making, but the reliability of math CBMs has received little research, particularly using more rigorous methods such as generalizability (G) theory. Math CBM is historically organized into two domains: mastery measures and general outcome…

In this technical report, the Center for Research Use in Education (CRUE) presents the methodological design of a large-scale quantitative investigation of research use by school-based practitioners through the "Survey of Evidence in Education for Schools (SEE-S)." It documents the major technical aspects of the development of SEE-S,…

To meet recent accountability mandates, school districts are implementing assessment frameworks to document teachers' effectiveness. Observational assessments play a key role in this process, albeit without compelling evidence of their psychometric rigor. Using a sample of kindergarten teachers, we employed Generalizability theory to investigate…

Rater and occasion generalizability (G) coefficients were derived by an intraclass correlation method for 11 scales and two total scores on an open-ended questionnaire, The Occupational Information Inventory, which measures vocational decision making. (Author)

Generalizability theory is used to show that the assumption that reliability for groups is greater than that for persons (and that error variance for groups is less than that for persons) is not necessarily true. Examples are provided from course evaluation and performance test literature. (SLD)

Most training programs in education and psychology focus on classical test theory techniques for assessing score dependability. This paper discusses generalizability theory and explores its concepts using a small heuristic data set. Generalizability theory subsumes and extends classical test score theory. It is able to estimate the magnitude of…

The application of generalizability theory to the reliability and error variance estimation for performance assessment scores is discussed. Decision makers concerned with performance assessment need to realize the restrictions that limit generalizability such as limitations that lead to reductions in the number of tasks possible, rater quality,…

Increased demands for colleges and universities to engage in outcomes assessment for accountability purposes have accelerated the need to bridge the gap between higher education practice and the fields of measurement, assessment, and evaluation. The "Handbook on Measurement, Assessment, and Evaluation in Higher Education" provides higher…

Although portfolio assessment is becoming increasingly popular, it may not survive unless portfolio scoring can meet the demands of large-scale assessment standards. The results of studies of interrater reliability with large-scale portfolio assessments have been mixed. This paper reports the scoring results of a nationwide portfolio pilot in…

This paper assesses the credibility of a single total instrument score and various logical sub-scores derived from a series of summative judgments about the quality of teaching performance. The objectives were to compare the generalizability of alternative Teacher Performance Assessment Instrument (TPAI) scores, to compare the dependability of…