Sparse-rated data are common in operational performance-based language tests, as an inevitable result of assigning examinee responses to a fraction of available raters. The current study investigates the precision of two generalizability-theory methods (i.e., the rating method and the subdividing method) specifically designed to accommodate the…

Social scientists are frequently interested in assessing the qualities of social settings such as classrooms, schools, neighborhoods, or day care centers. The most common procedure requires observers to rate social interactions within these settings on multiple items and then to combine the item responses to obtain a summary measure of setting…

Critics of standardized testing have recommended replacing standardized tests with more authentic assessment measures, such as classroom assignments, projects, or portfolios rated by a panel of raters using common rubrics. Little research has examined the consistency of scores across multiple authentic assignments or the implications of this…

Students with significant disabilities must participate in large-scale assessments, often using an alternate assessment judged against alternate achievement standards. The development and administration of this type of assessment must necessarily balance meaningful participation with accurate measurement. In this study, generalizability theory is…

Significant momentum exists for next-generation assessments to increasingly utilize technology to develop and deliver performance-based assessments. Many traditional challenges with this assessment approach still apply, including psychometric concerns related to performance-based tasks (PBTs), which include low reliability, efficiency of…

Reviews relevant aspects of generalizability theory related to performance assessments and discusses the role of various facets in assessing the generalizability of performance assessments. Also considers some popular estimates of reliability for performance assessments from the perspective of generalizability theory. (SLD)

Introduces the articles of this theme issue focusing on performance assessment methodology. Papers address: (1) merging item formats; (2) scoring models; (3) equating and linking; (4) generalizability theory; (5) standard setting methods; and (6) validity issues and methods. (SLD)

Writing fits well within the realm of outcomes suitable for observation by performance assessments. Studies of the reliability of performance assessments have suggested that interrater reliability can be consistently high. Scoring consistency, however, is only one aspect of quality in decisions based on assessment results. Another is…

There is increasing pressure, both from institutions central to the national scientific mission and from regional and national accrediting agencies, on natural sciences faculty to move beyond course examinations as measures of student performance and to instead develop and use reliable and valid authentic assessment measures for both individual…

It is proposed that criteria for evaluating the quality of performance should be defined, at least tentatively, during the initial design of a performance assessment. Six characteristics of sound criteria are (1) significance; (2) fidelity; (3) generalizability; (4) developmental appropriateness; (5) accessibility; and (6) utility. (SLD)

This annotated bibliography contains nine items addressing assessment methodology. The titles are: "Performance and Product Evaluation"; "The Critical Incident Technique"; "Constructing Achievement Tests"; "Applying the Assessment Center Method"; "Performance Assessment in Education and Training:…

Increasing emphasis on assessment and concern about assessment techniques have stirred interest in alternative assessment forms, for which evidence is needed about consequences, transfer of performance on specific assessment tests, and assessment fairness. Criteria concerning consequences, fairness, transfer-generalizability, cognitive complexity,…

Industrial and organizational psychologists for the Department of Defense have been working for the past 10 years to develop high fidelity measures of job performance for use in validating job selection procedures and standards. Information on developing and scoring performance exercises in the Job Performance Measurement (JPM) Project is…