Researchers have documented the impact of rater effects, or raters' tendencies to give different ratings than would be expected given examinee achievement levels, in performance assessments. However, the degree to which rater effects influence person fit, or the reasonableness of test-takers' achievement estimates given their response patterns,…

In this paper we assert that the practice of principled assessment design renders traditional standard-setting methodology redundant at best and contradictory at worst. We describe the rationale for, and methodological details of, Embedded Standard Setting (ESS; previously, Engineered Cut Scores. Lewis, 2016), an approach to establish performance…

Many large-scale performance assessments include score resolution procedures for resolving discrepancies in rater judgments. The goal of score resolution is conceptually similar to person fit analyses: To identify students for whom observed scores may not accurately reflect their achievement. Previously, researchers have observed that…

The purpose of this study was to test methods that strengthen the comparability claims about annual determinations of student proficiency in English language arts, math, and science (Grades 3-12) in the New Hampshire Performance Assessment of Competency Education (NH PACE) pilot project. First, we examined the literature in order to define…

Discusses extending research into differential item functioning (DIF) to performance assessment and considers some of the best options available at this time. The most effective strategy for assessing DIF is to use a combination of several methods, preferably one from each of the observed score nonparametric, latent score nonparametric, and…

This instructional module presents and illustrates rules for designing performance assessments in a step-by-step strategy through: (1) reason for the assessment; (2) type of performance evaluated; (3) exercises to elicit performance; and (4) systematic rating procedures. Guidelines for maximizing reliability, validity, and economy are presented.…

A quiz is presented that may call the attention of school personnel to regression effects in educational measurement, and clarify the meaning of statistical regression for educators in schools. Ten questions and their answers explore regression in the contexts of classroom testing, grading, and performance assessment. (SLD)