The number of performance assessments continues to increase around the world, and it is important to explore new methods for evaluating the quality of ratings obtained from raters. This study describes an unfolding model for examining rater accuracy. Accuracy is defined as the difference between observed and expert ratings. Dichotomous accuracy…

Responding to an Illinois law that took effect in 2010, the Performance Evaluation Reform Act (PERA), Chicago Public Schools (CPS) created and implemented the REACH (Recognizing Educators Advancing Chicago's Students) educator evaluation and support system. REACH was piloted in 2012-13. It is still being implemented as of the 2019-20 school year,…

This dissertation is comprised of three papers that propose and apply psychometric models to deal with complexities and challenges in large-scale assessments, focusing on modeling rater effects and complex learning progressions. In particular, three papers investigate extensions and applications of multilevel and multidimensional item response…

Prior research indicates that the overall reliability of performance ratings can be improved by using ordinary least squares (OLS) regression to adjust for rater effects. The present investigation extends previous work by evaluating the impact of OLS adjustment on standard errors of measurement ("SEM") at specific score levels. In…

This research study was designed to inform the development of a new teacher assessment model. The goal of the Teacher Success Model (TSM) initiative is to develop a systematic, evidence-informed statewide model for teacher assessment. While the initiative included all teachers, support professionals, and evaluators, this study focused on induction…

There has been a recent groundswell of support in the American Evaluation Association's Independent Consulting Topical Interest Group (IC TIG) for evaluating evaluators' work just as evaluators evaluate the work of their clients. To facilitate this self-evaluation, the IC TIG elected to create a peer review process that focuses on written…

This paper examines whether individuals who train themselves to score a performance assessment will rate acceptably when compared to known standards. Research on the efficacy of rater self-training materials developed by the Center for Applied Linguistics for the Texas Oral Proficiency Test (TOPT) is examined. Rater self-materials are described…

This study illustrates the use of three least-squares models to control for rater effects in performance evaluation: (1) ordinary least squares (OLS); (2) weighted least squares (WLS); and (3) OLS subsequent to applying a logistic transformation to observed ratings (LOG-OLS). The three models were applied to ratings obtained from four…