Rater comparison analysis is commonly necessary in the social sciences. Conventional approaches to the problem generally focus on calculation of agreement statistics, which provide useful but incomplete information about rater agreement. Importantly, one-number agreement statistics give no indication regarding the nature of disagreements, nor do…

Cognitive diagnostic models (CDMs) provide pedagogically relevant information in the form of a student profile of multiple binary categorizations of students into mastery or nonmastery statuses on latent traits called attributes. Federal educational accountability requires accountability measures to designate students into one of at least three…

Fitting wrongly specified models to observed data may lead to invalid inferences about the model parameters of interest. The current study investigated the performance of the posterior predictive model checking (PPMC) approach in detecting model-data misfit of the hierarchical rater model (HRM). The HRM is a rater-mediated model that incorporates…

For decades, evaluators have noted that it is difficult for stakeholders to accept negative evaluation results (Carter, 1971; Taut & Brauns, 2003). There is a need for additional research on evaluation to better understand when and why stakeholders reject negative or critical evaluation findings. Drawing on social identity theory (SIT), the…

The Performance Assessment and Evaluation System (PAES) is used by all major universities in the state of Utah to measure the effective teaching skills of preservice candidates as they progress through their teaching preparation program. The resulting ratings are used to make high-stakes decisions relating to course completion as well as…

The educative Teacher Performance Assessment (edTPA) has been adopted by many state legislatures and teacher preparation programs (TPP). These states require teacher candidates to pass the edTPA with a state-specific passing score to be recommended for licensure. In the 19 states where passing the edTPA has not been required as a condition of…

Rater-mediated activities in educational research occur when an expert judge or rater utilizes an instrument to judge persons or items and generates scale scores. Scale scores are from a subjective judgment and must undergo a quality control measure called rating quality. Rating quality in this study is broadly defined as the extent to which…

Grades serve as one of the primary indicators of student learning, directing subsequent actions for students, instructors, and administrators, alike. Therefore, grade validity--that is, the extent to which grades communicate a meaningful and credible representation of what they purport to measure--is of utmost importance. However, a grade cannot…

The Resident Assistant (RA) position is the foundational role in the student affairs staffing structure. RA job description has expanded exponentially since the earliest iterations (Boone et al., 2016). The selection of RAs presents unique challenges because it is costly, requires a lot of staff, typically draws a large candidate pool, and…

The process of conducting child and adolescent psychoeducational assessments has changed over the past 2 decades (Shapiro & Heick, 2004). In the past, the school psychologists commonly concentrated on behavioral, achievement, and projective assessments and usually did not include systematic multi-rater observation rating scales of behavior.…

Many methods are used to measure interrater reliability for studies where each target receives ratings by a different set of judges. The purpose of this study is to explore the use of hierarchical modeling for estimating interrater reliability using the intraclass correlation coefficient. This study provides a description of how the ICC can be…

Teacher evaluations are routinely conducted across the United States for licensure and professional development supports. However, there is limited research on the interrater reliability of these evaluation assessment systems, despite federal recommendations (Graham et al., 2012). This research explores the systematic approach to interrater…

Assessment and evaluation are at the heart of all functioning systems, as progress can be determined with the information collected by these assessments. The problem addressed in this study is the lack of actionable data to evaluate whether assessments aligned with Montessori benchmarks support the achievement of student outcomes. To address the…

The shortage of autism specialists and lack of culturally sensitive autism assessment tools are helping to perpetuate racial and ethnic disparities in autism identification and treatment. Using DisCrit as a framework, this quantitative study examined the utility of one autism assessment tool, the Social Responsiveness Scale, second edition (SRS-2)…

A mixed methods action research study was designed to answer three research questions based on inter-rater reliability (IRR) in compliance calls for transition at a state education agency, perceived confidence levels in making and discussing compliance calls, and perceived confidence in sharing transition resources. An innovation based on…