Scoring procedures for many rater-mediated performance assessments include score resolution procedures in which a third rater adjudicates discrepancies between two raters' ratings of the same performance. There are numerous approaches for calculating resolved scores that involve different combinations of the original and third ratings. Using data…

The growing recognition of "how much" principals matter for student learning and "how" they make a difference has fueled the need to ensure that effective principals are leading every school. One way to achieve this is through principal evaluation, which has experienced significant changes in the last decade. We conducted a…

An Angoff standard setting study generally yields judgments on a number of items by a number of judges (who may or may not be nested in panels). Variability associated with judges (and possibly panels) contributes error to the resulting cut score. The variability associated with items plays a more complicated role. To the extent that the mean item…

Research ethics reviews are an integral component of the research process in education, but little is known about educators' perceptions of and experiences with this process. This exploratory study examines educators' perceptions of and experiences with the reviews of research ethics and, in this way, it contributes to a better understanding of…

When conducting human subjects research, social and behavioral researchers seeking to study current issues involving immigrants, refugees, and undocumented students must submit their research to an institutional review board (IRB). Research applications proposing to enroll these populations lie outside the scope of vulnerable populations named in…

The purpose of this study was to determine the male-female ratio of band directors serving on adjudication panels for Texas University Interscholastic League (UIL) concert and sight reading evaluations from 2010-2019. The Texas Music Forms database was used to access a list of UIL band concert and sight reading adjudicators from across the state…

In many occupations there has been a trend towards requiring an entry-level qualification where a qualification was previously not required for the occupation. For various other occupations, the type of qualification needed has changed to one of a higher level. Does this so-called 'credential creep' impact on the outcomes for individuals who…

The study established the validity and reliability of a weighted individual performance-based assessment tool within the utility scope of middle school orchestral strings. The following research questions guided this study: 1. What specific string-playing behaviors and corresponding criteria validate a weighted individual performance-based…

The impact school leaders have on student achievement is prominent in the national conversation regarding educational reform. Perhaps, one of the most highly debated topics is how to assess their impact, and recent legislation tasked every state with determining how to evaluate principal effectiveness. Any new or customized evaluation tool…

Reviewer comments in research articles such as journal papers or dissertations guide students during the revision process to improve the quality of their articles. Our goal is to make the comments more meaningful to the students' revision process. Revision involves implicit cognitive processes and ICT has the potential to make such processes…

Researchers have explored a variety of topics related to identifying and distinguishing among specific types of rater effects, as well as the implications of different types of incomplete data collection designs for rater-mediated assessments. In this study, we used simulated data to examine the sensitivity of latent trait model indicators of…

Sign languages present particular challenges to language assessors in relation to variation in signs, weakly defined citation forms, and a general lack of standard-setting work even in long-established measures of productive sign proficiency. The present article addresses and explores these issues via a mixed-methods study of a human-rated…

Researchers frequently evaluate rater judgments in performance assessments for evidence of differential rater functioning (DRF), which occurs when rater severity is systematically related to construct-irrelevant student characteristics after controlling for student achievement levels. However, researchers have observed that methods for detecting…

Both within the field of world language (WL) teacher education and across teacher education in other disciplines, critiques of the edTPA have increased over the past several years. In WL language education, scholars have identified issues related to raters' use of edTPA rubrics and a serious lack of transparency about rater expertise. To better…

Differential rater severity (DRS), one prevalent case of differential rater functioning (aka rater bias or rater interaction) effects, manifests itself when a rater assigns unusually severe or lenient ratings, threatening the validity and fairness of rater-mediated assessment. Building on a many-facets Rasch measurement (MFRM) approach, this study…