The purpose of this study was to investigate Access Arrangements (AA) which allow students with specific learning difficulties (SpLDs), disabilities or medical needs to access assessments by making "reasonable" adjustments, as established by the Equality Act 2010. A review of the literature revealed some problematic issues relating to…

Rater effects, or raters' tendencies to assign ratings to performances that are different from the ratings that the performances warranted, are well documented in rater-mediated assessments across a variety of disciplines. In many real-data studies of rater effects, researchers have reported that raters exhibit more than one effect, such as a…

The performance decision tree (PDT; Fulcher et al., 2011) is a rubric style that is applicable to performance assessment, with origins in Upshur and Turner's (1995) empirically derived binary-choice, boundary-definition (EBB) scale. It is easier for raters to assess performance by evaluating multiple binary-choice descriptors. Additionally,…

Even when raters demonstrate agreement in the use of a measure, limited score variability or violation of often-ignored statistical assumptions can result in lower reliability estimates than intuitively expected. This article uses data drawn from two randomized controlled trials of schema therapy and cognitive behavioral therapy for the treatment…

The relationship between "stability" and "change" is a central paradox of administration that pervades all forms of organizing. Evaluation is not unfamiliar with paradoxical objectives and roles, which can result in tensions for evaluators and stakeholders. In this article, paradoxes between stability and change in the…

Given the ever-evolving role of school counselors and increasing demands placed on them in today's sociopolitical climate, proficiency in program evaluation is more important than ever as school counselors advocate for their role and for comprehensive school counseling program delivery in schools. This article presents a reexamination of the…

Recurring forms of abuse like domestic violence are unfortunately common. When an individual makes an allegation about their experience, however, there is rarely additional evidence to corroborate their claim. The veracity of the allegation is thus likely to be a central concern in subsequent proceedings. This experiment explored evaluator's…

We evaluate the feasibility of developing predictive models of rater behavior, that is, "rater-specific" models for predicting the scores produced by a rater under operational conditions. In the present study, the dependent variable is the score assigned to essays by a rater, and the predictors are linguistic attributes of the essays…

When a fingerprint is located at a crime scene, a human examiner is counted upon to manually compare this print to those stored in a database. Several experiments have now shown that these professional analysts are highly accurate, but not infallible, much like other fields that involve high-stakes decision-making. One method to offset mistakes in…

Synthesis writing is widely taught across domains and serves as an important means of assessing writing ability, text comprehension, and content learning. Synthesis writing differs from other types of writing in terms of both cognitive and task demands because it requires writers to integrate information across source materials. However, little is…

This study employed a mixed-methods approach to examine how rater performance develops during a semester-long rater certification program for an English as a Second Language (ESL) writing placement test at a large US university. From 2016 to 2018, we tracked three groups of novice raters (n = 30) across four rounds in the certification program.…

We investigated the relationship between the scores assigned by an Automated Essay Scoring (AES) system, the Intelligent Essay Assessor (IEA), and grades allocated by trained, professional human raters to English essay writing by instigating two procedures novel to written-language assessment: the logistic transformation of AES raw scores into…

This paper extends the use of Generalizability Theory to the measurement of extemporaneous L2 speech through the lens of speech perception. Using six datasets of previous studies, it reports on "G studies"--a method of breaking down measurement variance--and "D studies"--a predictive study of the impact on reliability when…

According to expectation violation theory, job applicants can be upgraded or downgraded during an interview when their accent does not match employers' speech expectations. Focusing on the employment of second language French job candidates in Québec, this study explored this issue dynamically in terms of how expectations may impact the trajectory…

In this study we show that the perception of lateral variants by Puerto Rican listeners changes according to who the listener believes is speaking. Puerto Rican listeners heard sentences with target words featuring either rhotic [voiced alveolar tap or flap] or lateral [l] (amo[voiced alveolar tap or flap] -- amo[l]) codas, a sociophonetic…