Prior research has evaluated the reliability and validity of structured visual inspection (SVI) criteria for interpreting functional analysis (FA) outcomes (Hagopian et al., 1997; Roane et al., 2013). We adapted these criteria to meet the unique needs of interpreting latency-based FA outcomes and examined the reliability and validity of applying…

Creativity research crucially relies on creativity evaluations by external raters, but it is not clear what properties characterize good raters. In the present study, we investigated whether rater personality and rater creativity are related to discernment (i.e., the ability to distinguish creative from uncreative responses) when evaluating…

Using item-response theory to model rater effects provides an alternative solution for rater monitoring and diagnosis, compared to using standard performance metrics. In order to fit such models, the ratings data must be sufficiently connected in order to estimate rater effects. Due to popular rating designs used in large-scale testing scenarios,…

Maintaining the quality and comparability of qualifications has become a major priority for vocational education and training (VET) systems world-wide, especially for those countries which have decentralised assessment and reporting systems. In the absence of external examinations, one policy solution to align the assessment standards of different…

Australia has recently implemented Teaching Performance Assessments (TPAs) as a national accreditation requirement to assess final year preservice teachers' classroom readiness. In 2019, an Australian university developed a TPA to meet this requirement, comprising three written components and one oral component. This exploratory study investigated…

TESOL practitioners, especially in Asia, tend to believe that reliable assessment of students' L2 English speech can be done solely by L1 English native speakers with sufficient teaching and assessment experiences. Such a belief, however, may need to be reconsidered from a new perspective of "diversity and inclusivity." This study used…

Rater comments tend to be qualitatively analyzed to indicate raters' application of rating scales. This study applied natural language processing (NLP) techniques to quantify meaningful, behavioral information from a corpus of rater comments and triangulated that information with a many-facet Rasch measurement (MFRM) analysis of rater scores. The…

Human raters' assessment of interpreting is a complex process. Previous researchers have mainly relied on verbal reports to examine this process. To advance our understanding, we conducted an empirical study, collecting raters' eye-movement and retrospection data in a computerised interpreting assessment in which three groups of raters (n = 35)…

The current study examines the contribution of racial congruence on young children's speech and language production to determine if African American children produce more or different language with an African American assessor, as opposed to a White assessor. Participants for the current study were selected from a larger group of children…

Behavior analysts typically rely on visual inspection of single-case experimental designs to make treatment decisions. However, visual inspection is subjective, which has led to the development of supplemental objective methods such as the conservative dual-criteria method. To replicate and extend a study conducted by Wolfe et al. (2018) on the…

The purpose of this study was to examine the effects of focus of attention instructions (i.e. focus on the solo, accompaniment, or collaboration) on listeners' performance evaluations. Participants (N = 159) were inservice band directors who listened to and evaluated the accuracy and expressivity of four excerpts of Haydn's "Concerto for…

The European Research Council (ERC) receives many high-quality applications, but funds only a few. We analyze how members of ERC review panels assess applications in the first, highly competitive step of evaluations for ERC Starting and Consolidator Grants. Drawing on interviews with ERC panel members in different fields, we show that they adopt a…

This article reflects on my personal experience as a TLI reviewer. It draws upon a decade of learning with and from colleagues, and connects the lessons learned from being both a reviewer and producer of SoTL output. I signpost the challenges and opportunities that belie the role of a TLI reviewer and celebrate the success the role brings. Through…

The purpose of this study was to explore how evaluators justify using story-based methodologies when examining causality. The two primary research questions of the study included: 1) what arguments are made by evaluators to justify the credibility of story-based causal methodologies to evaluation stakeholders; and 2) from the perspective of…

There is presently little consensus on how evaluators evaluate special education teachers. The lack of agreement may be due to the training of evaluators. Special education teachers work under various conditions with a diverse population. In this dissertation, we reviewed the challenges associated with evaluating special education teachers,…