With a lack of consensus of what evaluation is within the field of evaluation, there is a difficulty in communicating to nonevaluators what evaluation is and how evaluation differs from research. To understand how evaluation is defined, both evaluators and researchers were asked how they defined evaluation and, if at all, differentiated evaluation…

Background: Secondary school teachers sometimes teach and assess material outside their specialisms for reasons including staff shortages or the growing popularity of the interdisciplinary courses. We hypothesised that teacher-assessors with different subject specialisms may differ in their interpretations of frequently used words in teaching and…

Efficient constructed response (CR) scoring requires both accuracy and speed from human raters. This study was designed to determine if setting scoring rate expectations would encourage raters to score at a faster pace, and if so, if there would be differential effects on scoring accuracy for raters who score at different rates. Three rater groups…

This experimental study explores how source use features impact raters' judgment of argumentation in a second language (L2) integrated writing test. One hundred four experienced and novice raters were recruited to complete a rating task that simulated the scoring assignment of a local English Placement Test (EPT). Sixty written responses were…

Innovation capability has become a necessary requirement for qualified teachers in the context of informatization. However, the validity and objectivity of existing assessments are unclear. Therefore, this study selected nine researchers to evaluate the instructional innovation capabilities of 60 pre-service teachers from a Chinese normal…

Assessment materials must be checked for errors before they are presented to candidates. Any errors have the potential to reduce validity. For example, in the most extreme cases, an error may turn an otherwise well-designed exam question into one that is impossible to answer. In Cambridge University Press & Assessment, assessment materials are…

Prior research has evaluated the reliability and validity of structured visual inspection (SVI) criteria for interpreting functional analysis (FA) outcomes (Hagopian et al., 1997; Roane et al., 2013). We adapted these criteria to meet the unique needs of interpreting latency-based FA outcomes and examined the reliability and validity of applying…

Creativity research crucially relies on creativity evaluations by external raters, but it is not clear what properties characterize good raters. In the present study, we investigated whether rater personality and rater creativity are related to discernment (i.e., the ability to distinguish creative from uncreative responses) when evaluating…

The purpose of this quantitative correlational study was to determine if, and to what extent, a relationship existed between teachers' perceptions of their evaluator's instructional leadership and the teachers' evaluation scores at the middle and high school levels in one Florida school district. Ozcan's theory of teacher motivation states that…

In an era of test-based accountability, school inspections can offer a more nuanced understanding of why schools fail. Yet, we have limited knowledge of how inspectors arrive at their decisions on school quality. Analyzing inspectors' decision-making can reveal the underlying views regarding school accountability and open opportunities for school…

Using item-response theory to model rater effects provides an alternative solution for rater monitoring and diagnosis, compared to using standard performance metrics. In order to fit such models, the ratings data must be sufficiently connected in order to estimate rater effects. Due to popular rating designs used in large-scale testing scenarios,…

Maintaining the quality and comparability of qualifications has become a major priority for vocational education and training (VET) systems world-wide, especially for those countries which have decentralised assessment and reporting systems. In the absence of external examinations, one policy solution to align the assessment standards of different…

Australia has recently implemented Teaching Performance Assessments (TPAs) as a national accreditation requirement to assess final year preservice teachers' classroom readiness. In 2019, an Australian university developed a TPA to meet this requirement, comprising three written components and one oral component. This exploratory study investigated…

TESOL practitioners, especially in Asia, tend to believe that reliable assessment of students' L2 English speech can be done solely by L1 English native speakers with sufficient teaching and assessment experiences. Such a belief, however, may need to be reconsidered from a new perspective of "diversity and inclusivity." This study used…

Rater comments tend to be qualitatively analyzed to indicate raters' application of rating scales. This study applied natural language processing (NLP) techniques to quantify meaningful, behavioral information from a corpus of rater comments and triangulated that information with a many-facet Rasch measurement (MFRM) analysis of rater scores. The…