Evaluation of music performance in competitive contexts often produces discrepancies between the expert judges. These discrepancies can be reduced by using appropriate rubrics that minimise the differences between judges. The objective of this study was the design and validation of an analytical evaluation rubric, which would allow the most…

In the context of examinations, the phrase "maintaining standards" usually refers to any activity designed to ensure that it is no easier (or harder) to achieve a given grade in one year than in another. Specifically, it tends to mean activities associated with setting examination grade boundaries. Benton et al (2020) describes a method…

Recently, comparative judgement has been introduced as an alternative method for scoring essays. Although this method is promising in terms of obtaining reliable scores, empirical evidence concerning its validity is lacking. The current study examines implications resulting from two critical assumptions underpinning the use of comparative…

While portfolios are increasingly used to assess competence, the validity of such portfolio-based assessments has hitherto remained unconfirmed. The purpose of the present research is therefore to further our understanding of how assessors form judgments when interpreting the complex data included in a competency-based portfolio. Eighteen…

The Consensual Assessment Technique (CAT) is a common creativity assessment. According to this technique, the best judges of creativity are qualified experts. Yet what does it mean to be an expert in a domain? What level of expertise is needed to rate creativity? This article reviews the literature on novice, expert, and quasi-expert creativity…

Standard setting (the process of establishing minimum passing scores on high-stakes exams) is a highly evaluative and policy-driven process. It is a common belief that standard setting panels should be diverse and representative. There is concern, however, that panelists with varying characteristics may differentially influence the results of the…

Most professors have mixed feelings about participating on peer-review panels. It's an honor. It helps the discipline. It's a waste of time. It's biased. Michele Lamont wanted to know whether it works: specifically, whether, and how, professors identify excellence. So the multi-titled Harvard University scholar--professor of European studies,…

Purpose: The purposes of this study were (a) to determine whether highly experienced clinicians and researchers agreed with each other in judging the presence or absence of stuttering in the speech of children who stutter and (b) to determine how those binary stuttered/nonstuttered judgments related to categorizations of the same speech based on…

The purpose of this study was to determine whether viewing a musical score while listening (as opposed to not viewing the score) would affect musicians' ratings of choral performance excerpts. University musicians (N = 240) listened to four excerpts of choral music (from Vivaldi's "Gloria") and rated them on a 10-point Likert-type scale for…

In education, standards have to be interpreted, for planning of teaching, for development of assessments and for alignment analysis. In most cases, it is important that there is an agreement between individuals and organizations about how to interpret standards. However, there is a lack of studies of how consistent different group of judges are…

This article presents a chronology of three empirical studies that outline the measurement process by which two new subscales ("Doubts about Actions" and "Organization") were developed and integrated into a revised version of Dunn, Causgrove Dunn, and Syrotuik's (2002) "Sport Multidimensional Perfectionism Scale"…

The consistency of assessment demands is important to validity. This research investigated the comparability of the demands of college-assessed units within a vocationally related qualification, drawing on methodological approaches that have previously been used to compare assessments. Assessment materials from five colleges were obtained. After…

Absolute standard setting procedures are recommended for assessment in medical education. Absolute, test-centred standard setting procedures were introduced for written assessments in the Liverpool MBChB in 2001. The modified Angoff and Ebel methods have been used for short answer question-based and extended matching question-based papers,…

Describes a study of judge leniency and consistency using a Rasch approach and involving 4,683 candidates and 53 judges. (SLD)

Proposes a generalization of Cohen's kappa coefficient (J. Cohen, 1960) to address the problem of accounting for overall chance-corrected interobserver agreement among the multivariate ratings of several judges. The statistic's metric is conventional and in the univariate case it is equivalent to existing extensions of the kappa coefficient to…