Comparative Judgement (CJ) is an increasingly widely investigated method in assessment for creating a scale, for example of the quality of essays. One area that has attracted attention in CJ studies is the optimisation of the selection of pairs of objects for judgement. One approach is known as adaptive comparative judgement (ACJ). It has been…

In this study we explored the potential of adaptive comparative judgement (ACJ) as a medium for peer assessment and for the giving and receiving of peer feedback. ACJ is a marking protocol in which the assessor (or judge) merely compares two answers and chooses a winner. Repeated judgements and a suitable sorting algorithm allow marks to be…

When collecting data, science education researchers frequently have multiple respondents evaluate multiple artifacts using multiple criteria. Herein, the authors introduce Multifaceted Rasch Measurement (MFRM) analysis and explain why MFRM must be used when "judges'" data are collected. The authors use data from elementary science…

We investigated the use of adaptive comparative judgment to evaluate the middle school student learning, engagement, and experience with the design process in an open-ended problem assigned in a technology and engineering education course. Our results indicate that the adaptive comparative judgment tool effectively facilitated the grading of the…

The paper proposes a recommender system approach to cover online judge's domains. Online judges are e-learning tools that support the automatic evaluation of programming tasks done by individual users, and for this reason they are usually used for training students in programming contest and for supporting basic programming teachings. The…

The Consensual Assessment Technique (CAT) is a common creativity assessment. According to this technique, the best judges of creativity are qualified experts. Yet what does it mean to be an expert in a domain? What level of expertise is needed to rate creativity? This article reviews the literature on novice, expert, and quasi-expert creativity…

Social scientists collect vital information that bears on issues of education policy. When the courts are faced with an opportunity to make a decision that shapes education, judges need access to high-quality research, but they must also be convinced that it can be useful in their decision making. This article approaches the question of how social…

The consistency of assessment demands is important to validity. This research investigated the comparability of the demands of college-assessed units within a vocationally related qualification, drawing on methodological approaches that have previously been used to compare assessments. Assessment materials from five colleges were obtained. After…

The Consensual Assessment Technique (CAT), developed by Amabile [Amabile, T.M. (1982). "Social psychology of creativity: A consensual assessment technique." "Journal of Personality and Social Psychology," 43, 997-1013], is frequently used to evaluate the creativity of productions. Judgments obtained with CAT are usually reliable and valid.…

Absolute standard setting procedures are recommended for assessment in medical education. Absolute, test-centred standard setting procedures were introduced for written assessments in the Liverpool MBChB in 2001. The modified Angoff and Ebel methods have been used for short answer question-based and extended matching question-based papers,…

The manuscript explores gender stereotyping in performance evaluations in undergraduate mock trial competitions. The central hypothesis is that gendered expectations for attorney and witness behavior in the legal community inform and influence the evaluation of undergraduate performance in relation to communication methods, argumentation…

Logical estimates of item difficulty made by judges were compared to empirical estimates derived from a test administration. Results indicated substantial correspondence between logical and empirical estimates, and substantial variation among judges. Further, the more elaborate the system used by judges to make estimates, the more accurate the…

The validity of two subjective approaches to judging in synchronized swimming were examined through a multitrait-multimethod matrix. Results indicated that judging panels tended not to differentiate between execution and content scores. (Author/JKS)

A coefficient of unfairness in the allocation of goods to people can be extended to measuring consensus among judges. The notion of relative variability underlies the formation of these measures.

Central to the standards-based assessment validation process is an examination of the alignment between state standards and test items. Several alignment analysis systems have emerged recently, but most rely on either traditional rating or matching techniques. Little, if any, analyses have been reported on the degree of consistency between the two…