The aim of this study was to examine how people mentally represent and depict true and false statements about claimed future actions--so-called true and false intentions. On the basis of construal level theory, which proposes that subjectively unlikely events are more abstractly represented than likely ones, we hypothesized that false intentions…

As theory-based evaluation (TBE) engages in situations where multiple stakeholders help develop complex program theory about dynamic phenomena in politically contested settings, it becomes difficult to develop and use program theory without ambiguity. The purpose of this article is to explore ambiguity as a fruitful perspective that helps TBE face…

This research aimed to compare the G and Phi coefficients estimated in Decision studies in Generalizability theory and obtained in actual cases for the same conditions of similar facets by using crossed design. The research was conducted as pure research on 120 individuals (students), six items and 12 raters. An achievement test composed of six…

This study investigated the hypothesis that cognitive performance of students with physical disabilities may be influenced by the evaluators' identity. Students with or without a physical disability completed a logic test and were informed that they would be evaluated by students from their own group (ingroup condition) or from an other group…

As global discussions of evaluation standards become more contextually nuanced, culturally responsive conceptions of ethics have not been sufficiently discussed. In academic social research, ethical clearance processes have been designed to protect vulnerable people from harm related to participation in a research project. This article expands the…

To score an essay response, raters draw on previously trained skills and knowledge about the underlying rubric and score criterion. Cognitive processes such as remembering, forgetting, and skill decay likely influence rater performance. To investigate how forgetting influences scoring, we evaluated raters' scoring accuracy on TOEFL and GRE essays.…

With the wide spread large-scale e-learning environments such as MOOCs, peer assessment has been popularly used to measure the learner ability. When the number of learners increases, peer assessment is often conducted by dividing learners into multiple groups to reduce the learner's assessment workload. However, in such cases, the peer assessment…

This study explicates the nature of second language (L2) pragmatic interaction focusing on the quantitative function of interactional features. A relationship between the fine-grained interactional features elicited from learners' role-play performances at varying levels and trained raters' scores was investigated. The corpus of 102 learners'…

The Culture of Assessment Matrix is a one-page rubric intended to provide institutions of higher education with a simplistic way to quickly gauge the status of their institution's assessment culture and the degree to which the culture is an integral part of the institution's operations, expectations, methods of accountability, and reward systems.…

Academic reviews take on new directions in an ever-increasing fast-paced development. Unit reviewers should be provided different samples, with varied cultural contents also providing nuances, as related to the 'object' of the evaluation, as well as a final overall product, i.e. more of a synthesis, through which one will check the degree of…

To help evaluation professionals better understand what an authentic attempt at Developmental Evaluation (DE) ought to look like and when it is appropriate to use, Michael Patton's provocative essay (this issue) offers evaluators eight sensitizing concepts to call on as guides. Patton states these concepts succinctly define DE and persuasively…

The second article in this series on the history of evaluation use has three sections. The first and longest develops a functional definition of the term "use," noting that a thorough definition of evaluation use includes the initial stimulus (i.e., evaluation findings or process), the user, the way people use the information, the aspect…

Automated scoring of free drawings or images as responses has yet to be used in large-scale assessments of student achievement. In this study, we propose artificial neural networks to classify these types of graphical responses from a TIMSS 2019 item. We are comparing classification accuracy of convolutional and feed-forward approaches. Our…

This study compared holistic and analytic marking methods for their effects on parameter estimation (of examinees, raters, and items) and rater cognition in assessing speech act production in L2 Chinese. Seventy American learners of Chinese completed an oral Discourse Completion Test assessing requests and refusals. Four first-language (L1)…

In the last quarter of the nineteenth century, the limitations imposed by the restrictive payment-by-results curriculum were gradually relaxed, and many members of Her Majesty's Inspectorate were able to welcome the opportunity to focus their efforts upon supporting schools, teachers, and even individual pupils. This article looks at the work of…