Nonverbal behavior can impact language proficiency scores in speaking tests, but there is little empirical information of the size or consistency of its effects or whether language proficiency may be a moderating variable. In this study, 100 novice raters watched and scored 30 recordings of test takers taking an international, high stakes…

Collective impact (CI) initiatives provide a promising collaborative community-based approach for addressing health disparities, but they are challenging to evaluate. This paper provides a detailed strategy for addressing these challenges through a case study, with the goal of not only of evaluating the CI, but also putting organizations in a…

In this paper we outline the use of a concept mapping narrative method to elicit the complex and interconnected conceptualisations of specialized knowledge (assessment literacy) for a specific cohort of participants (teacher-examiners). In particular, we are interested in exploring how relational concept mapping and narrative interviews can be…

Automatic machine learning (AutoML) methods automate the time-consuming, feature-engineering process so that researchers produce accurate student models more quickly and easily. In this paper, we compare two AutoML feature engineering methods in the context of the National Assessment of Educational Progress (NAEP) data mining competition. The…

Severity and centrality are two main kinds of rater effects posing threats to the validity and fairness of performance assessments. Adopting Jin and Wang's (2018) extended facets modeling approach, we separately estimated the magnitude of rater severity and centrality effects in the web-based TestDaF (Test of German as a Foreign Language) writing…

The purpose of this report is to present our process and results for establishing validity and reliability of an observation tool used to investigate teaching practices that high school mathematics teachers use to engage students. We developed our tool using established practices, such as reviewing literature to develop a framework for instruction…

The background of this article is an interest in analysing how assessment of professional skills is conducted in higher education contexts, drawing on video data from a course on maritime navigation. The empirical study focuses on a) how students working in a bridge simulator are able to display their knowledge about how to calculate the relation…

Purpose: The aim of this study was to advance the use of structured, monologic discourse analysis by validating an automated scoring procedure for core lexicon (CoreLex) using transcripts. Method: Forty-nine transcripts from persons with aphasia and 48 transcripts from persons with no brain injury were retrieved from the AphasiaBank database. Five…

Nonverbal behavior is essential in human interaction (Gullberg, de Bot, & Volterra, 2008; McNeill, 1992, 2005). For second language speakers, nonverbal features can be helpful for successful and efficient communication (e.g., Dahl & Ludvigsen, 2014). However, due to the complexity of nonverbal features, language testing institutions have…

This systematic review examines 22 studies (2024-2025) on the use of generative AI, primarily ChatGPT, for providing feedback in English writing instruction for language learners. It identifies the types of feedback AI offers, its effectiveness relative to teacher and peer feedback, and perceptions from students and teachers. Findings show AI…

This case study examined the functional role activities of developmental evaluators in a school and community partnership employing a collaborative program model to improve an urban underperforming public school. Program implementers' assessments of program developments and their experience with developmental evaluators were also examined. Data…

In a complex, multisite initiative, a well-developed theory of change can help an organization's leadership make strategic decisions about program elements, organizational direction and priorities, as well as provide structure and accountability for evaluation. This article describes the multistep process used by the evaluation team of a large…

This study investigated the effectiveness of statistical adjustments applied to rater bias in many-facet Rasch analysis. Some changes were first made in the dataset that did not include "rater × examinee" bias to cause to have "rater × examinee" bias. Later, bias adjustment was applied to rater bias included in the data file,…

Institutions of higher education are required by U.S. accrediting bodies to articulate their conceptual framework and develop a viable program improvement plan through a continuous reflective process of progressive problem solving that enables data-driven decision-making practices. To meet or exceed national accreditation standards for leadership,…

The purpose of this quantitative study was to examine the perceived impact of principal evaluation on leadership practices in Regional Office #28 Bureau, Henry, and Stark counties. This study used quantitative research methods with a descriptive and comparative research design. The research question that guided this study were: (1) To what degree…