While significant attention has been given to test equating to ensure score comparability, limited research has explored equating methods for rater-mediated assessments, where human raters inherently introduce error. If not properly addressed, these errors can undermine score interchangeability and test validity. This study proposes an equating…

To address response style or bias in rating scales, forced-choice items are often used to request that respondents rank their attitudes or preferences among a limited set of options. The rating scales used by raters to render judgments on ratees' performance also contribute to rater bias or errors; consequently, forced-choice items have recently…

Background: Competency models provide useful frameworks for organizing learning and assessment programs, but their construction is both time intensive and subject to perceptual biases. Some aspects of model development may be particularly well-suited to automation, specifically natural language processing (NLP), which could also help make them…

This article shares the findings of a qualitative case study examining the experiences of teacher leaders as they engaged as teacher evaluators alongside school principals. Data collection included observations and interviews with four teacher leaders and three school principals to answer these research questions: (1) What TDEM structures have…

Researchers have explored a variety of topics related to identifying and distinguishing among specific types of rater effects, as well as the implications of different types of incomplete data collection designs for rater-mediated assessments. In this study, we used simulated data to examine the sensitivity of latent trait model indicators of…

Researchers in many disciplines work with ranking data. This data type is unique in that it is often deterministic in nature (the ranks of items "k"-1 determine the rank of item "k"), and the difference in a pair of rank scores separated by "k" units is equivalent regardless of the actual values of the two ranks in…

This cross-case synthesis gives voice to evaluators in EC-12 and higher education settings who are enacting a state-mandated system of teacher evaluation and support by examining their perceptions of the Texas Teacher Evaluation and Support System (T-TESS). Questions addressed included: How do differently situated school administrators and…

The literature on Automated Essay Scoring (AES) systems has provided useful validation frameworks for any assessment that includes AES scoring. Furthermore, evidence for the scoring fidelity of AES systems is accumulating. Yet questions remain when appraising the scoring performance of AES systems. These questions include: (a) which essays are…

Interest from Congress, executive branch leadership, and various other stakeholders for greater accountability in government continues to gain momentum today with government-wide efforts. However, measuring the impact of research programs has proven particularly difficult. Cause and effect linkages between research findings and changes to…

The importance of engaging in high quality program evaluation is a generally accepted principle underscored by external pressure from funders. High quality evaluation necessarily begins with good evaluation planning. This paper outlines Evolutionary Evaluation and specifically the Systems Evaluation Protocol, an approach that emphasizes…

Responding to an Illinois law that took effect in 2010, the Performance Evaluation Reform Act (PERA), Chicago Public Schools (CPS) created and implemented the REACH (Recognizing Educators Advancing Chicago's Students) educator evaluation and support system. REACH was piloted in 2012-13. It is still being implemented as of the 2019-20 school year,…

Rhode Island educators believe that implementing a fair, accurate, and meaningful educator evaluation and support system will help improve teaching, learning, and school leadership. The primary purpose of the Rhode Island Model Building Administrator Evaluation and Support System (Rhode Island Model) is to help all building administrators improve.…

Problem-based learning has been applied over the last three decades to a diverse range of learning environments. In this educational approach, different problems are posed to the learners so that they can develop different solutions while learning about the problem domain. When applied to conceptual modelling, and particularly to Qualitative…

This essay explores the implications of the zeitgeist that emerged in the United States during the 1960s for the conceptualization of ethical issues in evaluation and community psychology, and how perspectives from the latter field might enhance ethical practice in the former. Special attention is paid to articulations of social justice,…

In this study, we examined whether interpreters and interpreting trainees are better predisposed to respeaking than people with no interpreting skills. We tested 57 participants (22 interpreters, 23 translators and 12 controls) while respeaking 5-minute videos with two parameters: speech rate (fast/slow) and number of speakers (one/many). Having…