This scope-review presents the milestones of how Hierarchical Rater Models (HRMs) become operable to used in automated essay scoring (AES) to improve instructional evaluation. Although essay evaluations--a useful instrument for evaluating higher-order cognitive abilities--have always depended on human raters, concerns regarding rater bias,…

Engaging students in creating learning resources has demonstrated pedagogical benefits. However, to effectively utilize a repository of student-generated content (SGC), a selection process is needed to separate high- from low-quality resources as some of the resources created by students can be ineffective, inappropriate, or incorrect. A common…

Assessment practitioners in higher education follow a variety of paths to their roles. Diverse preparation supports creative problem-solving in a changing educational landscape. However, it can also lead to inconsistency in language, preparation, and background knowledge. Further, the chasms between assessment practitioners' paths can lead to…

Institutions of higher education are asked to consider how their work can advance equity in institutional outcomes. Assessment, too, has been asked to consider the ways in which traditional student assessment "privileges and validates certain types of learning and evidence of learning over others" (Montenegro and Jankowski 2017, p. 5).…

This article presents the performance of item response theory (IRT) models when double ratings are used as item scores over single ratings when rater effects are present. Study 1 examined the influence of the number of ratings on the accuracy of proficiency estimation in the generalized partial credit model (GPCM). Study 2 compared the accuracy of…

Comparative judgment is gaining popularity as an assessment tool, including for high-stakes testing purposes, despite relatively little research on the use of the technique. Advocates claim two main rationales for its use: that comparative judgment is valid because humans are better at comparative than absolute judgment, and because it distils the…

The George Mason University (Mason) offices responsible for institutional reporting and assessment were merged to strengthen and solidify their functions and expertise. The organizational structure creating the Office of Institutional Effectiveness and Planning (OIEP) was intended to engineer a new paradigm where assessment would inform the…

Topic models are mathematical and statistical models used to analyze textual data. The objective of topic models is to gain information about the latent semantic space of a set of related textual data. The semantic space of a set of textual data contains the relationship between documents and words and how they are used. Topic models are becoming…

As the development and application of large language models (LLMs) in physics education progress, the well-known AI-based chatbot ChatGPT4 has presented numerous opportunities for educational assessment. Investigating the potential of AI tools in practical educational assessment carries profound significance. This study explored the comparative…

Scoring procedures for the constructed-response (CR) items in large-scale mixed-format educational assessments often involve checks for rater agreement or rater reliability. Although these analyses are important, researchers have documented rater effects that persist despite rater training and that are not always detected in rater agreement and…

In recent decades, the assessment of instructional quality has grown into a popular and well-funded arm of educational research. The present study contributes to this field by exploring first impressions of untrained raters as an innovative approach of assessment. We apply the thin slice procedure to obtain ratings of instructional quality along…

This article questions whether social justice can live within the structural racism present in the field of evaluation. Structural racism refers to the totality of ways in which societies foster racial discrimination through mutually reinforcing systems of housing, education, employment, earnings, benefits, credit, media, health care, and criminal…

Science, Technology, Engineering, and Mathematics (STEM) education has become increasingly important in the US due to its influence on the nation's educational needs, the creation of a skilled labor force, and opportunities for more tech-savvy workers. However, the evaluation approaches and methodologies used in STEM education programs have come…

The investigators sought to determine whether education evaluators, mainly school psychologists, complied with federal, state, and professional practice guidelines when assessing English learning (EL) school-aged children suspected of a learning disability in three northern California school districts. In accordance with the Individuals with…

Kazakhstan has made tremendous progress in ensuring universal access to primary and lower-secondary education. Nevertheless, results international surveys reveal that almost two-thirds of students from Kazakhstan complete schooling without mastering the basic skills needed to be successful, and that student achievement is increasingly inequitable.…