In this article, we systematize the factors influencing performance and feasibility of automatic content scoring methods for short text responses. We argue that performance (i.e., how well an automatic system agrees with human judgments) mainly depends on the linguistic variance seen in the responses and that this variance is indirectly influenced…

For predicting and improving the quality of essays, text analytic metrics (surface, syntactic, morphological, and semantic features) can be used to provide formative feedback to the students in higher education. In this study, the goal was to identify a sufficient number of features that exhibit a fair proxy of the scores given by the human raters…

The purpose of this action research study was to examine grading practices in one military-connected high school and to make recommendations for aligning grading practices to improve student achievement, engagement, and equity. Through iterative research cycles of look, think, and act, teachers from one military-connected high school participated…

In this three-study investigation, we applied various approaches to score drawings created in response to both Form A and Form B of the Torrance Tests of Creative Thinking-Figural (broadly TTCT-F) as well as the Multi-Trial Creative Ideation task (MTCI). We focused on TTCT-F in Study 1, and utilizing a random forest classifier, we achieved 79% and…

Recent developments in the use of large-language models have led to substantial improvements in the accuracy of content-based automated scoring of free-text responses. The reported accuracy levels suggest that automated systems could have widespread applicability in assessment. However, before they are used in operational testing, other aspects of…

Background: Integrating information from multiple sources is a common yet challenging learning task for secondary school students. Many underuse metacognitive skills, such as monitoring and control, which are essential for promoting engagement and effective learning outcomes. Objective: This study aims to examine the relationship between…

Since the rise of state assessments whose primary function is to yield scores that can be used to compare schools and groups of students, most states have developed their state assessment programs under the assumption that either: (a) state tests are not intended to meaningfully shape instruction, or (b), if they are, the information provided in…

Authentic assessment allows students to demonstrate knowledge and skills in real-world tasks. In research, peer review is one such task that researchers learn by doing, as they evaluate other researchers' work. This means peer review could serve as an authentic assessment that engages students' critical thinking skills in a process of active…

RISE was designed and revised to provide a quality system, aligned with current legislative requirements that local corporations can adopt in its entirety, or use as a model as they develop evaluation systems to best suit their local contexts. RISE was developed over the course of a year by the Indiana Teacher Evaluation Cabinet, a diverse group…

Background: Concept maps are a valid assessment tool to explore student understanding of diverse topics. Many types of academic programs have integrated concept mapping into their courses, resulting in various activities and scoring methods to understand student perceptions. Purpose: Few prior reviews of concept mapping have addressed their use…

Traditionally in England, exam results in General Certificates of Secondary Education (GCSEs) (and before them O levels) and A levels have been reported as letter grades, with A (or A*) as the top grade, then B, C etc. The reforms gave the opportunity to revisit the arguments for different formats of reporting, and Cambridge Assessment contributed…

Oral reading fluency (ORF) assessments are commonly used to screen at-risk readers and evaluate interventions' effectiveness as curriculum-based measurements. Similar to the standard practice in item response theory (IRT), calibrated passage parameter estimates are currently used as if they were population values in model-based ORF scoring.…

Standardized observation systems seek to reliably measure a specific conceptualization of teaching quality, managing rater error through mechanisms such as certification, calibration, validation, and double-scoring. These mechanisms both support high quality scoring and generate the empirical evidence used to support the scoring inference (i.e.,…

Contribution: This concept article shows how the mathematical competencies research framework (MCRF) can guide the design of rubrics to assess engineering mathematics tasks. Practical guidance is given for engineering mathematics educators wanting to create effective rubrics that support student learning and promote academic success. Background:…

The rubric, a canonical matrix of criteria presented to students as the road map to academic success. An "Ah-ha" moment, "that is what I'm looking for" utopia for the instructor. While rubrics provide the possibility for solving the complexity of some teaching problems, we have come to know them as a tool that is as useful as…