We used multivariate generalizability theory to examine the reliability of hand-scoring and automated essay scoring (AES) and to identify how these scoring methods could be used in conjunction to optimize writing assessment. Students (n = 113) included subsamples of struggling writers and non-struggling writers in Grades 3-5 drawn from a larger…

New challenges in today's world have contributed to increased attention toward evaluating individuals' collaborative problem solving (CPS) skills. One difficulty with this work is identifying evidence of individuals' CPS capabilities, particularly when interacting in digital spaces. Often human-driven approaches are used but are limited in scale.…

To examine the utility of spoken dialog systems (SDSs) for learning and low-stakes assessment, we administered the same role-play task in two different modalities to a group of 47 tertiary-level learners of English. Each participant completed the task in an SDS setting with a fully automated agent and engaged in the same task with a human…

Classical automated test assembly (ATA) methods assume fixed and known coefficients for the constraints and the objective function. This hypothesis is not true for the estimates of item response theory parameters, which are crucial elements in test assembly classical models. To account for uncertainty in ATA, we propose a chance-constrained…

Background: Learners' writing skills are critical to their academic and professional development. Previous studies have shown that learners' self-assessment during writing is essential for assessing their writing products and monitoring their writing processes. However, conducting practical self-assessments of writing remains challenging for…

Reflective writing is an important educational practice to train reflective thinking. Currently, researchers must manually analyze these writings, limiting practice and research because the analysis is time and resource consuming. This study evaluates whether machine learning can be used to automate this manual analysis. The study investigates…

This dissertation focuses on the dynamics between teachers and machines at the intersections of design, teaching labor, and pedagogy when automation is deployed in writing classrooms. The sites of analysis are Eli Review and Turnitin, two technologies that represent different design approaches that center around "informating" or…

Assessing scientific argumentation is one of main challenges in science education. Constructed-response (CR) items can be used to measure the coherence of student ideas and inform science instruction on argumentation. Published research on automated scoring of CR items has been conducted mostly in English writing, rarely in other languages. The…

This article focuses on preservice and in-service teachers' pedagogical reasoning, decision making and action concerning technology integration for learning. We examine this topic in light of three contemporary barriers in policy, practice and research, namely: the lack of an integrative model that considers how teachers come to shape their…

Fluency in terms of speed of speech and (lack of) hesitations such as silent and filled pauses ('uhm's) is part of oral proficiency. Language assessment rubrics therefore include aspects of fluency. Measuring fluency, however, is highly time-consuming because of the manual labour involved. The current paper aims to automatically measure aspects of…

The Fourth Industrial Revolution is widely acknowledged as a digital technological revolution building on the convergence of Robotics, the Internet of Things, and the Internet of Services. What is less often acknowledged is that it is also a revolution in the social practice of work. Literacy is a core social technology of work and it is changing…

Digitalization and automation of test administration, score reporting, and feedback provision have the potential to benefit large-scale and formative assessments. Many studies on automated essay scoring (AES) and feedback generation systems were published in the last decade, but few connected AES and feedback generation within a unified framework.…

For assessment professionals who are also automated scoring (AS) professionals, there is no single set of standards of best practice. This paper reviews the assessment and AS literature to identify key standards of best practice and ethical behavior for AS professionals and codifies those standards in a single resource. Having a unified set of AS…

Providing consistent, individualized feedback to teachers is essential for improving instruction but can be prohibitively resource intensive in most educational contexts. We develop an automated tool based on natural language processing to give teachers feedback on their uptake of student contributions, a high-leverage teaching practice that…

In this digital ITEMS module, Dr. Sue Lottridge, Amy Burkhardt, and Dr. Michelle Boyer provide an overview of automated scoring. Automated scoring is the use of computer algorithms to score unconstrained open-ended test items by mimicking human scoring. The use of automated scoring is increasing in educational assessment programs because it allows…