Assessing students' answers and in particular natural language answers is a crucial challenge in the field of education. Advances in transformer-based models such as Large Language Models (LLMs), have led to significant progress in various natural language tasks. Nevertheless, amidst the growing trend of evaluating LLMs across diverse tasks,…

Though qualitative research has become more prevalent in practice over the last 30 years, there is still considerable uncertainty among researchers regarding how to ensure inter-rater consistency when teams are tasked with coding qualitative data. In this article, we offer an explanation of a methodology our qualitative team used to achieve…

For randomized trials that use text as an outcome, traditional approaches for assessing treatment impact require that each document first be manually coded for constructs of interest by trained human raters. This process, the current standard, is both time-consuming and limiting: even the largest human coding efforts are typically constrained to…

Linking data together can be a powerful way for governments and researchers alike to tackle vexing public policy research problems. However, for researchers, finding ways to link data directly between two departments can often be more challenging than even obtaining the data in the first place. Even when a researcher develops the necessary…

When text datasets are very large, manually coding line by line becomes impractical. As a result, researchers sometimes try to use machine learning algorithms to automatically code text data. One of the most popular algorithms is topic modeling. For a given text dataset, a topic model provides probability distributions of words for a set of…

Reliable and valid assessment of parenting and child behaviors is critical for clinicians and researchers alike, and observational measures of parenting behaviors are often considered the gold standard for assessing parenting and parent-child interaction quality. The current study sought to evaluate the reliability and validity of the Coder…

Worked examples, which present an explained code for solving typical programming problems are among the most popular types of learning content in programming classes. Most approaches and tools for presenting these examples to students are based on line-by-line explanations of the example code. However, instructors rarely have time to provide…

Text classification has allowed researchers to analyze natural language data at a previously impossible scale. However, a text classifier is only as valid as the the annotations on which it was trained. Further, the cost of training a classifier depends on annotators' ability to quickly and accurately apply the coding scheme to each text. Thus,…

Purpose: The aim of this study was to advance the analysis of written language transcripts by validating an automated scoring procedure using an automated open-access tool for calculating morphological complexity (MC) from written transcripts. Method: The MC of words in 146 written responses of students in fifth grade was assessed using two…

Computational thinking (CT) has emerged as a key topic of interest in K-12 education. Children that are exposed at an early age to STEM curriculum, such as computer programming and computational thinking, demonstrate fewer obstacles entering technical fields. Increased knowledge of programming and computation in early childhood is also associated…

This project explored the reliability and utility of transcription in coding qualitative data across two studies in a program evaluation context. The first study tested the method of direct audio coding, or coding audio files without transcripts, using qualitative data software. The presence and frequency of codes applied in direct audio coding…

More than ever in the past, researchers have access to broad, educationally relevant text data from sources such as literature databases (e.g., ERIC), an open-ended response from online courses/surveys, online discussion forums, digital essays, and social media. These advances in data availability can dramatically increase the possibilities for…

Purpose: As the federal government has retreated from taking a dominant role in encouraging implementation of common K-12 standards, states and districts have moved to fill this education policy vacuum. This study aims to understand how state and district leaders are navigating this new policy environment. Research Methods/Approach: Drawing upon…

The aim of this study was to explore how the Coding as Another Language (CAL) curriculum, developed by Boston College's DevTech Research Group and utilizing the ScratchJr app, impacted students' computational thinking, coding skills, and reading comprehension. To accomplish this, the research team randomly assigned thirteen schools in a…

Learning coding during early childhood is an effective way for children to practice computational thinking. Aspects of children's motivation can increase the likelihood that children approach computational thinking activities with enthusiasm and deep engagement. Gender inequities may interfere with children's readiness to take advantage of…