Although it is commonly believed that automation aids human decision-making, conflicting evidence raises questions about whether individuals would gain greater advantages from automation in difficult tasks. Our study examines the combined influence of task difficulty and automation reliability on aided decision-making. We assessed decision…

In recent years, educators, administrators, policymakers, and measurement experts have called for assessments that support educators in making better instructional decisions. One promising approach to measurement to support instructional decision-making is diagnostic classification models (DCMs). DCMs are flexible psychometric models that…

Mathematical games are widely used in primary school classrooms, yet the activities that are labelled as "games" vary considerably in their structure, cognitive demands, and potential to support student reasoning. This conceptual paper offers a typology that distinguishes between pseudo-games, superficial games, gamification, and…

In describing how people generalize from observed samples of data to novel cases, theories of inductive inference have emphasized the learner's reliance on the contents of the sample. More recently, a growing body of literature suggests that different assumptions about how a data sample was generated can lead the learner to draw qualitatively…

A decision consistency (DC) index is an estimate of the consistency of a classification decision on an exam. More specifically, DC estimates the percentage of examinees that would have the same classification decision on an exam if they were to retake the same or a parallel form of the exam again without memory of taking the exam the first time.…

In many large-scale testing programs, examinees are frequently categorized into different performance levels. These classifications are then used to make high-stakes decisions about examinees in contexts such as in licensure, certification, and educational assessments. Numerous approaches to estimating the consistency and accuracy of this…

This dissertation examined classification outcome differences among four popular individual supervised machine learning (ISML) models (logistic regression, decision tree, support vector machine, and multilayer perceptron) when predicting minor class membership within imbalanced datasets. The study context and the theoretical population sampled…

The causal view of categories assumes that categories are represented by features and their causal relations. To study the effect of causal knowledge on categorization, researchers have used Bayesian causal models. Within that framework, categorization may be viewed as dependent on a likelihood computation (i.e., the likelihood of an exemplar with…

Evaluation competency frameworks across the globe regard evaluation approaches as important to know and use in practice. Prior classifications have been developed to aid in understanding important differences among varying approaches. Nevertheless, there is an opportunity for a new classification of evaluation approaches, in particular one that is…

This study investigates how 11- to 12-year-old students construct data-based decision trees using data cards for classification purposes. We examine the students' heuristics and reasoning during this process. The research is based on an eight-week teaching unit during which students labeled data, built decision trees, and assessed them using test…

Multiple measures, such as multiple content domains or multiple types of performance, are used in various testing programs to classify examinees for screening or selection. Despite the popular usages of multiple measures, there is little research on classification consistency and accuracy of multiple measures. Accordingly, this study introduces an…

When categorizing faces by gender, children show evidence of intersectionality--miscategorizing Black females, but not White females, as male. Here we examined whether differential biases for Black and White females extend beyond perceptual tasks to judgments of social preferences. Children (N = 97, ages 4-9 years; 59.8% White and 40.2% non-White)…

Test scores are often used to make decisions about examinees, such as in licensure and certification testing, as well as in many educational contexts. In some cases, these decisions are based upon compensatory scores, such as those from multiple sections or components of an exam. Classification accuracy and classification consistency are two…

We report findings of this replication study, examining the internalizing subscale (SRSS-I4) of the revised version of the Student Risk Screening Scale for Internalizing and Externalizing behavior (SRSS-IE 9) and the internalizing subscale of the Teacher Report Form (TRF). Using the sample from 13 elementary schools across three U.S. states with…

It's critical to foster artificial intelligence (AI) literacy for high school students, the first generation to grow up surrounded by AI, to understand working mechanism of data-driven AI technologies and critically evaluate automated decisions from predictive models. While efforts have been made to engage youth in understanding AI through…