This study presents several multivariate Generalizability theory designs for analyzing automatic item-generated (AIG) based test forms. The study used real data to illustrate the analysis procedure and discuss practical considerations. We collected the data from two groups of students, each group receiving a different form generated by AIG. A…

Predicting student performance has been a critical focus of educational research. With an effective predictive model, schools can identify potentially at-risk students and implement timely interventions to support student success. Recent developments in educational data mining (EDM) have introduced several machine learning techniques that can…

Most existing solutions to the current replication crisis in science address only the factors stemming from specific poor research practices. We introduce a novel mechanism that leverages the experts' predictive abilities to analyze the root causes of replication failures. It is backed by the principle that the most accurate predictor is the most…

The multistage testing (MST) design has been gaining attention and popularity in educational assessments. For testing programs that have small test-taker samples, it is challenging to calibrate new items to replenish the item pool. In the current research, we used the item pools from an operational MST program to illustrate how research studies…

The Monitoring the Future (MTF) study monitors historical and developmental changes in substance use prevalence among key subgroups of the general U.S. adolescent and adult population. The current study first devised and evaluated a cohort-specific pooled analysis weighing procedure for the MTF panel study that weighted back to the initial 12th…

While score comparability across test forms typically relies on common (or randomly equivalent) examinees or items, innovations in item formats, test delivery, and efforts to extend the range of score interpretation may require a special data collection before examinees or items can be used in this way--or may be incompatible with common examinee…

The Data Quality Campaign (DQC) has been reviewing state report cards for the past seven years. They continue to examine the landscape of state report cards because they believe states must increase transparency and build trust by sharing information. But after many years, it was time to look at state report cards with fresh eyes. In addition to…

The EdSurvey package helps users download, explore variables in, extract data from, and run analyses on large-scale assessment data. The analysis functions in EdSurvey account for the use of plausible values for test scores, survey sampling weights, and their associated variance estimator. We describe the capabilities of the package in the context…

Background: This study describes a method for harmonizing data collected with different tools to compute a rating of compliance with national recommendations for school physical activity (PA) and nutrition environments. Methods: We reviewed questionnaire items from 84 elementary schools that participated in the Childhood Obesity Research…

Technical documentation for educational tests focuses primarily on properties of individual scores at single points in time. Reliability, standard errors of measurement, item parameter estimates, fit statistics, and linking constants are standard technical features that external stakeholders use to evaluate items and individual scale scores.…

Large-scale assessment survey (LSAS) data are collected via complex sampling designs with special features (e.g., clustering and unequal probability of selection). Multilevel models have been utilized to account for clustering effects whereas the probability weighting approach (PWA) has been used to deal with design informativeness derived from…

We examined the outcomes of elementary school-based problem-solving teams (PSTs) who participated in a multi-component consultation focused on enhancing systematic problem solving. Consultation provided to each PST included training in the use of a problem-solving protocol (i.e., "Outcomes: Planning Monitoring, and Evaluating"…

Feedback on how students progress through completing subgoals can improve students' learning and motivation in programming. Detecting subgoal completion is a challenging task, and most learning environments do so either with "expert-authored" models or with "data-driven" models. Both models have advantages that are…

Recommendations from multiple professional organizations (e.g., American Psychological Association, Council for Exceptional Children, National Association of School Psychologists) suggest that collection of data on the social validity in practice and research is necessary. The purpose of this study was to systematically review the inclusion of…

Previous research shows that the direction of rating scales can influence participants' response behavior. Studies also suggest that the device used to complete online surveys might affect the susceptibility to these effects due to the different question layouts (e.g., horizontal grids vs. vertical individual questions). This article contributes…