Research in psychology and education tend to use large-N group designs that necessitate reporting of mean measures analyzed mostly with null hypothesis statistical testing (NHST), but sometimes with Bayesian, or the estimation approaches in inferential statistics. These approaches all render the average person or student as the the putative…

The use of Bayesian Knowledge Tracing (BKT) models in predicting student learning and mastery, especially in mathematics, is a well-established and proven approach in learning analytics. In this work, we report on our analysis examining the generalizability of BKT models across academic years attributed to "detector rot." We compare the…

This study utilizes an analysis of diminishing returns to examine the relationship between sample size and item parameter estimation precision when utilizing the Masters' Partial Credit Model for polytomous items. Item data from the standardization of the Batelle Developmental Inventory, 3rd Edition were used. Each item was scored with a…

Computational thinking (CT) is a fundamental ability for learners in today's society. Although CT assessments and interventions have been studied widely, little is known about CT predictions. This study predicted students' CT achievement in the ICILS 2018 using five machine learning models. These models were trained on the data from five European…

Randomized trials play an important role in estimating the effect of a policy or social work program in a given population. While most trial designs benefit from strong internal validity, they often lack external validity, or generalizability, to the target population of interest. In other words, one can obtain an unbiased estimate of the study…

The big data imposes the key problem of generalizability of the results. In the present contribution, we discuss statistical tools which can help to select variables adequate for target level of abstraction. We show that a model considered as over-fitted in one context can be accurate in another. We illustrate this notion with an example analysis…

High school graduation is "not yet a reliable indicator of college readiness", (Gaertner & McClarty, 2015, p2). As such, researchers are investigating the use of non-cognitive factors as predictors of College and Career Readiness (CCR). The College and Career Readiness Mathematical Practice Scale (CCRMS) was designed to measure…

Policymakers and school administrators have embraced value-added models of teacher effectiveness as tools for educational improvement. Teacher value-added estimates may be viewed as complicated scores of a certain kind. This suggests using a test validation model to examine their reliability and validity. Validation begins with an interpretive…

Although students' invented strategies typically prove to be effective in the improvement of students' mathematical understanding, little is known about how preservice teachers interpret and respond to student-invented strategies on whole number multiplication. This study investigated the nature of 25 preservice teachers' interpretations of and…

When validating assessment models built with data mining, generalization is typically tested at the student-level, where models are tested on new students. This approach, though, may fail to find cases where model performance suffers if other aspects of those cases relevant to prediction are not well represented. We explore this here by testing if…

For the past few decades, instructional materials enriched with multimedia elements have enjoyed increasing popularity. Multimedia-based instruction incorporating stimulating visuals, authentic audios, and interactive animated graphs of different kinds all provide additional and valuable opportunities for students to learn beyond what conventional…

In alignment studies, expert raters evaluate assessment items against standards and ratings are used to compute various alignment indices. Questions about rater reliability, however, are often ignored or inadequately addressed. This paper reports the results of a generalizability theory study of cognitive demand and rigor ratings of assessment…

Significant momentum exists for next-generation assessments to increasingly utilize technology to develop and deliver performance-based assessments. Many traditional challenges with this assessment approach still apply, including psychometric concerns related to performance-based tasks (PBTs), which include low reliability, efficiency of…

Reviews some of the more salient conceptual features of generalizability (G) theory. Provides a framework for considering some misconceptions about G theory. Focuses on particularly prevalent or serious misconceptions. G theory provides a rich conceptual framework and tools for working with a defined universe, but it does not tell an investigator…

Generalizability theory provides a technique for accurately estimating the reliability of measurements. The power of this theory is based on the simultaneous analysis of multiple sources of error variances. Equally important, generalizability theory considers relationships among the sources of measurement error. Just as multivariate inferential…