The purpose of this study was to evaluate the degree of classification quality in the basic latent class model when covariates are either included or are not included in the model. To accomplish this task, Monte Carlo simulations were conducted in which the results of models with and without a covariate were compared. Based on these simulations,…

We have maintained the customer grade system that is being implemented to customers with excellent performance through customer segmentation for years. Currently, financial institutions that operate the customer grade system provide similar services based on the score calculation criteria, but the score calculation criteria vary from the financial…

One of the many impacts of the COVID-19 pandemic has been the increasing prevalence and accessibility of online education. This trend has also introduced challenges for students, instructors, and institutions. This study examines factors affecting online course satisfaction, focusing on individual, instructor, and institutional level…

The synthetic control method (SCM) is a popular approach for estimating the impact of a treatment on a single unit in panel data settings. The "synthetic control" is a weighted average of control units that balances the treated unit's pre-treatment outcomes and other covariates as closely as possible. A critical feature of the original…

Differential item functioning (DIF) analysis refers to procedures that evaluate whether an item's characteristic differs for different groups of persons after controlling for overall differences in performance. DIF is routinely evaluated as a screening step to ensure items behavior the same across groups. Currently, the majority DIF studies focus…

Early identification of students needing additional support is a foundational component of Multi-Tiered Systems of Support (MTSS). Due to the resource-intensive nature of implementing MTSS, it is critical that universal screening procedures are maximally accurate and efficient. The purpose of this study was to compare the classification accuracy…

Knowledge tracing (KT) models are a popular approach for predicting students' future performance at practice problems using their prior attempts. Though many innovations have been made in KT, most models including the state-of-the-art Deep KT (DKT) mainly leverage each student's response either as correct or incorrect, ignoring its content. In…

The assessment of program functionality can generally be accomplished with straight-forward unit tests. However, assessing the design quality of a program is a much more difficult and nuanced problem. Design quality is an important consideration since it affects the readability and maintainability of programs. Assessing design quality and giving…

Metrics including Cohen's kappa, precision, recall, and F[subscript 1] are common measures of performance for models of discrete student states, such as a student's affect or behaviour. This study examined discrete model metrics for previously published student model examples to identify situations where metrics provided differing perspectives on…

In Dynamic Assessment (DA), the observation that individuals respond differently to support, or mediation, is important for diagnoses of development. The concept of learning potential refers to openness to mediation, i.e., the extent of change to performance when mediation is available, which may suggest learners will need less overall instruction…

The explosion of Open Educational Resources (OERs) in the recent years creates the demand for scalable, automatic approaches to process and evaluate OERs, with the end goal of identifying and recommending the most suitable educational materials for learners. We focus on building models to find the characteristics and features involved in…

Multilayer perceptron neural networks, Gaussian naive Bayes, and logistic regression classifiers were compared when used to make early predictions regarding one-year college student persistence. Two iterations of each model were built, utilizing a grid search process within 10-fold cross-validation in order to tune model parameters for optimal…

Model evaluation -- the process of making inferences about the performance of predictive models -- is a critical component of predictive modelling research in learning analytics. We survey the state of the practice with respect to model evaluation in learning analytics, which overwhelmingly uses only naïve methods for model evaluation or…

In experimental evaluations of health and social programs, the role of dosage is rarely explored because researchers cannot usually randomize individuals to experience varying dosage levels. Instead, such evaluations reveal the average effects of exposure to an intervention, although program exposure may vary widely. This article compares three…

Measures of student disadvantage--or risk--are critical components of equity-focused education policies. However, the risk measures used in contemporary policies have significant limitations, and despite continued advances in data infrastructure and analytic capacity, there has been little innovation in these measures for decades. We develop a new…