Despite the availability and potential usefulness of demographic and contextual data in many quantitative studies within engineering education, the preference for ANOVA over regression models remains prevalent, often without clear justification. A mapping review of literature from the EJEE and JEE spanning 2012-2022 identified 98 studies using…

Matching-adjusted indirect comparison (MAIC) enables indirect comparisons of interventions across separate studies when individual patient-level data (IPD) are available for only one study. Due to its similarity with propensity score weighting, it has been speculated that MAIC can be combined with outcome regression models in the spirit of…

This paper compares the use of confidence intervals (CIs) and a sensitivity analysis called the number needed to disturb (NNTD), in the analysis of research findings expressed as 'effect' sizes. Using 1,000 simulations of randomised trials with up to 1,000 cases in each, the paper shows that both approaches are very similar in outcomes, and each…

When there are no personal consequences associated with test performance for examinees, rapid guessing (RG) is a concern and can differ between subgroups. To date, the impact of differential RG on item-level measurement invariance has received minimal attention. To that end, a simulation study was conducted to examine the robustness of the…

To further understand the properties of data-generation algorithms for multivariate, nonnormal data, two Monte Carlo simulation studies comparing the Vale and Maurelli method and the Headrick fifth-order polynomial method were implemented. Combinations of skewness and kurtosis found in four published articles were run and attention was…

Conditional independence (CI) between response time and response accuracy is a fundamental assumption of many joint models for time and accuracy used in educational measurement. In this study, posterior predictive checks (PPCs) are proposed for testing this assumption. These PPCs are based on three discrepancy measures reflecting different…

Ragin's Qualitative Comparative Analysis (QCA) is often used with small to medium samples where the researcher has good case knowledge. Employing it to analyse large survey datasets, without in-depth case knowledge, raises new challenges. We present ways of addressing these challenges. We first report a single QCA result from a configurational…

Three decision consistency and accuracy (DC/DA) methods, the Livingston and Lewis (LL) method, LEE method, and the Hambleton and Han (HH) method, were evaluated. The purposes of the study were: (1) to evaluate the accuracy and robustness of these methods, especially when their assumptions were not well satisfied, (2) to investigate the "true"…

Robustness and power of parametric, semi-parametric, and nonparametric tests of between-group discordance were compared in this simulation study. The empirical Type I error rates and power of nine tests were compared. When data were sampled from the any differences especially favor young women in single-sex Catholic secondary schools, and whether…

In this paper, six statistical tests of stochastic equality are compared with respect to Type I error and power through a Monte Carlo simulation. In the simulation, the skewness and kurtosis levels and the extent of variance heterogeneity of the two parent distributions were varied across a wide range. The sample sizes applied were either small or…

Five methods for testing the hypothesis of independence between two sets of variates were compared through simulation. Results indicate that two new methods, based on robust measures reflecting the linear association between two random variables, provide reasonably accurate control over Type I errors. Drawbacks to rank-based methods are discussed.…

The main goal of this study was to assess how sensitive unidimensional parameter estimates derived from BILOG were when the unidimensionality assumption was violated and the underlying ability distribution was not multivariate normal. A multidimensional three-parameter logistic distribution that was a straightforward generalization of the…

C. L. Olson (1976, 1979) suggests the Pillai-Bartlett trace (V) as an omnibus multivariate analysis of variance (MANOVA) test statistic for its superior robustness to heterogeneous variances. J. Stevens (1979, 1980) contends that the robustness of V, Wilk's lambda (W) and the Hotelling-Lawley trace (T) are similar, and that their power functions…

Comparing variability of test scores across alternate forms, test conditions, or subpopulations is a fundamental problem in psychometrics. A confidence interval for a ratio of standard deviations is proposed that performs as well as the classic method with normal distributions and performs dramatically better with nonnormal distributions. A simple…