This paper discusses the issues involved in calculating indices of composite reliability for "modular" or "unitised" assessments of the kind used in GCSEs, AS and A level examinations in England. The increasingly widespread use of on-screen marking has meant that the item-level data required for calculating indices of…

Propensity score analysis (PSA) is a methodological technique which may correct for selection bias in a quasi-experiment by modeling the selection process using observed covariates. Because logistic regression is well understood by researchers in a variety of fields and easy to implement in a number of popular software packages, it has…

The purpose of this paper is to review major statistical and psychometric issues impacting the study of psychophysiological reactivity and discuss their implications for applied developmental researchers. We first cover traditional approaches such as the observed difference score (DS) and the observed residual score (RS), including a review of…

Test fairness and test bias are not synonymous concepts. Test bias refers to statistical evidence that the psychometrics or interpretation of test scores depend on group membership, such as gender or race, when such differences are not expected. A test that is grossly biased may be judged to be unfair, but test fairness concerns the broader, more…

As result replicability is essential to science and difficult to achieve through external replicability, the present paper notes the insufficiency of null hypothesis statistical significance testing (NHSST) and explains the bootstrap as a plausible alternative, with a heuristic example to illustrate the bootstrap method. The bootstrap relies on…

Covariate adjustment can increase the precision of estimates by removing unexplained variance from the error in randomized experiments, although chance covariate imbalance tends to counteract the improvement in precision. The author develops an easy measure to examine chance covariate imbalance in randomization by standardizing the average…

Partially clustered designs, where clustering occurs in some conditions and not others, are common in psychology, particularly in prevention and intervention trials. This article reports results from a simulation comparing 5 approaches to analyzing partially clustered data, including Type I errors, parameter bias, efficiency, and power. Results…

Formal testing of 3 year old children is a new feature in the revised version of the Movement Assessment Battery for Children (Movement ABC-2). Our study evaluated the reliability and explored the clinical applicability of the Movement ABC-2 Test in this young age group. A total of 50 typically children were given two trials of the test within a…

Student-teacher interactions are dynamic relationships that change and evolve over the course of a school year. Measuring classroom quality through observations that focus on these interactions presents challenges when observations are conducted throughout the school year. Variability in observed scores could reflect true changes in the quality of…

The effect of unreliability of measurement on propensity score (PS) adjusted treatment effects has not been previously studied. The authors report on a study simulating different degrees of unreliability in the multiple covariates that were used to estimate the PS. The simulation uses the same data as two prior studies. Shadish, Clark, and Steiner…

Recent school science curriculum developments in many countries emphasise that scientists derive evidence for their claims through different approaches; that such practices are bound up with disciplinary knowledge; and that the quality of data should be appreciated. This position paper presents an understanding of the validity of data as a set of…

Researchers have become increasingly interested in programs' main and interaction effects of two variables (A and B, e.g., two treatment variables or one treatment variable and one moderator) on outcomes. A challenge for estimating main and interaction effects is to eliminate selection bias across A-by-B groups. I introduce Rubin's causal model to…

This study examined the psychometric properties and measurement invariance of the Arabic version of "My Class Activities" (MCA), an instrument designed to measure students' perceptions of interest, challenge, choice, and enjoyment in classrooms. Scores of 3,516 Sudanese students in Grades 2 to 8 were used. Confirmatory factor analysis…

Language learners' productive role in teaching and learning processes has recently been the focus of attention. Therefore, this study aimed at investigating the effect of oral vs. written output-based instruction on English as a foreign language (EFL) learners' vocabulary learning with a focus on reflective vs. impulsive learning styles. To this…

Goodness-of-fit testing in factor analysis is based on the assumption that the test statistic is asymptotically chi-square, but this property may not hold in small samples even when the factors and errors are normally distributed in the population. Robust methods such as Browne's (1984) asymptotically distribution-free method and Satorra Bentler's…