This study investigates the effect of survey order in measuring self-reported cognitive load. Understanding how survey order influences responses is crucial, but it has been largely overlooked in the context of cognitive load. Using a 2 × 2 experimental design with 319 high school students, the study manipulated intrinsic cognitive load (ICL)…

This study compared three approaches for handling a fourth level of nesting when analyzing cluster-randomized trial (CRT) data. Although CRT data analyses may include repeated measures, individual, and cluster levels, there may be an additional fourth level that is typically ignored. This study examined the impact of ignoring this fourth level,…

As online learning platforms become more ubiquitous throughout various curricula, there is a growing need to evaluate the effectiveness of these platforms and the different methods used to structure online education and tutoring. Towards this endeavor, some platforms have performed randomized controlled experiments to compare different user…

As online learning platforms become more ubiquitous throughout various curricula, there is a growing need to evaluate the effectiveness of these platforms and the different methods used to structure online education and tutoring. Towards this endeavor, some platforms have performed randomized controlled experiments to compare different user…

More aggregate data on school performance is available than ever before, opening up new possibilities for applied researchers interested in assessing the effectiveness of school-level interventions quickly and at a relatively low cost by implementing comparative interrupted times series (CITS) designs. We examine the extent to which effect…

We designed a serious game, MolWorlds, to facilitate conceptual change about molecular emergence by using game mechanics (resource management, immersed 3rd person character, sequential level progression, and 3-star scoring system) to encourage cycles of productive negativity. We tested the value-added effect of game design by comparing and…

This randomised controlled trial used a mixed-methods approach to investigate the frequency and how elementary teachers integrated engineering design (ED) principles into their science instruction following professional development (PD). The ED components of the PD were aligned with Cunningham and Carlsen's [(2014). "Teaching engineering…

Design-based methods have recently been developed as a way to analyze data from impact evaluations of interventions, programs, and policies. The impact estimators are derived using the building blocks of experimental designs with minimal assumptions, and have good statistical properties. The methods apply to randomized controlled trials (RCTs) and…

A new design-based theory has recently been developed to estimate impacts for randomized controlled trials (RCTs) and basic quasi-experimental designs (QEDs) for a wide range of designs used in social policy research (Imbens & Rubin, 2015; Schochet, 2016). These methods use the potential outcomes framework and known features of study designs…

Design-based methods have recently been developed as a way to analyze data from impact evaluations of interventions, programs, and policies (Imbens and Rubin, 2015; Schochet, 2015, 2016). The estimators are derived using the building blocks of experimental designs with minimal assumptions, and are unbiased and normally distributed in large samples…

The Common Guidelines for Education Research and Development were created as a joint effort between the Institute of Education Science and the National Science Foundation in an effort to streamline education research and contribute to an accumulation of knowledge that will lead to improved student outcomes. One type of research that emerged in the…

Randomized experiments are commonly used to evaluate if particular interventions improve student achievement. While these experiments can establish that a treatment actually "causes" changes, typically the participants are not randomly selected from a well-defined population and therefore the results do not readily generalize. Three…

I discuss three common practices that obfuscate or invalidate the statistical analysis of randomized controlled interventions in applied linguistics. These are (a) checking whether randomization produced groups that are balanced on a number of possibly relevant covariates, (b) using repeated measures ANOVA to analyze pretest-posttest designs, and…

School-based improvement programs represent a core strategy in improving education because they can leverage pre-existing social and organizational structures to promote coordinated and comprehensive change across multiple facets of schooling. School-based programs are generally designed to be implemented by intact schools/districts, frequently…

Pair programming is a strategy that grew out of industry and has shown promise for performance and retention in computer programming courses at universities and in industry. In this study, we examine whether pair programming is effective in K-12, what it is effective for, and how partners influence each other. We collected the data from 320 middle…