Researchers tasked with understanding the effects of educational technology innovations face the challenge of providing evidence of causality. Given the complexities of studying learning in authentic contexts interwoven with technological affordances, conducting tightly-controlled randomized experiments is not always feasible nor desirable. Today,…

The clustered observational study (COS) design is the observational study counterpart to the clustered randomized trial. In a COS, a treatment is assigned to intact groups, and all units within the group are exposed to the treatment. However, the treatment is non-randomly assigned. COSs are common in both education and health services research. In…

Multisite field experiments using the (generalized) randomized block design that assign treatments to individuals within sites are common in education and the social sciences. Under this design, there are two possible estimands of interest and they differ based on whether sites or blocks have fixed or random effects. When the average treatment…

Background: Education researchers typically estimate average program effects with regression; if they are interested in heterogeneous effects, they include an interaction in the model. Such models quantify and infer the influences of each covariate on the effect via interaction coefficients and their associated p-values or confidence intervals.…

This report from the Career and Technical Education (CTE) Research Network Lead team provides final results from an evaluability assessment of CTE programs that feasibly could be evaluated using a rigorous experimental design. Evaluability assessments (also called feasibility studies) are used in education and other fields, such as international…

To identify the ways teachers and educational systems can improve learning, researchers need to make causal inferences. Analyses of existing datasets play an important role in detecting causal patterns, but conducting experiments also plays an indispensable role in this research. In this article, we advocate for experiments to be embedded in real…

Evaluators of education interventions are increasingly designing studies to detect impacts much smaller than the 0.20 standard deviations that Cohen (1988) characterized as "small." While the need to detect smaller impacts is based on compelling arguments that such impacts are substantively meaningful, the drive to detect smaller impacts…

Professors Richard J. Murnane and John B. Willett set out to capitalize on recent developments in education data and methodology by attempting to answer the following questions: How can new methods and data be applied most effectively in educational and social science research? What kinds of research designs are most appropriate? What kinds of…

In randomized control trials (RCTs) in the education field, the complier average causal effect (CACE) parameter is often of policy interest, because it pertains to intervention effects for students who receive a meaningful dose of treatment services. This article uses a causal inference and instrumental variables framework to examine the…

In the wake of an increasing political commitment to evidence-based decision making and evidence-based educational reform that emerged with the No Child Left Behind effort, the question of what counts as evidence has become increasingly important in the field of science education. In current public discussions, academics, politicians, and other…

This substantially revised text provides a comprehensive, highly accessible, and student friendly introduction to the principles, concepts, and methods currently used in educational research. This text provides a balanced combination of quantitative and qualitative methods and enables students to master skills in reading, understanding,…

American educational leaders seek to educate children (regardless of race, class, or economic status) to become productive citizens. In order for our country to compete with other countries where students are well educated and highly motivated, we need to ensure that our educational system is equipped with the technology and tools that would allow…

Presents an opinion on the appropriate use of significance tests, especially in the context of regression analysis, the most commonly encountered statistical technique in education and related disciplines. Briefly discusses the appropriate use of power analysis. Contains 47 references. (Author/SV)

Contrasts aptitude by treatment interaction (ATI) and hierarchical linear modeling (HLM) methods for making cross-level inferences between individual-level and group-level factors in school effectiveness research. Recommends HLM when intraclass correlations are high. ATI is suitable when intraclass correlations are low, but partitioning the…

This third in a series of articles on work conducted at the Texas Center for Educational Technology and the University of North Texas focuses on research designs for teachers to determine the impact of technology in the classroom. Highlights include research and the scientific method; qualitative versus quantitative research; and statistical…