In a clustered observational study, a treatment is assigned to groups and all units within the group are exposed to the treatment. We develop a new method for statistical adjustment in clustered observational studies using approximate balancing weights, a generalization of inverse propensity score weights that solve a convex optimization problem…

In education settings, treatments are often non-randomly assigned to clusters, such as schools or classrooms, while outcomes are measured for students. This research design is called the clustered observational study (COS). We examine the consequences of common support violations in the COS context. Common support violations occur when the…

The field of education relies heavily on instructional coaches to build teacher capacity in the implementation of evidence-based practices (EBPs). Although observation tools are commonly used to measure the fidelity of implementation by teachers, fewer tools are available to identify specific coaching behaviors used during in situ coaching…

Two- and three-level designs in educational and psychological research can involve entire populations of Level-3 and possibly Level-2 units, such as schools and educational districts nested within a given state, or neighborhoods and counties in a state. Such a design is of increasing relevance in empirical research owing to the growing popularity…

Single-case experimental designs (SCEDs) have been used with increasing frequency to identify evidence-based interventions in education. The purpose of this study was to explore how several procedural characteristics, including within-phase variability (i.e., measurement error), number of baseline observations, and number of intervention…

The field of education relies heavily on instructional coaches to build teacher capacity in the implementation of evidence-based practices (EBPs). Although observation tools are commonly used to measure the fidelity of implementation by teachers, fewer tools are available to identify specific coaching behaviors used during in situ coaching…

The clustered observational study (COS) design is the observational study counterpart to the clustered randomized trial. In a COS, a treatment is assigned to intact groups, and all units within the group are exposed to the treatment. However, the treatment is non-randomly assigned. COSs are common in both education and health services research. In…

This paper explores the concept of adaptive research design, in which topic, theoretical framing, method, and data are "in principle" open to adaptation during the research process. The main premise is that adaptations in one element of the research process can trigger changes in other elements. Both positive and negative reasons for…

Context: A perennial problem in applied statistics is the inability to justify strong claims about cause-and-effect relationships without full knowledge of the mechanism determining selection into treatment. Few research designs other than the well-implemented random assignment study meet this requirement. Researchers have proposed partial…

In the complex world of educational policy, causal inferences will be debated. As we review non-experimental designs in educational policy, we focus on how to clarify and focus the terms of debate. We begin by presenting the potential outcomes/counterfactual framework and then describe approximations to the counterfactual generated from the…

The same L2 speaking performance may be analysed and evaluated in very different ways by different teachers or raters. We present a new, technology-assisted research design which opens up to investigation the trajectories of convergence and divergence between raters. We tracked and recorded what different raters noticed when, whilst grading a…

Studies of play often involve face-to-face research with children in settings such as homes, schools, and playgrounds. The COVID-19 pandemic affected children's play in many significant ways; yet, traditional in-person research methods were not possible due to restrictions such as lockdowns, school closures, and social distancing. "A National…

Observation research can shed light on the degree to which students have access to research-based instruction and intervention. In this systematic review of reading observation research for students with and at risk for emotional and behavioral disorders, we sought to identify trends in the settings and student populations investigated and…

Background: Over the past 20 years, much has been written about the merits and limitations of matching methods, particularly propensity score matching (see, e.g., Stuart, 2010 for a review of matching approaches), and the myriad ways matching can be implemented to identify a comparison group that is similar to the treatment group on observable…

The struggle to cast a net around the numerous ways prosocial behavior is expressed lends itself to the absence of widely accepted methods of measurement. Additionally, research intent on evaluating the psychometric properties of current approaches has been somewhat limited. Weaving together seminal as well as contemporary research, the current…