Purpose: Systematic reviews of literature are routinely conducted to identify practices that are effective in addressing educational and clinical problems. One complication, however, is how best to combine data from both group experimental design (GED) studies and single-case experimental design (SCED) studies. Percent of Goal Obtained (PoGO) has…

Purpose: Systematic reviews of literature are routinely conducted to identify practices that are effective in addressing educational and clinical problems. One complication, however, is how best to combine data from both group experimental design (GED) studies and single-case experimental design (SCED) studies. Percent of Goal Obtained (PoGO) has…

This article develops new closed-form variance expressions for power analyses for commonly used difference-in-differences (DID) and comparative interrupted time series (CITS) panel data estimators. The main contribution is to incorporate variation in treatment timing into the analysis. The power formulas also account for other key design features…

Covariate adjustment can increase the precision of estimates by removing unexplained variance from the error in randomized experiments, although chance covariate imbalance tends to counteract the improvement in precision. The author develops an easy measure to examine chance covariate imbalance in randomization by standardizing the average…

The constraints of conducting evaluations in real-world settings often necessitate the implementation of less than ideal designs. Unfortunately, the standard method for estimating the precision of a result (i.e., confidence intervals [CI]) cannot be used for evaluative conclusions that are derived from multiple indicators, measures, and data…

A methodology is presented for minimizing the mean error variance-covariance component in studies with resource constraints. The method is illustrated using a one-facet multivariate design. Extensions to other designs are discussed. (SLD)

Incidence sampling is a parsimonious method whereby a large number of examinees can be measured on many variables (such as test items) to assess group characteristics. Parameters used to describe an incidence sample are estimated using the theory of generalized symmetric means and generalizability theory. (Author/JKS)

Four general criteria are proposed for the choice of a metrical solution for a causal effect: (1) compatibility with the effect; (2) ease of communication; (3) lack of measurement error bias; and (4) stability across subjects and situations. These criteria are illustrated for randomized and nonrandomized designs. (SLD)

Since meta-analysis was described in 1976 (Glass) as the application of familiar experimental methods to the integration of available research, at least five coherent approaches to meta-analysis have appeared in common use. These approaches can be divided into two broad groups. In the first group (including procedures by Robert Rosenthal, Larry…

The authors describe 2 efficiency (planned missing data) designs for measurement: the 3-form design and the 2-method measurement design. The 3-form design, a kind of matrix sampling, allows researchers to leverage limited resources to collect data for 33% more survey questions than can be answered by any 1 respondent. Power tables for estimating…

Sampling distributions for ten tests for comparing population variances in a two group design were generated for several combinations of equal and unequal sample sizes, population means, and group variances when distributional forms differed. The ten procedures included: (1) O'Brien's (OB); (2) O'Brien's with adjusted degrees of freedom; (3)…

The framework and procedures of generalizability theory are introduced and illustrated in this instructional module that uses a hypothetical scenario involving writing proficiency. Generalizability analyses are useful for understanding the relative importance of various sources of error and for designing efficient measurement procedures. (SLD)

A measurement approach called generalizability theory (G-theory) is an important alternative to the more familiar classical measurement theory that yields less useful coefficients such as alpha or the KR-20 coefficient. G-theory is a theory about the dependability of behavioral measurements that allows the simultaneous estimation of multiple…

Generalizability theory, which subsumes classical measurement theory as a special case, provides a general model for estimating the reliability of observational rating data by estimating the variance components of the measurement design. Research data from the "Heart Smart" health intervention program were analyzed as a heuristic tool.…

A teaching aid appropriate for a beginning course on experimental design is presented. The aid is a numerical example which illustrates some of the theoretical interrelations among three competing design analysis strategies for estimating treatment effects in random assignment designs. The analysis strategies considered are analysis of variance…