In this response, we first show that Simpson's proposed analysis answers a different and less interesting question than ours. We then justify the choice of prior for our Bayes factors calculations, but we also demonstrate that the substantive conclusions of our article are not substantially affected by varying this choice.

A recent paper uses Bayes factors to argue a large minority of rigorous, large-scale education RCTs are "uninformative." The definition of "uninformative" depends on the authors' hypothesis choices for calculating Bayes factors. These arguably overadjust for effect size inflation and involve a fixed prior distribution,…

Hypothesis generation and testing is outlined as an additional domain for program evaluators. Program evaluation involves a thorough analysis of the processes that contribute to change (or a lack of change) among program recipients. This process of change is analyzed in two ways: (1) treating programs as naturally occurring field studies; and (2)…

Statistical methods for program evaluation with nonexperimental data are reviewed with emphasis on circumstances in which nonexperimental data are valid. Three solutions are proposed for problems of selection bias, and implications for evaluation design and data collection and analysis are discussed. (SLD)

The first author set out to design and secure funding for an hypothesis-based program in a public school setting. The natural history of what happened to that study as it proceeded from design, to funding, to actual implementation, to final reporting serves as the case history of two idealistic evaluators' wildest nightmares. (Author)

Title I evaluations using the RMC Model C design depend for their interpretation on the assumption that the regression of posttest on pretest is linear across the cut score level when there is no treatment; but there are many instances where nonlinearities may occur. If one applies the analysis of covariance, or model C analysis, large errors may…