In this response, we first show that Simpson's proposed analysis answers a different and less interesting question than ours. We then justify the choice of prior for our Bayes factors calculations, but we also demonstrate that the substantive conclusions of our article are not substantially affected by varying this choice.

A recent paper uses Bayes factors to argue a large minority of rigorous, large-scale education RCTs are "uninformative." The definition of "uninformative" depends on the authors' hypothesis choices for calculating Bayes factors. These arguably overadjust for effect size inflation and involve a fixed prior distribution,…

The normative implications of applying significance tests in evaluation research are examined. The authors conclude that evaluators often make normative decisions, based on the traditional .05 significance level in studies with small samples. Additional reporting of the magnitude of impact, the significance level, and the power of the test is…

A novel solution to the statistical problems in an evaluation of rare events is described. The significance of variations in the number of child homicides is analyzed in a binary time series of "active" months for monitoring future incidence and related systemic events. (Author/CM)

Critical analysis of an article that attempts to refute the claims of Sesame Street'' concerning its educational value for children from poverty areas, (Author/MF)

The validity of the equipercentile hypothesis of the Title I Evaluation and Reporting System norm-referenced evaluation model was examined using 3,224 seventh- and ninth-grade students. Findings from confidence interval procedures contradicted the equipercentile hypothesis. There was a pattern of large gains for students not receiving any special…

The degree to which the chosen units of analysis are likely to produce spurious findings in staged combinations of multiple linear regression procedures are examined. The effects of grouping variables (e.g., classroom, school, and school district) on Procedures such as Coleman's semipartial regression and Mayeske's commonalities, in light of…

Although "quality of education" issues are currently popular in educational evaluation, the author argues for an empirically resolvable evaluation paradigm: Given the target population, program goals, and boundary conditions, how do educational treatments compare in terms of learning effects? Agreement on pre-set specifications and the…

Statistical methods for program evaluation with nonexperimental data are reviewed with emphasis on circumstances in which nonexperimental data are valid. Three solutions are proposed for problems of selection bias, and implications for evaluation design and data collection and analysis are discussed. (SLD)

Dismissing research that is not theory driven, hypothesis testing, or generalization producing does injustice to the variety of contributions that qualitative research can make. Studies conducted through qualitative approaches are reviewed to summarize the desirable outcomes that can result. No research model has a monopoly on quality. (SLD)

The first author set out to design and secure funding for an hypothesis-based program in a public school setting. The natural history of what happened to that study as it proceeded from design, to funding, to actual implementation, to final reporting serves as the case history of two idealistic evaluators' wildest nightmares. (Author)

Title I evaluations using the RMC Model C design depend for their interpretation on the assumption that the regression of posttest on pretest is linear across the cut score level when there is no treatment; but there are many instances where nonlinearities may occur. If one applies the analysis of covariance, or model C analysis, large errors may…

Many topics in aging research address questions about group differences in prediction. Such questions can be viewed in terms of interaction or moderator effects, and use of appropriate methods to test these hypotheses are necessary to arrive at accurate conclusions about age differences. This article discusses the conceptual, methodological, and…

Research is a useful tool for educators because of its ability to uncover causal relationships between variables that are often obscured in field investigations. This textbook presents basic methodological information that has been designed for and proven to assist the educator who seeks to move himself from the laboratory to the field. The book…