This article discusses the use of propensity scoring in experimental program evaluations to estimate impacts for subgroups defined by program features and participants' program experiences. The authors discuss estimation issues and provide specification tests. They also discuss the use of an overlooked data collection design--obtaining predictions…

The choice between experimental research or process-oriented oriented research as the only valid paradigm of evaluation research is rejected. It is argued that there is a middle ground. Suggestions are made for mixing the two approaches to suit particular research settings. (Author/GK)

Introspective interviewing can often determine the magnitude of relations more meaningfully than statistical analysis. Deduction from empirically validated premises avoids many research design problems. Guesswork can be combined with sensitivity analysis to determine the effects of guesses and missing information on conclusions. (Author/DWH)

The use of evaluations for policy and program development and change is critically reviewed. Existing conceptions of utilization are discussed and improvements in the methods of detecting use are suggested. Five clusters of variables affecting utilization are described and hypotheses about the reasons for their effects are outlined. (Author/AL)

Trade-offs are implicit in choosing a subjective or objective method for evaluating social programs. The differences between Bayesian and traditional statistics, decision and cost-benefit analysis, and anthropological and traditional case systems illustrate trade-offs in choosing methods because of limited resources. (SLD)

Six potential problems with the use of randomized experiments to evaluate programs in the field are addressed. Problems include treatment dilution, treatment contamination or confounding, inaccurate case flow and power estimates, violations of the random assignment processes, changes in the environmental context, and changes in the treatment…

This response to TM 505 708 briefly reviews the five major points of that article, and adds seven points that evaluators should consider when preparing reports. Illustrations are taken from Project Follow Through. (BW)

This article presents a synopsis of each of seven presentations given at a conference on design and analysis in community trial studies. Papers identify problems with community trials and discuss strengths and weaknesses associated with design and analysis strategies. Areas of consensus are summarized. (SLD)

Design, analysis, and reporting considerations for the application of analysis of covariance (ANCOVA) techniques in educational settings are described. Numerous examples are drawn from the national follow through evaluation, and suggestions for improving reports using ANCOVA-type techniques are presented. (Author/BW)

Electricity conservation programs were implemented in matched pairs of office-classroom-laboratory buildings and dormitories. The methodological problems of predicting consumption levels, interpreting why changes in consumption occurred, and estimating initial waste levels are discussed along with their implications for the conduct of behavioral…

Meta-analysis of Title I program evaluations shows the norm-referenced model overestimates positive effectiveness; while the regression-discontinuity design underestimates it. Potential biases include residual regression artifacts, attrition and time-of-testing problems in the norm-referenced design, and assignment, measurement, and data…

Trends in European practice that relate to qualitative assessment in the evaluation of the impact of research and innovation are discussed and analyzed. To date, European evaluations have been mainly concerned with quality and direct impact with few assessments of medium- or long-term impact. (SLD)

A meta-analysis of school-based substance abuse prevention studies revealed that the mean proportion of subjects retained dropped from 81.4 percent at the three-month followup to 67.5 percent at the three-year followup. Researchers should interpret their results in light of these normative data and adopt second-effort strategies to reduce…

Statistical methods for program evaluation with nonexperimental data are reviewed with emphasis on circumstances in which nonexperimental data are valid. Three solutions are proposed for problems of selection bias, and implications for evaluation design and data collection and analysis are discussed. (SLD)

A solution to the problem of specification error due to excluded variables in statistical models of treatment effects in nonrandomized (nonequivalent) control group designs is presented. It involves longitudinal observation with at least two pretests. A maximum likelihood estimation program such as LISREL may provide reasonable estimates of…