In an effort to maintain internal validity while maximizing the generalizability of evaluation results, the theory-testing methodology that links program activities to outcomes (program theory) is proposed. To evaluate a program by explicating and testing the program theory is to perform evaluation as theory testing. (PN)

External and implementation factors and their potential threats to field test validity and the practical realities of the field testing process on results are discussed. External factors include site selection, negotiations, contracts, site monitoring, training, and type of product. Implementation factors include motivation, dependence on local…

Improving the validity and reliability of surveys is a critical part of the response to the call for improved rigor of education research, policy analysis and evaluation. Too often we create inquiry tools without validating our measures against how respondents interpret our questions, and therefore collect data of questionable quality. The purpose…

A study of school district uses of evaluation and test data is used to illustrate the variety of ambiguities that can appear in qualitative data. Rules of evidence to distinguish valid and potentially nonvalid testimony are proposed. Uses of testimony in evaluation are discussed. (BS)

A model is described which illustrates two general uses of multiple analyses to evaluate quasi-experiments: obtaining estimates of the treatment effect, and checking the validity of the experiment. (MH)

ifferent evaluation models were applied to data from a high school career education program to investigate problems in statistical conclusion validity and program effectiveness judgments. If potential threats to internal validity are analyzed and protection strategies are developed, more confidence in unplanned ex post facto design using a…

To determine the criterion-related validity of the Dade County (Florida) Management Assessment Center for the selection of school-level administrators, ratings of 121 school principals, assistant principals, and candidates were correlated with subsequent job performance ratings. The center predicted job performance to a significant degree. (SLD)

An approach for involving beneficiaries in specifying program attributes for evaluators to address is discussed. Methods are outlined for sharing influence among program decision makers and beneficiaries. Application of this approach in the evaluation of a problem-based medical school curriculum is described, with evidence on the enhancement of…

The internal validity of the RMC models, especially Model A, is examined. Concern centers on limiting evaluation to cognitive outcomes, using constant percentile as the no-treatment expectation, and using norms for one test to establish the expected no-treatment performance level for another test. (MH)

This researcher discusses what he has learned about the field of program evaluation and makes three observations: (1) Testing does not define key objectives in evaluation. (2) Politics influences what clients want to know. (3) Case studies are necessary for epistemological reasons. (LMO)