Background: Return on investment (ROI) is one of the most popular evaluation metrics. ROI analysis (when applied correctly) is a powerful tool of evaluating existing information systems and making informed decisions on the acquisitions. However, practical use of the ROI is complicated by a number of uncertainties and controversies. The article…

To use environmental program evaluation to increase effectiveness, predictive power, and resource allocation efficiency, evaluators need good data. Data require sufficient credibility in terms of fitness for purpose and quality to develop the necessary evidence base. The authors examine elements of data credibility using experience from critical…

"How New York City's Charter Schools Affect Achievement" estimates the effects on student achievement of attending a New York City charter school rather than a traditional public school and investigates the characteristics of charter schools associated with the most positive effects on achievement. Because the report relies on an…

The aim of this paper is to explore methods for evaluating the effects of randomized interventions in a longitudinal design. The focus is on methods for modeling the possibly nonlinear relationship between treatment effect and baseline and evaluating the treatment effect taking this nonlinear relationship into account. A control/treatment growth…

Some of the threats to internal validity that arise when evaluating the impact of water conservation programs during a drought are illustrated. These include differential response to the drought, self-selection bias, and measurement error. How to deal with these problems when high-quality disaggregate data are available is discussed. (SLD)

Recent methodological advances are described that enable the analyst to extract additional information from the data envelopment analysis (DEA) methodology, including goal programming to develop cross-efficiencies, cluster analysis, analysis of variance, and pooled cross section time-series analysis. Some shortcomings of DEA are discussed. (LMO)

The reliability of criterion-referenced tests (CRTs) used in health program evaluation can be conceptualized in different ways. Formulas are presented for estimating appropriate standard error of measurement (SEM) for CRTs. The SEM can be used in computing confidence intervals for domain score estimates and for a cut-score. (Author/LMO)

Stockton Unified School District's successful strategy of using a locally developed, non-normed achievement test to implement the norm referenced model of the Title I Evaluation and Reporting System (Model A2) is described. Documented are the procedures involved in the development of a curriculum guide and test items, and the administration of…

Potential methods are explored for removing or otherwise controlling random measurement error, assessment artifacts, irrelevant variation in outcome measures, and confounding sources of covariation in a structural equations model. Using examples with measures of quality of life and functioning, the authors consider these methods for field…

The reliability of criterion referenced tests, which are often used to evaluate health education programs, may be conceptualized in different ways. Classical conceptualizations of test reliability have limited usefulness when applied to health-related criterion referenced tests. When a cutting score is set, test reliability can be represented as…

The regression-discontinuity approach to evaluating educational programs is reviewed, and regression-discontinuity post-program mean differences under various conditions are discussed. The regression-discontinuity design is used to determine whether post-program differences exist between an experimental program and a control group. The difference…

The statistical technique, "Zero-One Linear Programming," that has successfully been used to create multiple tests with similar characteristics (e.g., item difficulties, test information and test specifications) in the area of educational measurement, was deemed to be a suitable method for creating multiple sets of matched samples to be…

Relevance of the intersection of sociology, statistics, and public policy to the study of quality control in three family assistance programs--food stamps, Aid to Families with Dependent Children (AFDC), and Medicaid--is reviewed using a study by the National Academy of Sciences of methods for improving quality control systems. (SLD)

A study explored which nonexperimental comparison group methods provide the most accurate estimates of the impacts of mandatory welfare-to-work programs and whether the best methods work well enough to substitute for random assignment experiments. Findings were compared for nonexperimental comparison groups and statistical adjustment procedures…

This document contains three papers developed by the Center for the Study of Evaluation's (CSE's) Research into Practice Project. The first paper, "A Process for Designing and Implementing a Dual Purpose Evaluation System," by Pamela Aschbacher and James Burry, provides a model for evaluating programs for two purposes simultaneously: (1) program…