Research synthesis of evaluation findings is a multistep process. An investigator identifies a research question, acquires the relevant literature, codes findings from that literature, and analyzes the coded data to estimate the average treatment effect and its distribution in a population of interest. The process of estimating the average…

Complex survey data are collected by means other than simple random samples. This creates two analytical issues: nonindependence and unequal selection probability. Failing to address these issues results in underestimated standard errors and biased parameter estimates. Using data from the nationally representative Head Start Family and Child…

The determinants of sentencing are of much interest in criminal justice and legal research. Understanding the determinants of sentencing decisions is important for ensuring transparent, consistent, and justifiable sentencing practice that adheres to the goals of sentencing, such as the punishment, rehabilitation, deterrence, and incapacitation of…

Web-based surveys have become more prevalent in areas such as evaluation, research, and marketing research to name a few. The proliferation of these online surveys raises the question, how do response rates compare with traditional surveys and at what cost? This research explored response rates and costs for Web-based surveys, paper surveys, and…

In recent years researchers have paid substantial attention to the issue of college students' alcohol use. One limitation to the current literature is an over reliance on retrospective, self-report survey data. This article presents field methodologies for measuring college students' alcohol consumption in natural drinking environments.…

This paper discusses conceptual and methodological issues surrounding the measurement of the construct, degree of implementation, drawn from a review of 74 previous studies which included such measures. Assessments against five measurement criteria indicate that an adequate scientific bias has not been established for this construct. (Author/PN)

Discusses potential sources of self-selection bias in quasi-experimental evaluations of whole-school reform and considers how individual school-level data might be used to provide valid impact estimates. Develops an instrumental variable strategy that can be used to improve on common value-added estimators when only post-treatment measures are…

Introduces an new approach for measuring the impact of whole school reforms. The approach, based on "short" interrupted time-series analysis, is explained, its statistical procedures are outlined, and how it was used in the evaluation of a major whole-school reform, Accelerated Schools is described (H. Bloom and others, 2001). (SLD)

A group of defenses of qualitative evaluation methods is examined, based on a hard relativistic interpretation of the work of Thomas Kuhn. A promising defense of qualitative evaluation may be found in a soft-relativist interpretation of Kuhn's analysis of the nature of scientific discovery. (Author/LMO)

The normative implications of applying significance tests in evaluation research are examined. The authors conclude that evaluators often make normative decisions, based on the traditional .05 significance level in studies with small samples. Additional reporting of the magnitude of impact, the significance level, and the power of the test is…

The authors demonstrate how exponential smoothing can play a role in the identification of the intervention component of an interrupted time-series design model that is analogous to the role that the sample autocorrelation and partial autocorrelation functions serve in the identification of the noise portion of such a model. (Author/BW)

It is common practice to associate recommendations for program improvement with evaluation reports. In this article, several classes of recommendations are identified, and aspects of one class are explored in some detail. Consideration is given to certain logical, methodological, and practical issues relating to the derivation of recommendations…

Properties of three multidimensional measures of occupational segregation by race and gender are compared. The generalization of the Gini index (D. Boisso and others, 1994) is shown to be flawed, and calculations with six groups of U.S. Department of Labor data show that the generalization of the index of T. Karmel and M. Maclachlan by J. Silber…

Presents conversion factors to be used to translate the quality of the respective contraception methods distributed to a single measure of protection for calculating couple-years of protection in family planning studies. Discusses the implications for the evaluation of family planning programs. (SLD)

This article presents an overview, description, and assessment of the Associative Group Analysis method for studying conceptual development and changes in belief systems. Three specific applications to evaluation research tasks are used to illustrate its potential. Uses in areas of cross-cultural issues, beliefs, integration, and adaptation are…