Most existing solutions to the current replication crisis in science address only the factors stemming from specific poor research practices. We introduce a novel mechanism that leverages the experts' predictive abilities to analyze the root causes of replication failures. It is backed by the principle that the most accurate predictor is the most…

Performance management in university-based scientific research institutions is essential for driving reform, advancing education quality, and fostering innovation. However, current performance evaluation models often focus solely on research indicators, neglecting the critical interdependence between the education and research systems. This…

Homelessness is a serious and growing issue. Evaluations of systemic-level changes are needed to determine progress in reducing or ending homelessness. The report card methodology is one means of systems-level assessment. Rather than solely establishing an enumeration, homelessness report cards can capture pertinent information about structural…

The determinants of sentencing are of much interest in criminal justice and legal research. Understanding the determinants of sentencing decisions is important for ensuring transparent, consistent, and justifiable sentencing practice that adheres to the goals of sentencing, such as the punishment, rehabilitation, deterrence, and incapacitation of…

This study explores the possibility that any drug prevention program might be considered "evidence-based" given the use of data analysis procedures that optimize the chance of producing statistically significant results by reanalyzing data from a Drug Abuse Resistance Education (DARE) program evaluation. The analysis produced a number of…

Proposes group mean centering as a solution to the problem of pooling data from multiple sites. Uses California's Work Pays Demonstration Project to illustrate three analysis strategies for data from multiple sites. Discusses considerations in choosing the centering strategy. (SLD)

This article explores the statistical methodologies used in demonstration and effectiveness studies when the treatments are applied across multiple settings. The importance of evaluating and how to evaluate these types of studies are discussed. As an alternative to standard methodology, the authors of this article offer an empirical binomial…

Examined six published papers using a theory-based evaluation (TBE) approach. Found that authors of the papers did not always make explicit the relation of their data to the theory of the program. Identified some ways in which TBE contributed to the merit of the evaluation. Outlines lessons for the future of TBE. (SLD)

This article evaluates three general types of computer-assisted approaches to multicriteria decision problems in which criteria are attributes as opposed to objectives. Several programs specifically designed for multiattribute problems, as well as spreadsheets and decision-free software, are discussed. (Author/BS)

Studied the analytic problems associated with a design in which one identifiable group is allocated to each treatment condition and members of these groups are measured to assess the intervention. Results from a simulation study underscore the analytic problems associated with these quasi-experimental or group-randomized designs. (SLD)

When procedures for developing recommendations are integrated into evaluation methods, it is easier to produce useful recommendations for program improvement. (RB)

A recent Delphi study on recreational boating safety priorities of 94 safety experts is used to illustrate the utility of applying the modified scree test to Delphi data. The scree test does not oversimplify decisions and can be applied to fairly complicated data. (SLD)

Four aspects of the relationship between auditing and evaluation in their approaches to program assessment are examined: (1) their different origins; (2) the definitions and purposes of both, and the questions they seek to answer; (3) contrasting viewpoints and emphases of auditors and evaluators; and (4) commonalities of interest and potential…

The authors demonstrate how exponential smoothing can play a role in the identification of the intervention component of an interrupted time-series design model that is analogous to the role that the sample autocorrelation and partial autocorrelation functions serve in the identification of the noise portion of such a model. (Author/BW)

The methodology and design of the process evaluation system of the Pawtucket Heart Health Program (an 11-year project for changing/documenting change in cardiovascular disease risk status of a northeastern city's residents) are described. Examples of the system's use are provided. The maintenance of such a system without substantial evaluation…