The use of concept mapping in research and evaluation has expanded dramatically over the past 20 years. Researchers in academic, organizational, and community-based settings have applied concept mapping successfully without the benefit of systematic analyses across studies to identify the features of a methodologically sound study. Quantitative…

Program theory evaluation, which has grown in use over the past 10 years, assesses whether a program is designed in such a way that it can achieve its intended outcomes. This article describes a particular type of program theory evaluation--logic analysis--that allows us to test the plausibility of a program's theory using scientific knowledge.…

This paper presents a mixed-method approach to measuring the functioning of Safe Schools/Healthy Students (SS/HS) Initiative partnerships. The SS/HS national evaluation team developed a survey to collect partners' perceptions of functioning within SS/HS partnerships. Average partnership functioning scores were used to rank each site from lowest to…

Illicit drug use remains at high levels in the U.S. The federal Office of National Drug Control Policy evaluates the outcomes of national drug demand reduction policies by assessing annual changes in drug use from several federally sponsored annual national surveys. Such survey methods, relying exclusively on drug use as self-reported on…

The goal of this article is to contribute to the validation of a self-evaluation method, which can be used by schools to evaluate the quality of their Competence Assessment Program (CAP). The outcomes of the self-evaluations of two schools are systematically compared: a novice school with little experience in competence-based education and…

A theory-driven approach to validity is proposed. The central argument is that a model or theory should be formulated in a program evaluation and the modeling process should include the identification of potential threats to validity in research. (Author/LMO)

Several recent studies are identified that used expert ratings of the job performance of certified professionals as a validation criterion. These studies showed a strong relationship between expert ratings and certification, but were susceptible to alternative interpretations due to design constraints. Inclusion of control constructs is presented…

Evaluators often make key decisions about what content to include when designing new scales. However, without clear conceptual grounding, there is a risk these decisions may compromise the scale's validity. Techniques such as concept mapping are available to evaluators for the specification of conceptual frameworks, but have not been used as a…

This paper reports on the psychometric properties of a multidimensional inventory battery used for evaluating the parenting behavior of 693 parents enrolled in family support programs. Parents, depression, knowledge of child development, attitudes toward child rearing, and available social support were studied. Inventory can be administered with…

Agrees with the power of randomized clinical trials, but makes the case that random assignment raises other, unanticipated threats to internal validity as a result of failing to consider treatment preference in research participant behavior. Reviews benefits and limitations of research strategies that augment randomization while respecting…

The Combined Modified Design is presented to overcome problems of internal and external validity in the conduct of true experiments in longitudinal studies. It combines a modified version of the Randomized Invitation Design with the lottery longitudinal experimental field trial situation. (SLD)

Results of empirical validation studies of six statistical models available for indirectly estimating the need for alcohol, drug abuse, and mental health services across a large geographic area are presented. Models were evaluated using Colorado Social Health Survey data for 4,745 adults from 48 subareas of Colorado. (SLD)

A discussion of current issues in evaluation, related to society's expectation that evaluation is based on scientific authority, is followed by an exploration of concepts of "validity" and "values." The role of the evaluator is increasingly involved in issues of ethics and social justice. (SLD)

Reports on the use of the tailored design method (TDM) to achieve high survey response in two separate studies of the dissemination of Treatment Improvement Protocols (TIPs). Findings from these two studies identify six factors may have influenced nonresponse, and show that use of TDM does not, in itself, guarantee a high response rate. (SLD)

A fuzzy set model is offered for interpreting vague frequency expressions, such as "rarely" and "sometimes." Results with 152 undergraduates reporting marijuana use reflect different frequency expressions for the same level of use and suggest that self-report validity may be enhanced by analyzing frequency expressions as fuzzy…