The Campbellian validity typology has been used as a foundation for outcome evaluation and for developing evidence-based interventions for decades. As such, randomized control trials were preferred for outcome evaluation. However, some evaluators disagree with the validity typology's argument that randomized controlled trials as the best design…

A valuable extension of the single-rating regression discontinuity design (RDD) is a multiple-rating RDD (MRRDD). To date, four main methods have been used to estimate average treatment effects at the multiple treatment frontiers of an MRRDD: the "surface" method, the "frontier" method, the "binding-score" method, and…

Assessments in response formats with ordered categories are ubiquitous in the social and health sciences. Although the assumption that the ordering of the categories is working as intended is central to any interpretation that arises from such assessments, testing that this assumption is valid is not standard in psychometrics. This is surprising…

The motivation for this paper is the authors' recent work on several randomized control trials in which they found the primary result, which averaged across subgroups or sites, to be moderated by demographic or site characteristics. They are led to examine a distinction that the Institute of Education Sciences (IES) makes between "confirmatory"…

Concomitant with the rise of rationalizing accountability in higher education has been an increase in theoretical reflection about the forms accountability has taken and the ones it should take. The literature is now peppered by a wide array of distinctions (e.g. internal/external, inward/outward, vertical/horizontal, upward/downward,…

The purpose of this study was to identify the decision-making styles of college students for sports apparel. Modifying the Consumer Style Inventory (CSI) developed by Sproles and Kendall (1986), a quantitative research design was developed to analyze the effect of seven factors. Eight-hundred and twenty-two subjects composed of 376 male and 446…

The following four integrative data analysis strategies for mixed-method evaluation designs are derived from and illustrated by empirical practice: (1) data transformation; (2) typology development; (3) extreme case analysis; and (4) data consolidation and merging. Use of these methods to realize the full potential of mixed-method approaches is…

Evaluation research is conceived as a question answering process. Four strategies are suggested which emerged from four approaches to the study of questions. Evaluation questions must be identified and classified before research can effectively begin. This paper discusses the four approaches to studying evaluation research questions and their…

This paper shares the findings of an inquiry that evaluated 50 survey articles published in a refereed journal, "Educational Administration Quarterly," by examining the 53 sample designs reported in the articles. The paper presents a typology of the sample designs identified, discusses the problems of sample selection and estimation…

This paper posits a taxonomy for categorizing issues that arise in the evaluation of Intelligent Tutoring Systems (ITSs). The taxonomy has three dimensions: Life Cycle of Evaluation, Research Issues, and Methodological Issues. The Life Cycle dimension has four levels: pre-experimental, laboratory study, field study, and initial operational test…

There is a need for clarity regarding different research strategies in order to assess the relative contribution of each. Using the framework of the Facet Design Technique (FDT) for problem structuring, eight general research strategies are identified as they relate to measurement fidelity, representativeness of units, and treatment…

Reviewed research on practicum supervision in terms of findings and research designs employed. Classified research according to counselor-supervisor similarity, training methods, and evaluation. Conclusions indicated no support for supervisor-counselor matching, support for experiential, modeling and didatic methods, and peer supervision. (Author)

Current methods and instruments for assessing college student academic outcomes are identified and described, and possible outcome measures of NCRIPTAL's (National Center for Research to Improve Postsecondary Teaching and Learning) are suggested. Section I defines college outcomes from several perspectives, including pressures for outcome…

Describes a performance-based assessment approach utilizing science activities and questions to assess different levels of thinking. Objectives, materials, and procedures are provided for the assessment of student skills in classification, observation and inference, interpretation, and analysis. (PR)

This paper offers concrete examples from recent policy research about how qualitative and quantitative methods can be combined to better address research questions. Using a conceptual framework developed by Rossman and Wilson (1985), later expanded by Greene, Caracelli, and Graham (1989), the paper discusses how the design and analysis phases of…