Institutional performance benchmarking requires identifying a set of reference or comparator institutions. This paper describes a method by which an institution can identify other institutions that are most similar to itself using a methodology that identifies the nearest institutional neighbors based on a balanced set of metrics accessed from…

The popularity of sample surveys in evaluation and research makes it necessary for consumers to tell a good survey from a poor one. Several sources were identified that gave advice on how to evaluate a sample design used in a survey study. The sources are either too limited or too extensive to be useful practically. The purpose of this paper is to…

Researchers frequently deal with data that are cyclical in nature. This paper presents a methodological procedure that can be used to determine if a roller-coaster effect exists in repeated measures data. The procedure incorporates three key components. First, the researcher may find it necessary to analyze each participant's data set separately…

This paper helps researchers develop criteria for conducting surveys. Cost considerations frame the paper's suggestions. Specifically, the text advises that researchers consider the human and material resources needed, type of survey instrument, medium for conducting the survey (mail, telephone, personal interview, etc.), data analysis…

The terms "rigor" and "relevance" most often surface in discussions of methodological adequacy. Assessing epistemological relevance is equivalent to answering the question, "Is this particular research question worth asking at all?" Epistemological rigor refers to the properties of a "researchable" problem.…

A conceptualization of generalization and its place in educational research is proposed. Problems with existing approaches to generalization include limited concepts, misleading or simplistic results, perception variance or change, and the discouragement of sensitivity due to convenience. Researchers should distinguish between conclusions which…

Two relatively new tools for analysis of data compiled in evaluation studies are presented. The National Test-Equating Study in Reading, known as the Anchor Test Study, produced tables of score-correspondence between the eight reading comprehension and vocabulary tests most widely used in the United States. Two types of tables from this report…

Researchers increasingly recognize that significance tests are limited in their ability to inform scientific practice. Common errors in interpreting significance tests and three strategies for augmenting the interpretation of significance test results are illustrated. The first strategy for augmenting the interpretation of significance tests…

Noting that in Europe there is a growing criticism of the largely American communication research tradition that focuses on individuals, this paper examines European-style "critical analysis," which looks at social systems. The paper first examines the criticism of American-style "administrative" research, and then discusses…

The paper reviews the state of the art in evaluation research of direct instruction projects with handicapped students. The first section delineates research designs used (including norm referenced comparisons, nonequivalent comparison groups, and quasi experimental designs); major findings; and problems with each. A distinction is made between…

Third stream evaluation, the fusing of the ecological perspective with experimental or quasi-experimental evaluation design, is described. The ecological perspective necessitates that the conceptualization and analysis of a setting and the design of the study emphasize the interdependent relations among organisms, behavior and environment in…

A research design is described for the situation in which a program, particularly a compensatory education program funded by Chapter 1 of the Hawkins Stafford Act of 1988, can be evaluated when there is no available comparison group and no pretest data. The design requires content specialists to identify which objectives on the posttest were…

The one group posttest only evaluation model has been identified as a relatively inexpensive and useful model that can identify program components that are not being successful. The use of the model is discussed and illustrated through a hypothetical evaluation of a compensatory (Chapter 1) program. The one group posttest only evaluation model…

As the procedures of evaluation research have developed during recent years, it has become clear that the results of evaluation studies have failed to provide policy decision makers with useful information. An approach towards designing and implementing more useful evaluations--the stakeholder survey--is discussed. Stakeholders are individuals, or…

This presentation outlines a perspective on job analysis and comments on research reported at a symposium with reference to that perspective. The perspective discusses two questions pertaining to job analysis that are referred to as the Stage I and Stage II evaluation operations: What should be done? and How well was it done? Stage I evaluation is…