This article examines the concept of credible evidence in Extension evaluations with specific attention to the measures and measurement strategies used to collect and create data. Credibility depends on multiple factors, including data quality and methodological rigor, characteristics of the stakeholder audience, stakeholder beliefs about the…

Understanding the reason(s) for Differential Item Functioning (DIF) in the context of measurement is difficult. Although identifying potential DIF items is typically a statistical endeavor, understanding the reasons for DIF (and item repair or replacement) might require investigations that can be informed by qualitative work. Such work is…

In this article I advocate for a new discussion in the field of international large-scale assessments; one that calls for a reexamination of international large-scale assessments (ILSAs) and their use. Expanding on the high-quality work in this special issue I focus on three inherent limitations to international large-scale assessments noted by…

Meta-assessment, or the assessment of assessment, can provide meaningful information about the trustworthiness of an academic program's assessment results (Bresciani, Gardner, & Hickmott, 2009; Palomba & Banta, 1999; Suskie, 2009). Many institutions conduct meta-assessments for their academic programs (Fulcher, Swain, & Orem, 2012),…

Science educators have long been concerned with how formal schooling contributes to learners' capacities to engage with science after school. This article frames productive engagement as fundamentally about the coordination of claims with evidence, but such coordination requires a number of reasoning capabilities to evaluate the strength of…

The current review addresses the validity of published meta-analyses in education that determines the credibility and generalizability of study findings using a total of 56 meta-analyses published in education in the 2000s. Our objectives were to evaluate the current meta-analytic practices in education, identify methodological strengths and…

This research describes, classifies, and comparatively evaluates national models and mechanisms used to evaluate research and allocate research funding in 16 countries. Although these models and mechanisms vary widely in terms of how research is evaluated and financed, nearly all share the common characteristic of relating funding to some measure…

Stresses two essential characteristics that principals must keep in mind when constructing measures that yield accurate and relevant evaluation data. When constructing quantitative measures, validity and reliability are most important considerations. When designing qualitative measures, credibility and dependability are most important. (14…

The relevance of ethnographic research to evaluation of educational programs is discussed. The author focuses on evaluation questions that deal with finding out what is happening in a program and if what is happening is desirable. Ethnography suits the purposes of evaluators who answer such questions because they want to know how something works…

Selected aspects of the problem of obtaining unbiased program or product evaluation are discussed. An evaluator who is a member of the project staff will have difficulty producing an evaluation which is credible and valid. Project monitors will also have a problem since they are often required to assume the conflicting roles of external evaluator…

Evaluation is viewed as essential to decision making and social policy development. Since conventional methods have been disappointing or inadequate, naturalistic inquiry (N/I) differs from conventional science in minimizing constraints on antecedent conditions (controls) and on output (dependent variables). N/I is phenomenological rather than…

This paper describes six existing evaluator-auditor working formats and the conditions which foster credibility of evaluation findings. Evaluators were classified as: (1) member of project developmental team, accountable to project director; (2) independent internal evaluator, accountable to system in general but not to project directors, and (3)…

New methods are needed to judge the quality of alternative student assessment, methods which complement the philosophy underlying authentic assessments. This paper examines assumptions underlying validity, reliability, and objectivity, and why they are not matched to authentic assessment, concentrating on the constructivist paradigm of E. Guba and…

This paper presents criteria for establishing the trustworthiness of naturalistic inquiries, and specific techniques to facilitate their achievement or determine the degree of their achievement. The following criteria are briefly described: fairness; and ontological, educative, catalytic and tactical authenticity. Explored in greater detail,…

Definitions and short explanations of key concepts in the field of instructional evaluation are presented. Evaluation is defined as the process of determining the value of programs, projects, materials, and personnel. It is distinguished from research in that the aims of research are less time- and situation-specific, attempting to uncover…