Students' evidence-based reasoning was examined across two studies. In Study 1, students were asked to evaluate newspaper excerpts including anecdotal, descriptive, correlational, and causal evidence provided in support of causal claims as well as to justify their quality ratings for two of these excerpts. In Study 2, students' justifications for…

Background: The use of research in policy settings is complex, unpredictable and influenced by a range of poorly understood social factors. This makes it difficult to plan for, facilitate and evaluate policy impacts arising from research. Aims and Objectives: 1. Propose and test tools for planning for and facilitating research impact, based on a…

Increasing the use of learning outcome assessments to inform educational decisions is a major challenge in higher education. For this study we used a sense-making theoretical perspective to guide an analysis of the relationship of information characteristics and faculty assessment knowledge and beliefs with the use of general education assessment…

Evaluation capacity building (ECB) is an essential element for generating credible and actionable evidence on Extension programs. This paper offers a discussion of ECB efforts in Cooperative Extension and how such efforts enable Extension professionals to collect and use credible and actionable evidence on the quality and impacts of programs.…

Advances in validity theory and alacrity in validation practice have suffered because the term "validity" has been used to refer to two incompatible concerns: (1) the degree of support for specified interpretations of test scores (i.e. intended score meaning) and (2) the degree of support for specified applications (i.e. intended test…

The conventional focus of validity in educational measurement has been on intended interpretations and uses of test scores. Empirical studies of test use by teachers, administrators and policy-makers show that actual interpretations and uses of test scores in context are invariably shaped by local users' questions, which frequently require…

Diagnostic classification models (DCMs) classify examinees based on the skills they have mastered given their test performance. This classification enables targeted feedback that can inform remedial instruction. Unfortunately, applications of DCMs have been criticized (e.g., no validity support). Generally, these evaluations have been brief and…

The expectation for accountability is legitimate. In order to have the desired effects, evidence of what students know and can do must respond to genuine institutional needs and priorities. Far too often, that condition is not met. On too many campuses, assessment activity is mired in a culture of compliance rather than driven by collective…

The landscape of medical education is continuously evolving, as are the needs of the learner. The appropriate use of research and evaluation is key when assessing the need for change and instituting one's innovative endeavours. This paper demonstrates how research seeks to generate new knowledge, whereas evaluation uses information acquired from…

A fundamental goal of student learning outcomes assessment in higher education is to use student learning evidence in decision making to improve educational programs. Such use of assessment findings, however, is atypical. This article argues that a narrow conception of use contributes to this conclusion and an accurate appraisal of the…

From a utilization-focused evaluation perspective, the success of an evaluation is rooted in the extent to which the evaluation was used by stakeholders. This paper details the "Expectations to Change" (E2C) process, an interactive, workshop-based method designed to engage primary users with their evaluation findings as a means of…

Systems of accountability are "second order" ways of using evidence of the standard to which "first order" tasks are carried out for a great variety of purposes. However, more accountability is not always better, and processes of holding to account can impose high costs without securing substantial benefits. At their worst,…

This chapter describes a concrete process that stakeholders can use to make predictions about the future performance of programs in local contexts. Within the field of evaluation, the discussion of validity as it relates to outcome evaluation seems to be focused largely on questions of internal validity (Did it work?) with less emphasis on…

Policy makers at the federal level have embraced an educator effectiveness agenda, which in turn has driven many states across the country to rapidly develop and implement new and more complex teacher evaluation systems. It is increasingly clear that the success of these nascent teacher evaluation systems partly depends on the will, skill, and…

Data-based decision making is all the rage. Secretary of Education Arne Duncan (2009) has emphatically declared, "I am a deep believer in the power of data to drive our decisions. Data gives us the roadmap to reform. It tells us where we are, where we need to go, and who is most at risk." In the past few years, all 50 states have adopted most or…