This article describes three challenges encountered during a developmental evaluation and explains how these were addressed from the evaluators' perspective. The evaluation was conducted to support the implementation of a three-year educational technology leadership project funded by the Alberta provincial government. The developmental evaluation…

Since 2003, the Essential Skills Series training program developed by the Canadian Evaluation Society has been offered to more than 15 groups in the province of Quebec. The evaluations of these workshops were based on the participants' reactions collected by a Participant Feedback Questionnaire. This article describes the process used to assess…

An evaluability assessment (EA) of the 55 Alive program, a national older driver refresher course aimed at improving driving skills, was conducted. This EA adds to the evaluation literature as previous outcome evaluations neglected to explore whether this program was prepared for such assessments. A mixed-method protocol was executed across three…

This article presents a creative and practical process for dealing with the problem of selection bias. Taking an algorithmic approach and capitalizing on the known treatment-associated variance in the X matrix, we propose a data transformation that allows estimating unbiased treatment effects. The approach does not call for modelling the data,…

Evaluators know they are supposed to identify and engage with stakeholders. What happens when the client has a very narrow concept of the meaning of evaluation? What happens when the primary stakeholders, including the client, disappear just as the evaluation gets started? First, it is important to acknowledge the challenge, then develop a…

This article describes an evaluation that was judged to be unsuccessful from the point of view of key program stakeholders. This was due to the fact that the evaluation did not support program advocates who had much to gain from positive evaluation findings. We argue that, although the knowledge needs of stakeholders must be taken into account,…

In the domain of school-based tobacco control, there is a large gap between the research being performed and the type of research needed to inform school-based prevention activities. The tobacco module of the school health action planning and evaluation system is a data collection and feedback system designed to support school-based tobacco…

The clash between Western and Indigenous ways of knowing has been epitomized by the "parachuting model" of the Western researcher who drops onto the reservation, collects data, and leaves, never to be heard from again. The strengths of indigenous science, for example, observation and contextual factors, are either ignored or…

Professional identity is a precondition for the establishment of a profession. This examines the professional identity of program evaluators in Israel. The field of evaluation in Israel has developed differently than in most Western countries--from the ground and with minimal governmental interference--and thus it is an interesting case study. In…

While we may all agree in principle that both implementers and evaluators should be culturally sensitive and ethical as well as instrumentally effective in their work practices, we often ignore the extent to which these practice goals may conflict with one another in achieving bureaucratic competence, particularly in a multicultural society.…

Given today's climate of economic uncertainty and fiscal restraint, organizations providing child and youth mental health services are required to do so with limited resources. Within this context, service providers face added pressure to deliver evidence-based programs and demonstrate program effectiveness. The Ontario Centre of Excellence for…

Rapidly declining response rates and the associated threat of nonresponse bias call into question the validity of data obtained through telephone surveys, a tool often used in evaluation. This article explores changes in nonresponse bias over time by examining three data points (1991, 1996, and 2002) from an annual household telephone survey…

Presented are the results of a study using latent profile analysis to describe the self-reported practices of 138 evaluators. Four classes emerged and were labeled (a) indistinct pattern of practice, (b) method-focused, (c) user-focused, and (d) robust pattern of practice. Evaluators in the "indistinct pattern of practice" class had mean item…

Examines the theoretical foundations of the belief that stakeholder involvement empowers the stakeholders and increases evaluation utilization and the validity of the evaluation. Discusses conditions under which these benefits are realized and situations in which they are not. (SLD)

Studied the convergence of evaluator and non-evaluator perspectives about collaborative evaluation. Results based on work of 67 pairs of evaluators and program practitioners show that, generally, program practitioners were more conservative in their opinions. Many differences in the two perspectives are noted. (SLD)