Canada does not have a national system wide assessment of the socioeconomic impacts of academic research. We do not have a Research Excellence Framework such as in the United Kingdom. Yet Canadian researchers, funders and institutions are interested in research impact, particularly the methods and processes for generating impacts to complement…

With the opportunities and challenges stemming from the artificial intelligence developments and its integration into society, AI literacy becomes a key concern. Utilizing quality AI literacy instruments is crucial for understanding and promoting AI literacy development. This systematic review assessed the quality of AI literacy scales using the…

Peer review is integral to the evaluation of grant proposals. Reviewer perceptions and characteristics have received limited study, especially their associations with reviewers' evaluations. This mixed methods study analyzed the unstructured comments of 270 experienced peer reviewers after they scored proposals based on mock overall evaluations…

In the last three decades, the promotion of stakeholder involvement in evaluation has been gaining relevance in the Latin American and internationally, across varied agencies, institutions, and civic organizations. The 2030 Agenda and the Global Evaluation Agenda have also recognized the centrality of participation in evaluation. This article…

Debates about appropriate, fair and effective ways of assessing research and researchers have raged through the scientific community for decades, recently mostly concerned with discussing the merits and limitations of metric-based, quantitative assessments versus peer review-based, qualitative alternatives. Ample attention has been paid to formal…

Logic modeling, the process that explicates how programs are constructed and theorized to bring about change, is considered to be standard evaluation practice. However, logic modeling is often experienced as a transactional, jargon-laden, discrete task undertaken to produce a document to comply with the expectations of an external entity, the…

Globally, the appraisal of educators remains a contested issue as the process, which aims to provide support and development to educators, is generally considered a punitive measure for educators. Within the South African context, the Integrated Quality Management System is used to evaluate educators. Drawing on Fraser's theories of social…

Most program evaluation efforts concentrate on assessments of "program implementation" and "program outcomes." However, another area of programs that has not received sufficient attention in the literature is evaluating the plan of the program. Since the quality of the plan and planning process can influence program…

There is increasing interest in using factor scores in structural equation models and there have been numerous methodological papers on the topic. Nevertheless, sum scores, which are computed from adding up item responses, continue to be ubiquitous in practice. It is therefore important to compare simulation results involving factor scores to…

This paper is concerned with the process of selecting between the increasingly popular bi-factor model and the second-order factor model in measurement research. It is indicated that in certain settings widely used in empirical studies, the second-order model is nested in the bi-factor model and obtained from the latter after imposing appropriate…

The Research Excellence Framework (REF) follows a set of submission rules. Here we analyse whether the submission rules of the impact element arguably shaped the submission sizes of the submitting units -- a group of academics researching on a specific subject area in higher education providers -- to the REF in 2014. The number of impact case…

A critical revisiting of research methods with multilingual learners raises issues of validity that are often unaddressed in current discussions of research quality. Though validity is conceptualized and assessed differently across research traditions, validity broadly describes the production of findings that are credible and trustworthy. The…

Impact assessment research has developed theory-based approaches to trace the societal impact of scientific research. Impact assessment typically starts from the perspective of a research investment, organization, or project. Research users, non-academic actors involved in knowledge production, translation, and application, are well represented in…

The usual interpretation of the person and task variables in between-persons measurement models such as item response theory (IRT) is as attributes of persons and tasks, respectively. They can be viewed instead as ensemble descriptors of patterns of interactions among persons and situations that arise from sociocognitive complex adaptive system…

Causal evaluations in teacher education are rare. Underlying reasons include a lack of clearly defined treatments, a lack of research designs that can work in the context of teacher education programs, and a lack of resources for enacting these designs. This article provides a framework for how to fill these gaps. We first propose an approach to…