Debates about appropriate, fair and effective ways of assessing research and researchers have raged through the scientific community for decades, recently mostly concerned with discussing the merits and limitations of metric-based, quantitative assessments versus peer review-based, qualitative alternatives. Ample attention has been paid to formal…

Current approaches used in educational research and practice to evaluate the quality of written arguments often rely on structural analysis. In such assessments, credit is awarded for the presence of structural elements of an argument, such as claims, evidence, and rebuttals. In this article, we discuss limitations of such approaches, including…

The Sullivan Report (The Sullivan Commission on Diversity in the Healthcare Workforce, 2004) identified the lack of racial diversity among health care practitioners as a major issue. This report points out that, while the U.S. population has continued to become more diverse, the composition of health care workers has not kept pace with these…

This paper describes, and lays out an argument for, the use of a procedure to help groups of reviewers to judge the quality of prior research reports. It argues why such a procedure is needed, and how other existing approaches are only relevant to some kinds of research, meaning that a review or synthesis cannot successfully combine quality…

In this article we attempt to reconstruct the tacit and implicit notions of quality in the humanities. This reconstruction is based on a series of semi-structured qualitative interviews with 33 humanities scholars. Applying Max Weber's theory of authority, we argue the quality notions have two different sources--external and internal. External…

Critical appraisal of evidence is performed to assess its validity, trustworthiness and usefulness in evidence-based practice. There currently exists a large number and variety of critical appraisal tools (also named risk of bias tools and quality assessment instruments), which makes it challenging to identify and choose an appropriate tool to…

This paper presents a framework for utilizing Student Evaluations of Teaching (SET) in faculty evaluations. Recognizing the ongoing debate about the validity of SET as a measure of teaching effectiveness, the authors agree with scholars who propose viewing SET as a tool for gauging 'student perceptions of learning'. They present a method that…

Surveys are typical for student evaluation of teaching (SET). Survey research consistently confirms the negative impacts of careless responses on research validity, including low data quality and invalid research inferences. SET literature seldom addresses if careless responses are present and how to improve. To improve evaluation practices and…

Prospective students and their families use college rankings to navigate their higher education options. Rising tuition and fees have made the college decision more fraught. Recently, the major college ranking providers have revised their methodologies to reflect costs and other considerations. These revisions raise important questions about the…

In the years since the 2015 reauthorization of the Elementary and Secondary Education Act (ESEA) as the Every Student Succeeds Act (ESSA) placed a 1.0% cap for states on participation in the alternate assessment based on alternate academic achievement standards (AA-AAAS), states have worked hard to navigate the new restrictions on participation in…

We analysed the full text of all journal articles returned to the education subpanel of the 2021 Research Excellence Framework (REF2021). Using a latent Dirichlet allocation topic model, we identified 35 topics that collectively summarise the journal articles that research units, typically schools of education, selected for submission. We found…

Several AI-aided screening tools have emerged to tackle the ever-expanding body of literature. These tools employ active learning, where algorithms sort abstracts based on human feedback. However, researchers using these tools face a crucial dilemma: When should they stop screening without knowing the proportion of relevant studies? Although…

A key responsibility of higher education providers is the accurate certification of the knowledge and skills attained by their students. However, despite an intense focus on developing relevant quality assurance regulations, academic standards in higher education have remained resistant to explication and consistent application. In this paper, we…

Literature screening is the process of identifying all relevant records from a pool of candidate paper records in systematic review, meta-analysis, and other research synthesis tasks. This process is time consuming, expensive, and prone to human error. Screening prioritization methods attempt to help reviewers identify most relevant records while…

The assessment of workplace learning by educators at the workplace is a complex and inherently social process, as the workplace is a participatory learning environment. We therefore propose seeing assessment as a process of judgment embedded in a community of practice and to this purpose use the philosophy of inferentialism to unravel the judgment…