A recent issue of the Australian Journal of Language and Literacy included an article reporting on a systematic narrative review of the research literature that indicated that there was insufficient evidence to conclude whether genre theory and systemic functional linguistics either 'worked' or 'did not work'. The criteria used to evaluate these…

I. Purpose of the Study: Detecting whether interventions work or not (through main effect analysis) can provide empirical evidence regarding the causal linkage between malleable factors (e.g., interventions) and learner outcomes. In complement, moderation analyses help delineate for whom and under what conditions intervention effects are most…

Meta-analytic studies of second language (L2) learning typically employ a classic approach to meta-analysis. Although the classic approach can clarify findings, a multivariate, multilevel meta-analysis (3M) approach increases transparency by accounting for (a) dependencies in the evidence presented by primary studies, (b) methodological…

In 2020, the What Works Clearinghouse (WWC) for single-case experimental design (SCED) studies released an updated version of their standards and procedures handbook (Version 4.1). Because of these updates, there is a need to understand the implications for the field in terms of quality rating of the design, and subsequent synthesis of evidence at…

In fields (such as population health) where randomised trials are often lacking, systematic reviews (SRs) can harness diversity in study design, settings and populations to assess the evidence for a putative causal relationship. SRs may incorporate causal assessment approaches (CAAs), sometimes called 'causal reviews', but there is currently no…

When higher education students are assessed multiple times, teachers need to consider how these assessments can be combined into a single pass or fail decision. A common question that arises is whether students should be allowed to take a resit. Previous research has found little to no clear learning benefits of resits and therefore suggested they…

Higher education professionals often are tasked with providing evidence to stakeholders that programs, services, and practices implemented on their campuses contribute to student success. Furthermore, in the absence of a solid base of evidence related to effective practices, higher education researchers and practitioners are left questioning what…

Background/Context: Methodological background: Meta-analysis typically depends on the assumption that true effects follow the normal distribution. While assuming normality of effect "estimates" is often supported by a central limit theorem, normality for the distribution of interventions' "true" effects is a computational…

For research evidence to inform decision making, an appraisal needs to be made of whether the claims are justified and whether they are useful to the decisions being made. This chapter provides a high level framework of core issues relevant to appraising the "fitness for purpose" of evidence claims. The framework includes (I) the…

A randomized experiment that measures the impact of a social policy in a sample of the population reveals whether the policy will work on average with universal application. An experiment that includes only the subset of the population that volunteers for the intervention generates narrower "proof-of-concept" evidence of whether the…

The What Works Clearinghouse (WWC) identifies existing research on educational interventions, assesses the quality of the research, and summarizes and disseminates the evidence from studies that meet WWC standards. The WWC aims to provide enough information so educators can use the research to make informed decisions in their settings. This…

Background/Context: Some observational studies involve multiple layers of treatment selection, specifically in the context of the extended time accommodation (ETA) for English language learners (ELLs). In ETA settings, the first selection occurs due to the eligibility rule, where students whose ELL English proficiency is below a certain threshold…

This article, part of the "Learning Disability Quarterly" special series dedicated to single-case research design (SCRD), summarizes the review process and outlines how to provide high-quality SCRD study manuscript reviews. We offer recommendations for the entirety of the review process, including accepting the role of reviewer,…

Professional standards related to outcomes assessment call for student affairs professionals to use research to inform programming. If professionals are to rely on research to build programs that positively impact student learning outcomes, the research should be credible. We examined the quality of program effectiveness research available for…

This paper considers the adoption of computational techniques within research designs modeled after the extended case method. Echoing calls to augment the power of contemporary researchers through the adoption of computational text analysis methods, we offer a framework for thinking about how such techniques can be integrated into…