This paper looks at the tension between the desire to claim predictive ability for Agent-Based Models (ABMs) and its extreme difficulty for social and ecological systems, suggesting that this is the main cause for the continuance of a rhetoric of prediction that is at odds with what is achievable. Following others, it recommends that it is better…

Researchers continue to choose PCA in scale development and adaptation studies because it is the default setting and overestimates measurement quality. When PCA is utilized in investigations, the explained variance and factor loadings can be exaggerated. PCA, in contrast to the models given in the literature, should be investigated in…

Once research questions are posed, researchers must answer many a priori questions regarding research design before analysis can be performed and any conclusions can be made, including sample selection criteria, data collection method, model specification, analysis and estimation technique. The choices made by researchers along this forking path…

This study analyzed studies done in Turkey in the context of curriculum evaluation (CE) by asking, "How is it made? The study was carried out in two stages. In the first stage, the document analysis method used 215 theses written between 1991 and 2020 on CE were analyzed according to the "thesis review form." In the second stage,…

This article argues that Agent-Based Modelling, owing to its capabilities and methodology, has a distinctive contribution to make to delivering coherent social science prediction. The argument has four parts. The first identifies key elements of social science prediction induced from real research across disciplines, thus avoiding a straw person…

Despite the rising popularity of big data, there is speculation that evaluators have been slow adopters of these new statistical approaches. Several possible reasons have been offered for why this is the case: ethical concerns, institutional capacity, and evaluator capacity and values. In this method note, we address one of these barriers and aim…

In science in general and in the context of single-case experimental designs, replication of the effects of the intervention within and/or across participants or experiments is crucial for establishing causality and for assessing the generality of the intervention effect. Specific developments and proposals for assessing whether an effect has been…

Conventional advice discourages controlling for postoutcome variables in regression analysis. By contrast, we show that controlling for commonly available postoutcome (i.e., future) values of the treatment variable can help detect, reduce, and even remove omitted variable bias (unobserved confounding). The premise is that the same unobserved…

For more than a century, numerous studies examined and reported how principals allocated their time. Assuming that differences in selection, participation and satisfaction of tasks by principals resulted in different outcomes, researchers have continued to study principal time use. In 1989, Wimpelberg, Teddlie, and Stringfield indicated a need for…

The evidence continuum is a five-domain model for building evidence through needs assessment, program theory, process evaluation, outcomes and impact evaluation, and optimization studies. In this conceptual article, the first two domains of the evidence continuum are used to design a learner centered course on advanced research methods, and…

In educational contexts, students often self-select into specific interventions (e.g., courses, majors, extracurricular programming). When students self-select into an intervention, systematic group differences may impact the validity of inferences made regarding the effect of the intervention. Propensity score methods are commonly used to reduce…

Education decisionmakers need access to the best evidence about the effectiveness of education interventions, including practices, products, programs, and policies. It can be difficult, time consuming, and costly to access and draw conclusions from relevant studies about the effectiveness of interventions. The What Works Clearinghouse (WWC)…

Researchers strategically assess youth mental health by soliciting reports from multiple informants. Typically, these informants (e.g., parents, teachers, youth themselves) vary in the social contexts where they observe youth. Decades of research reveal that the most common data conditions produced with this approach consist of discrepancies…

Several programs of research have sought to assess the replicability of scientific findings in different fields, including economics and psychology. These programs attempt to replicate several findings and use the results to say something about large-scale patterns of replicability in a field. However, little work has been done to understand the…

The complexity associated with how interventions result--or fail to result--in outcomes and how context matters is increasingly recognised. Logic models provide an important tool for handling complexity, with contrasting uses in programme evaluation and evidence synthesis. To reconcile these, we developed an approach that combines the strengths of…