In the 2018 meta-analysis of "Educational Psychology Review" entitled "Null effects of perceptual disfluency on learning outcomes in a text-based educational context" by Xie, Zhou, and Liu, we identify some errors and inconsistencies in both the methodological approach and the reported results regarding coding and effect sizes.…

Purpose of the Study: Missing data are a common problem encountered while implementing measurement instruments. Yet the extent to which reliability, validity, average discrimination and difficulty of the test results are affected by the missing data has not been studied much. Since it is inevitable that missing data have an impact on the…

Practitioners and policymakers rely on meta-analyses to inform decision making around the allocation of resources to individuals and organizations. It is therefore paramount to consider the validity of these results. A well-documented threat to the validity of research synthesis results is the presence of publication bias, a phenomenon where…

This methodological brief introduces basic procedures and issues for conducting a high-quality meta-analysis in gifted education. Specifically, we discuss issues such as how to select a topic and formulate research problems, search for and identify qualified studies, code studies and extract data, choose and calculate effect sizes, analyze data,…

Researching writing-related dispositions is of critical concern for understanding writing transfer and writing development. However, as a field we need better tools and methods for identifying, tracking, and analyzing dispositions. This article describes a failed attempt to code for five key dispositions (attribution, self-efficacy, persistence,…

Considering scoring validity as encompassing both reliable rating scale use and valid descriptor interpretation, this study reports on the validation of a CEFR-based scale that was co-constructed and used by novice raters. The research questions this paper wishes to answer are (a) whether it is possible to construct a CEFR-based rating scale with…

This commentary reflects on the contributions to learning analytics and theory by a paper that describes how multiple theoretical frameworks were woven together to inform the creation of a new, automated discourse analysis tool. The commentary highlights the contributions of the original paper, provides some alternative approaches, and touches on…

Coding interactional data for statistical analysis presents theoretical, methodological and practical challenges. Coding schemes rely on categories that are decided by their relevance to the analytical problem under investigation. We suggest that (1) endogenous and publicly displayed concerns of participants provide for the observable organisation…

This paper is a report of the replication of a seminal study on cognitive presence in computer mediated conferencing (CMC) by Garrison, Anderson, and Archer (2001). A comparison of cognitive presence coding by three different researchers is also demonstrated. The study re-ignites debates about what constitutes the segment of CMC data to be coded…

Presents general principles for cleaning data sets. Suggests techniques: check for column shifts, data completeness, marginal frequencies, and selected cross-tabulations; examines a sample of cases. Gives strategies for preventing data processing errors. (SK)

Primary research on student ratings of instruction is reviewed, and questions about the validity of such ratings as measures of teacher effectiveness are explored using nomological coding of 43 validity studies to examine causes of variability in coefficients of validity. Deficiencies of prior analyses and research needs are identified. (SLD)