This paper investigates the methodological discrepancies underlying the measurement of adult learning and education (ALE) participation in the UK by focusing on four major surveys -- APiL, PIAAC, AES, and LFS. Grounded in the Total Survey Error (TSE) framework, we systematically examined the surveys' documentation and compared their definitions,…

RobotReviewer is a tool for automatically assessing the risk of bias in randomized controlled trials, but there is limited evidence of its reliability. We evaluated the agreement between RobotReviewer and humans regarding the risk of bias assessment based on 1955 randomized controlled trials. The risk of bias in these trials was assessed via two…

Although parallel analysis has been found to be an accurate method for determining the number of factors in many conditions with complete data, its application under missing data is limited. The existing literature recommends that, after using an appropriate multiple imputation method, researchers either apply parallel analysis to every imputed…

Outside of education, many academic disciplines developed and promulgated checklists to evaluate the quality of qualitative studies. This article used an embedded mixed methods approach in three stages: development of a conceptual framework reviewing 62 guidelines; a content analysis of key areas of the guidelines; and review of dissertations to…

The historical turn in social science has prompted scholars to engage with the work of historians on a large scale. Here, social scientists face two standard problems of selection bias: confirmation bias and convenience sampling. So far, the record of dealing with these problems has been poor, and little has been done to specify how social…

In this article, we examine whether cross-national studies disclose enough information for independent researchers to evaluate the validity and reliability of the findings (evaluation transparency) or to perform a direct replication (replicability transparency). The first contribution is theoretical. We develop a heuristic theoretical model…

Periodic changes in occupational classifications make it difficult to obtain consistent measures of social class over time, potentially jeopardizing research on class-based trends. The severity of this problem depends, in part, on the measurement strategies used to address those changes. The authors propose that when a sample has been coded partly…

Regression mixture models are a statistical approach used for estimating heterogeneity in effects. This study investigates the impact of sample size on regression mixture's ability to produce "stable" results. Monte Carlo simulations and analysis of resamples from an application data set were used to illustrate the types of problems that…

Transactional validity, a common approach in participatory research, is attained when preliminary analyses of research results are discussed with research participants and their feedback is incorporated in the analysis. Member checking is one way of achieving transactional validity, which has been heralded as a stronger version of validity reached…

Multilevel modeling (MLM) is a statistical technique for analyzing clustered data. Despite its long history, the technique and accompanying computer programs are rapidly evolving. Given the complexity of multilevel models, it is crucial for researchers to provide complete and transparent descriptions of the data, statistical analyses, and results.…

This study examines meta-analysis studies published in the field of educational sciences from 2010-2019, in journals indexed within the scope of the TÜBITAK ULAKBIM TR Directory. Of the 163 studies scanned, 26 meta-analysis studies that meet the inclusion criteria constitute the sample of this content analysis study. Within this research, a…

This article maps the territory of visual ethnography as a key and accessible research methodology in education. It aims to provide an overview and to present theory and practice for future research. The origins and principles of visual ethnography are disclosed as well as some methods to gather data. From the premise that either created by the…

Data analysis is a significant methodological component when conducting quantitative education studies. Guidelines for conducting data analyses in quantitative education studies are common but often underemphasize four important methodological components impacting the validity of inferences: quality of constructed measures, proper handling of…

This report presents a review of the statistical practices of 30 journals representative of the second language field. A review of 150 articles showed a number of prevalent statistical violations including incomplete reporting of reliability, validity, non-significant results, effect sizes, and assumption checks as well as making inferences from…

Purpose of the Study: Missing data are a common problem encountered while implementing measurement instruments. Yet the extent to which reliability, validity, average discrimination and difficulty of the test results are affected by the missing data has not been studied much. Since it is inevitable that missing data have an impact on the…