The reason for conducting the present case study was the problems the researchers encountered during data collection for another research project (Primary Study) entitled "The effects of virtual versus traditional flipped classes on EFL learners' grammar knowledge, self-regulation, and autonomy." Two online questionnaires were…

Background/Context: The COVID-19 pandemic challenged the way we conduct research. For some modes of data collection, such as interviews, there was a ready (if not perfect) analog: face-to-face became phone-based; paper and pen surveys moved online. Others, such as direct assessments of child development, proved more challenging. Despite the…

Background and context: The variables that researchers measure and how they measure them are central in any area of research, including computing education. Which research questions can be asked and how they are answered depends on measurement. Objective: To summarize the commonly used variables and measurements in computing education and to…

The measurement error in principal components extracted from a set of fallible measures is discussed and evaluated. It is shown that as long as one or more measures in a given set of observed variables contains error of measurement, so also does any principal component obtained from the set. The error variance in any principal component is shown…

The work presented in this case study results from a study conducted in 2012-2014 examining a newly created teacher evaluation system to determine the inter-rater reliability of the classroom observation instrument. The teacher evaluation system was the result of a partnership between the school district and the university in the same city…

This article presents the results of a systematic review of methods that have been used to measure or assess metacognition in children aged 4-16 years over a 20-year period (1992-2012). It includes an overview of the types of tool and methods used linked with the ages of the participants targeted and how metacognition and associated concepts are…

This study examines the proposed Reliability Generalization (RG) method for studying reliability. RG employs the application of meta-analytic techniques similar to those used in validity generalization studies to examine reliability coefficients. This study explains why RG does not provide a proper research method for the study of reliability,…

The pursuit of enhancing quality in tertiary education and educators is noble. Increasingly, however, universities are resorting to stark, reductionist representations of educational quality, such as decontextualised mean figures generated by student surveys, to measure and report on this. This paper questions the validity and reliability of such…

The authors have over the years tried to revise meta-analysis because it's basic premise is to add apples and oranges together and analyze. In other words, various data on the same subject are chosen using different samples, research strategies, and number properties. The findings are then homogenized and a statistical analysis is used (Snell, J.…

The Effective Classroom Practice project aimed to identify key factors that contribute to effective teaching in primary and secondary phases of schooling in different socioeconomic contexts. This article addresses the ways in which qualitative and quantitative approaches were combined within an integrated design to provide a comprehensive…

Reliability captures the influence of error on a measurement and, in the classical setting, is defined as one minus the ratio of the error variance to the total variance. Laenen, Alonso, and Molenberghs ("Psychometrika" 73:443-448, 2007) proposed an axiomatic definition of reliability and introduced the R[subscript T] coefficient, a measure of…

Meta-analytic reliability generalizations (RGs) are limited by the scarcity of reliability reporting in primary articles, and currently, RG investigators lack a method to quantify the impact of such nonreporting. This article introduces a stepwise procedure to address this challenge. First, the authors introduce a formula that allows researchers…

P. R. Killeen's (2005a) probability of replication ("p[subscript rep]") of an experimental result is the fiducial Bayesian predictive probability of finding a same-sign effect in a replication of an experiment. "p[subscript rep]" is now routinely reported in "Psychological Science" and has also begun to appear in…

We are currently at an exciting juncture in developing effective means for assessing so-called 21st-century skills in an innovative yet reliable fashion. One of these avenues leads through the world of "epistemic games" (Shaffer, 2006a), which are games designed to give learners the rich experience of professional practica within a discipline.…

"The Measures of Success" (MoS) project set out to develop, implement, and present the results of an evaluation model that could assess longer-term outcomes of workplace education programs for increasing literacy and Essential Skills. The model was tested at 18 work sites in Manitoba and Nova Scotia and included training programs that…