There is increasing interest in using factor scores in structural equation models and there have been numerous methodological papers on the topic. Nevertheless, sum scores, which are computed from adding up item responses, continue to be ubiquitous in practice. It is therefore important to compare simulation results involving factor scores to…

The population relationship between coefficient alpha and scale reliability is studied in the widely used setting of unidimensional multicomponent measuring instruments. It is demonstrated that for any set of component loadings on the common factor, regardless of the extent of their inequality, the discrepancy between alpha and reliability can be…

The purpose of this study is to review the status of differential item functioning (DIF) research in language testing, particularly as it relates to the investigation of sources (or causes) of DIF, which is a defining characteristic of the third generation DIF. This review included 110 DIF studies of language tests dated from 1985 to 2019. We…

The use of assessments that require rater judgment (i.e., rater-mediated assessments) has become increasingly popular in high-stakes language assessments worldwide. Using a systematic literature review, the purpose of this study is to identify and explore the dominant methods for evaluating rating quality within the context of research on…

The researchers of various disciplines often use qualitative and quantitative research methods and approaches for their studies. Some of these researchers like to be known as qualitative researchers; others like to be regarded as quantitative researchers. The researchers, thus, are sharply polarised; and they involve in a competition of pointing…

Replication is a foundation of the development of a knowledge base in an evidence-based field such as education. This study includes two direct replications of Hosp, Hensley, Huddle, and Ford which found evidence of criterion-related validity of curriculum-based measurement (CBM) for reading and mathematics with postsecondary students with…

Multisite trials, which are being used with increasing frequency in education and evaluation research, provide an exciting opportunity for learning about how the effects of interventions or programs are distributed across sites. In particular, these studies can produce rigorous estimates of a cross-site mean effect of program assignment…

Assessment is an integral part of the teaching-learning process in both conventional and distance education contexts. Literature suggests that with the increase in the use of Information and Communications Technology in the delivery of learning, a number of institutions are resorting to formative assessment practices that are mediated by…

Within the Higher Education sector in the UK, it is acknowledged that the area of 'Assessment and Feedback' receives consistently poor levels of satisfaction from students when they complete module level feedback, course level feedback and the National Student Survey (NSS). There is evidence to suggest that this problem is pronounced within…

There is a lively discussion on how to evaluate competence-based higher education in both evaluation and competence research. The instruments used are often limited to course evaluation or specific competences, taking a rather narrow perspective. Furthermore, the instruments often comprise predetermined competences that cannot be adapted to higher…

We examined the extent to which mathematics program evaluations that have been conducted according to methodologically rigorous standards have attended to the theories underlying the programs being evaluated. Our analysis focused on the 37 reports of K-12 mathematics program evaluations in the last two decades that have met standards for inclusion…

The purpose of this article is to clarify the discrepancy in the use of "queer" as a sexual identity classification in education survey research. This study extends the work completed by Dugan and Yurman (2011), who empirically demonstrated problems with treating LGB students as a homogenous population through collapsing all respondents…

This article discusses the nonuse, misuse, and proper use of pilot studies in experimental evaluation research. The authors first show that there is little theoretical, practical, or empirical guidance available to researchers who seek to incorporate pilot studies into experimental evaluation research designs. The authors then discuss how pilot…

In Australia, a review of the higher education sector is usually triggered by a change in government leadership, followed by the development and implementation of the government's response in the form of a reform package to enact change. The aim of this study was to conduct an independent evaluation of a large-scale national government policy…

The present article considers a fundamental question in evaluation research: "By how much do program effects vary across sites?" The article first presents a theoretical model of cross-site impact variation and a related estimation model with a random treatment coefficient and fixed site-specific intercepts. This approach eliminates…