The use of assessments that require rater judgment (i.e., rater-mediated assessments) has become increasingly popular in high-stakes language assessments worldwide. Using a systematic literature review, the purpose of this study is to identify and explore the dominant methods for evaluating rating quality within the context of research on…

Multisite trials, which are being used with increasing frequency in education and evaluation research, provide an exciting opportunity for learning about how the effects of interventions or programs are distributed across sites. In particular, these studies can produce rigorous estimates of a cross-site mean effect of program assignment…

Assessment is an integral part of the teaching-learning process in both conventional and distance education contexts. Literature suggests that with the increase in the use of Information and Communications Technology in the delivery of learning, a number of institutions are resorting to formative assessment practices that are mediated by…

This article discusses the nonuse, misuse, and proper use of pilot studies in experimental evaluation research. The authors first show that there is little theoretical, practical, or empirical guidance available to researchers who seek to incorporate pilot studies into experimental evaluation research designs. The authors then discuss how pilot…

In Australia, a review of the higher education sector is usually triggered by a change in government leadership, followed by the development and implementation of the government's response in the form of a reform package to enact change. The aim of this study was to conduct an independent evaluation of a large-scale national government policy…

The present article considers a fundamental question in evaluation research: "By how much do program effects vary across sites?" The article first presents a theoretical model of cross-site impact variation and a related estimation model with a random treatment coefficient and fixed site-specific intercepts. This approach eliminates…

A new approach to identify item clusters fitting the Rasch model is described and evaluated using simulated and real data. The proposed method is based on hierarchical cluster analysis and constructs clusters of items that show a good fit to the Rasch model. It thus gives an estimate of the number of independent scales satisfying the postulates of…

The purpose of this study was to examine the impact of dimensionality, common-item set format, and different scale linking methods on preserving equity property with mixed-format test equating. Item response theory (IRT) true-score equating (TSE) and IRT observed-score equating (OSE) methods were used under common-item nonequivalent groups design.…

Multiple-indicators multiple-causes (MIMIC) modeling is often used to test a latent group mean difference while assuming the equivalence of factor loadings and intercepts over groups. However, this study demonstrated that MIMIC was insensitive to the presence of factor loading noninvariance, which implies that factor loading invariance should be…

Several studies have analysed and assessed online performance and discourse using quantitative and qualitative methods. Quantitative measures have typically included the analysis of participation rates and learning outcomes in terms of grades. Qualitative measures of postings, discussions and context features aim to give insights into the nature…

A consistent message emerges from research on undergraduate students' perceptions of assessment which describes traditional assessment as detrimental to learning. However this literature has not included students in the pure sciences. Mathematics education literature advocates the introduction of innovative assessment at university. In this…

The concept of usefulness has implicitly played a pivotal role in evaluation research, but the meaning of usefulness has changed over time from system reliability to user performance and learnability/ease of use for non-experts. Despite massive technical and social changes, usability remains the "gold standard" for system evaluation.…

There is a lack of research on the effects of outliers on the decisions about the number of factors to retain in an exploratory factor analysis, especially for outliers arising from unintended and unknowingly included subpopulations. The purpose of the present research was to investigate how outliers from an unintended and unknowingly included…

Statistical modeling of school effectiveness data was originally motivated by the dissatisfaction with the analysis of (school-leaving) examination results that took no account of the background of the students or regarded each school as an isolated unit of analysis. The application of multilevel analysis was generally regarded as a breakthrough,…

Some researchers have argued for the use of or have attempted to make use of randomization tests in single-subject research. To address this tide of interest, the authors of this article describe randomization tests, discuss the theoretical rationale for applying them to single-subject research, and provide an overview of the methodological…