Background: A growing number of surveys are conducted online where respondents can choose to complete the questionnaire (Lehdonvirta et al., 2020). As respondents are self-selected, there is potential that the respondents will not be an accurate representation of the population. For example, white people are disproportionately more likely to…

The EdSurvey package helps users download, explore variables in, extract data from, and run analyses on large-scale assessment data. The analysis functions in EdSurvey account for the use of plausible values for test scores, survey sampling weights, and their associated variance estimator. We describe the capabilities of the package in the context…

Mediation analysis plays an important role in understanding causal processes in social and behavioral sciences. While path analysis with composite scores was criticized to yield biased parameter estimates when variables contain measurement errors, recent literature has pointed out that the population values of parameters of latent-variable models…

Work based assessment (WBA) is a common but contentious practice increasingly used to grade university students on professional degrees. A key issue in WBA is the potentially low assessment literacy of the assessors, which can lead to a host of unintended results, including grade inflation. We identified grade inflation in the WBA of the clinical…

During the past decade, the NAEP Validity Studies (NVS) Panel has been monitoring, studying, and commenting on potential issues with the validity of the National Assessment of Educational Progress (NAEP) arising from changes that have been brought about by the adoption of rigorous state college and career readiness standards, such as the Common…

We used meta-analysis to examine the criterion validity of four scoring procedures used in curriculum-based measurement of written language. A total of 22 articles representing 21 studies (N = 21) met the inclusion criteria. Results indicated that two scoring procedures, correct word sequences and correct minus incorrect sequences, have acceptable…

Automated essay scoring (AES) generally computes essay scores as a function of macrofeatures derived from a set of microfeatures extracted from the text using natural language processing (NLP). In the "e-rater"® automated scoring engine, developed at "Educational Testing Service" (ETS) for the automated scoring of essays, each…

Although International Association for the Evaluation of Educational Achievement-pioneered international large-scale assessment (ILSA) of education is now a well-established science, non-practitioners and many users often substantially misunderstand how large-scale assessments are conducted, what questions and challenges they are designed to…

Frustrations with traditional testing led a group of teachers at the English for adults program at Universidad EAFIT (Colombia) to design tests aligned with the institutional teaching philosophy and classroom practices. This article reports on a study of an item-by-item evaluation of a series of English exams for validity and reliability in an…

The results of large-scale international assessments such as Programme for International Student Assessment (PISA) have attracted a considerable attention worldwide and are often used by policy-makers to support educational policies. To ensure that the published results represent the actual population, these surveys go through a thorough scrutiny…

The scoring instrument described in this article is based on a deconstruction of the seven sections of an American Psychological Association (APA)-style empirical research report into a set of learning outcomes divided into content-, expression-, and format-related categories. A double-weighting scheme used to score the report yields a final grade…

This article proposes that sampling design effects have potentially huge unrecognized impacts on the results reported by large-scale district and state assessments in the United States. When design effects are unrecognized and unaccounted for they lead to underestimating the sampling error in item and test statistics. Underestimating the sampling…

Purpose: To develop a system for numerically quantifying a speaker's phonetic accuracy through transcription-based measures. With a focus on normal and disordered speech in children, the authors describe a system for differentially weighting speech sound errors on the basis of various levels of phonetic accuracy using a Weighted Speech Sound…

In 2002, an article entitled "Four assumptions of multiple regression that researchers should always test" by Osborne and Waters was published in "PARE." This article has gone on to be viewed more than 275,000 times (as of August 2013), and it is one of the first results displayed in a Google search for "regression…

The Applied Language Centre at University College Dublin offers foreign language modules to students in ten languages at CEFR [Common European Framework of Reference for Languages] levels ranging from A1 to B2. Efforts have been underway in the Centre to standardise the assessment components across languages to ensure parity between module credits…