Studies on automatic scoring systems in writing assessments have also evaluated the relationship between human and machine scores for the reliability of automated essay scoring systems. This study investigated the magnitudes of indices for inter-rater agreement and discrepancy, especially regarding human and machine scoring, in writing assessment.…

The use of accommodations is often recommended to remove barriers to academic testing among English Learners (ELs). However, it is unclear whether accommodations are particularly effective at improving ELs' test scores. A growing foundation of empirical work has explored this topic. We conducted a meta-analysis that examined several possible…

The COVID-19 pandemic has brought the rapid transition to online quantitative education, leading to unique challenges and opportunities in assessments for both students and instructors. Focusing on undergraduate students and teaching staff at a Canadian university, this study investigates and compares their experiences and perspectives regarding…

Any body of research evolves over time. Previous understandings become more nuanced, ideas are supported or rebuked, and, eventually we arrive at a clearer view of the issue. The research on score comparability across computerized devices is no exception. CCSSO [Council of Chief State School Officers] and the Center for Assessment have published…

The purpose of this document is to provide information and advice to support states in meeting USED Peer Review Requirements related to demonstrating the comparability of test scores across various devices used for technology-based testing. The document is divided into four main sections. In the first section, we discuss and define the…

Evidence of comparability is generally needed whenever there are variations in the conditions of an assessment administration, including variations introduced by the administration of an assessment on multiple digital devices (e.g., tablet, laptop, desktop). This article is meant to provide a comprehensive examination of issues relevant to the…

Complex Problem Solving (CPS) is seen as a cross-curricular 21st century skill that has attracted interest in large-scale-assessments. In the Programme for International Student Assessment (PISA) 2012, CPS was assessed all over the world to gain information on students' skills to acquire and apply knowledge while dealing with nontransparent…

This Information Capsule reviews research conducted on computer-based assessments. Advantages and disadvantages associated with computer-based testing programs are summarized and research on the comparability of computer-based and paper-and-pencil assessments is reviewed. Overall, studies suggest that for most students, there are few if any…

There have been many studies of the comparability of computer-administered and paper-administered tests. Not surprisingly (given the variety of measurement and statistical sampling issues that can affect any one study) the results of such studies have not always been consistent. Moreover, the quality of computer-based test administration systems…

Because of the great promise of distance learning for delivering cost-effective instruction, there is great interest in determining whether or not it actually is effective, and--more interesting--determining what variables of design and implementation make it more or less effective. Unfortunately, much of the research has been based on simple…

Examines research showing the equivalence of paper-and-pencil-administered tests and computer-administered tests, especially when they are exactly the same test. Four categories of research are examined: previous reviews of research, standards and guidelines for computerized testing, individual research studies, and studies about computer testing…

Reviews research literature on comparability issues in computerized adaptive testing (CAT) and synthesizes issues specific to comparability and test security. Develops a framework for evaluating comparability that contains three categories of criteria: (1) validity; (2) psychometric property/reliability; and (3) statistical assumption/test…

The Internet has become an inevitable tool for collecting health and health behavior questionnaires. This study compared the feasibility, presence of score differences, and subjective evaluations by children between Internet and identical paper (asthma/fruit) questionnaires in elementary schools. A randomized crossover design was applied, with…

This paper reports on existing studies and uses meta analysis to compare and synthesize the results of 20 studies from 8 research reports comparing the ability measure equivalence of computer adaptive tests (CAT) and conventional paper and pencil tests. Using the research synthesis techniques developed by Hedges and Olkin (1985), it is possible to…

A literature review was conducted to determine the current state of knowledge concerning the effects of computer administration of standardized educational and psychological tests on the psychometric properties of these instruments. Students were grouped according to a number of factors relevant to the administration of tests by computer. Based on…