In recent years, Automatic Item Generation (AIG) has increasingly shifted from theoretical research to operational implementation, a shift raising some unforeseen practical challenges. Specifically, generating high-quality answer choices presents several challenges such as ensuring that answer choices blend in nicely together for all possible item…

As access to smartphones and emerging technologies has become ubiquitous in our daily lives and in language learning, technology-mediated social interaction has become common in teaching and assessing L2 speaking. The changing ecology of L2 spoken interaction provides language educators and testers with opportunities for renewed test design and…

The top-down approach to designing a multistage test is relatively understudied in the literature and underused in research and practice. This study introduced a route-based top-down design approach that directly sets design parameters at the test level and utilizes the advanced automated test assembly algorithm seeking global optimality. The…

Computer-based educational assessments often include items that involve drag-and-drop responses. There are different ways that drag-and-drop items can be laid out and different choices that test developers can make when designing these items. Currently, these decisions are based on experts' professional judgments and design constraints, rather…

This research derived information functions and proposed new scalar information indices to examine the quality of multidimensional forced choice (MFC) items based on the RANK model. We also explored how GGUM-RANK information, latent trait recovery, and reliability varied across three MFC formats: pairs (two response alternatives), triplets (three…

U.S. Supreme Court justices and other federal judges are, effectively, appointed for life, with no built-in check on their cognitive functioning as they approach old age. There is about a century of research on aging and intelligence that shows the vulnerability of processing speed, fluid reasoning, visual-spatial processing, and working memory to…

In line with the widespread proliferation of digital technology in everyday life, many countries are now beginning to use computer-based exams (CBEs) in their post-primary education systems. To ensure that these CBEs are delivered in a manner that preserves their fairness, validity, utility and credibility, several factors pertaining to their…

Research indicates that the performance-gap between English Language Learners (ELLs) and their non-ELL peers is partly due to ELLs' difficulty in understanding assessment language. Accommodations have been shown to narrow this performance-gap, but many accommodations studies have not used a randomized design and are based on relatively small…

In this technical report we document the development and piloting of the Learning Receptiveness Assessment (LRA) Greenhouse for Prekindergarten. This technological tool was designed to prevent reading disabilities by supporting effective "assessment-guided instructional decision-making" by Preschool teachers. The LRA Greenhouse comprises…

This article discusses critical methodological design decisions for collecting, interpreting, and synthesizing empirical evidence during the design, deployment, and operational quality-control phases for automated scoring systems. The discussion is inspired by work on operational large-scale systems for automated essay scoring but many of the…

Clinical reasoning is an essential competency in medical education. This study aimed at developing and validating a test to assess diagnostic accuracy, collected information, and diagnostic decision time in clinical reasoning. A norm-referenced computer-based test for the assessment of clinical reasoning (ASCLIRE) was developed, integrating the…

This report summarizes the thirteenth survey of states by the National Center on Educational Outcomes (NCEO) at the University of Minnesota. Results are presented for 49 states and 6 of the 11 unique states. The purpose of this report is to provide a snapshot of the new initiatives, trends, accomplishments, and emerging issues during this…

Suggestions for software design in computer managed testing (CMT) cover instructions to testees, their physical format, provision of practice items, and time limit information; test item presentation, physical format, discussion of task demands, review capabilities, and rate of presentation; pedagogically helpful utilities; typefonts; vocabulary;…

The Office of Personnel Management (OPM) decided to explore alternative examining procedures for positions covered by the Administrative Careers with America (ACWA) examination. One requirement for new procedures was that they be automated for use with OPM's recently developed Microcomputer Assisted Rating System (MARS), a highly efficient system…

During the past few years, several Harvard paper surveys were converted to Web surveys. These were high-profile surveys endorsed by the Provost and the Dean of the College, and covered major portions of the university population (all undergraduates, all graduate students, tenured and non-tenured faculty). When planning for these surveys started in…