Despite the increasing demand for diagnostic information, observed subscores have been often reported to lack adequate psychometric qualities such as reliability, distinctiveness, and validity. Therefore, several statistical techniques based on CTT and IRT frameworks have been proposed to improve the quality of subscores. More recently, DCM has…

The current study aims to evaluate the performance of three non-IRT procedures (i.e., normal approximation, Livingston-Lewis, and compound multinomial) for estimating classification indices when the observed score distribution shows atypical patterns: (a) bimodality, (b) structural (i.e., systematic) bumpiness, or (c) structural zeros (i.e., no…

In this digital ITEMS module, Dr. Michael Bunch provides an in-depth, step-by-step look at how standard setting is done. It does not focus on any specific procedure or methodology (e.g., modified Angoff, bookmark, and body of work) but on the practical tasks that must be completed for any standard setting activity. Dr. Bunch carries the…

Various approaches help determine students' need for language support courses. Some programs use standardized language proficiency test scores, which are available from the admissions process; others use local placement tests, which are typically designed to be aligned with local language needs and ESL course curricula. Using local placement tests…

Multiple popular math curriculum-based measures have recently been revised in ways that may improve their utility for universal screening. However, the applied use of these tools has yet to be evaluated independently. We conducted a retrospective analysis of the diagnostic accuracy of prior year statewide test results and aimswebPlus math…

The study reported on here contributes to the growing body of knowledge on the use of standard setting methods for improving the reporting and utility value of assessment results in South Africa as well as for addressing the conceptual shortcomings of the Curriculum and Assessment Policy Statement (CAPS) reporting framework. Using data from the…

Research on high-ability postsecondary students has increased in recent years; yet identifying such students can be challenging. The International Cognitive Ability Resource (ICAR) is an online, open-access tool designed to facilitate measurement of cognitive abilities in research. We evaluated whether the ICAR is appropriate to identify…

The purpose of this study is to introduce a method for converting scored 4-option multiple-choice (MC) items into scored 3-option MC items without re-pretesting the 3-option MC items. This study describes a six-step process for achieving this goal. Data from a professional credentialing exam was used in this study and the method was applied to 24…

This study aims to demonstrate the optimal way to determine the cut-off score to be used to interpret the total scores obtained from an achievement test or scale using the Artificial Neural Networks method. To this end, the multiple-choice item responses in the Booklet-11 Mathematics subtest at the 8th grade level in the TIMSS 2015 Turkey sample…

Rasch and classification analyses on a field-test version of the third edition of the Assessment, Evaluation, and Programming System (AEPS-3), a curriculum-based assessment used to assess young children birth to age 6 years, were conducted. First, an evaluation of the psychometric properties of data from each developmental area of an AEPS-3…

Background/Context: The landscape of college remediation programs experienced significant shifts from prerequisite models to corequisite models in the past few years across the nation. The traditional prerequisite models required students placed below college level to pass a sequence of remedial courses before they could enroll in college-level…

Selective cutoff reporting in primary diagnostic accuracy studies with continuous or ordinal data may result in biased estimates when meta-analyzing studies. Collecting individual participant data (IPD) and estimating accuracy across all relevant cutoffs for all studies can overcome such bias but is labour intensive. We meta-analyzed the…

To estimate the long-term effects of the Reading Recovery® intervention, a regression discontinuity design (RD) was implemented in a randomly selected sample of Reading Recovery schools during each year of the federally-funded i3 Scale-Up external evaluation (2011-2015) and also in one additional cohort during the 2016-17 school year. Long-term…

Universal screening to predict students' risk for reading problems is a foundational component of the Multi-Tiered Systems of Support framework and is required by law in many US states. School or district administrators are tasked with selecting screening assessments that are both technically adequate and feasible given the resources of their…

Since the No Child Left Behind (NCLB) Act required classifications of students' performance levels, test scores have been used to measure students' achievement; in particular, test scores are used to determine whether students reach a proficiency level in the state assessment. Accordingly, school districts have started using benchmark assessments…