To address the computational constraints of parameter estimation in the polytomous Cognitive Diagnosis Model (pCDM) in large-scale high data volume situations, this study proposes two two-stage polytomous attribute estimation methods: P_max and P_linear. The effects of the two-stage methods were studied via a Monte Carlo simulation study, and the…

Dynamic Measurement Modeling (DMM) is a psychometric paradigm that uses longitudinal data to estimate students' capacity to learn over the course of an educational program (i.e., growth scores). Here, we provide justification for this approach in health professions education and demonstrate its proof of concept with three time-points of USMLE Step…

Portfolio assessment of teaching practice has been used in many states as one indicator to inform licensure decisions, yet the human scoring process is costly for all stakeholders. This paper documents a series of investigations of alternative scoring procedures for a state pre-service teacher performance assessment. The proposed partial double…

A predictive model for the USMLE Step 1 was created based on NBME Comprehensive Basic Sciences Self-Assessment (CBSSA) exams. All second year medical students at a southwestern university from 2014-2016 took a NBME CBSSA exam under controlled testing conditions six months (January) and three months (April) prior to their Step 1. A multiple…

We explore whether the item characteristics of four-response multiple-choice questions with "paired" (non-independent) response options differ from those with unpaired response options in an operational licensure exam in terms of guessing behavior. Our analyses suggest that they do not. We discuss conditions where response pairing might…

The current state of technology allows physicians to look up information in everyday practice using a wealth of medical resources. This change in practice may be important enough to consider it being reflected in medical certification exams, making the exam more similar to real practice. However, there is limited research comparing open- and…

The purpose of this study is to determine whether participation in self-study and self-assessment CME programs improves physicians' performance on a Board Maintenance of Certification (MOC) examination. Linear regression showed that participation in such CME programs within one year of taking the MOC exam predicted higher performance on the MOC…

Purpose: Concerns about variation in the quality of pre-service preparation provided by many university-based principal preparation programs (PPPs) has led to calls to use outcomes of program graduates to hold PPPs accountable, which the recent growth of state administrative data systems has made feasible. Little research, however, has assessed…

The purpose of this study was to examine the effectiveness of using a Competency-based Achievement System (CBAS) (a formative tool) in the practice ready assessment (PRA) of internationally-trained physicians seeking licensure to practice in Canada (summative assessment). The study was a retrospective analysis of over 5,000 FieldNotes completed…

The synthetic function, which is a weighted average of the identity (the trivial linking function for forms that are known to be completely parallel) and a traditional equating method, has been proposed as an alternative for performing linking with very small samples (Kim, von Davier, & Haberman, 2006). The purpose of the present study was to…

This study explored the effects of external anchor test length on final equating results of several equating methods, including equipercentile (frequency estimation), chained equipercentile, kernel equating (KE) poststratification PSE with optimal bandwidths, and KE PSE linear (large bandwidths) when using the nonequivalent groups anchor test…

This study was conducted to compare the usefulness of three item response theory (IRT) calibration packages (BILOG, BILOG-MG, and PIC) for examinations that include common and specialty components. Because small sample sizes and different mean abilities between specialty components are the most frequent problems that licensure/certification…

The purpose of this study was to determine if administrative aspirants from preparation programs, experienced school administrators, and groups of individuals with no connection to education would perform differently on a short vignette discontinued by the Educational Testing Service (ETS) but still used as a sample of the assessment in the School…

Conducted a validity study to examine the degree to which scores on the newly developed Diagnostic Readiness Test (DRT) and National League for Nursing Pre-Admission Test scores could predict success or failure on the National Council Licensure Examination for Registered Nurses (NCLEX-RN). Results for 5,698 students indicate that the DRT is a…

Two standard setting methods, the Angoff method and the Rasch model based Item Map approach, were compared for setting a standard for a high-stakes medical licensure examination, the last examination of a three-examination series of a national medical licensing examination. The standard setting committee consisted of 23 physicians who were…