To address the computational constraints of parameter estimation in the polytomous Cognitive Diagnosis Model (pCDM) in large-scale high data volume situations, this study proposes two two-stage polytomous attribute estimation methods: P_max and P_linear. The effects of the two-stage methods were studied via a Monte Carlo simulation study, and the…

Dynamic Measurement Modeling (DMM) is a psychometric paradigm that uses longitudinal data to estimate students' capacity to learn over the course of an educational program (i.e., growth scores). Here, we provide justification for this approach in health professions education and demonstrate its proof of concept with three time-points of USMLE Step…

Portfolio assessment of teaching practice has been used in many states as one indicator to inform licensure decisions, yet the human scoring process is costly for all stakeholders. This paper documents a series of investigations of alternative scoring procedures for a state pre-service teacher performance assessment. The proposed partial double…

A predictive model for the USMLE Step 1 was created based on NBME Comprehensive Basic Sciences Self-Assessment (CBSSA) exams. All second year medical students at a southwestern university from 2014-2016 took a NBME CBSSA exam under controlled testing conditions six months (January) and three months (April) prior to their Step 1. A multiple…

We explore whether the item characteristics of four-response multiple-choice questions with "paired" (non-independent) response options differ from those with unpaired response options in an operational licensure exam in terms of guessing behavior. Our analyses suggest that they do not. We discuss conditions where response pairing might…

The current state of technology allows physicians to look up information in everyday practice using a wealth of medical resources. This change in practice may be important enough to consider it being reflected in medical certification exams, making the exam more similar to real practice. However, there is limited research comparing open- and…

The purpose of this study is to determine whether participation in self-study and self-assessment CME programs improves physicians' performance on a Board Maintenance of Certification (MOC) examination. Linear regression showed that participation in such CME programs within one year of taking the MOC exam predicted higher performance on the MOC…

Purpose: Concerns about variation in the quality of pre-service preparation provided by many university-based principal preparation programs (PPPs) has led to calls to use outcomes of program graduates to hold PPPs accountable, which the recent growth of state administrative data systems has made feasible. Little research, however, has assessed…

The purpose of this study was to examine the effectiveness of using a Competency-based Achievement System (CBAS) (a formative tool) in the practice ready assessment (PRA) of internationally-trained physicians seeking licensure to practice in Canada (summative assessment). The study was a retrospective analysis of over 5,000 FieldNotes completed…