Job and practice analysis is a commonly used method for determining examination content specifications. However, difficulties arise when many domains are present, as mainstream approaches do not fully adhere to the essence of the weighing process, namely a "comparison-evaluation-decision" framework for assigning percentage values to the…

Administrative problems such as computer malfunction and power outage occasionally lead to missing item scores, and hence to incomplete data, on credentialing tests such as the United States Medical Licensing examination. Feinberg compared four approaches for reporting pass-fail decisions to the examinees with incomplete data on credentialing…

With the increased restrictions on physical distancing due to the COVID-19 pandemic, remote proctoring has emerged as an alternative to traditional onsite proctoring to ensure the continuity of essential assessments, such as computer-based medical licensing exams. Recent literature has highlighted the significant impact of different proctoring…

In response to the COVID-19 pandemic, the American Board of Surgery (ABS) shifted from in-person to remote administrations of the oral certifying exam (CE). Although the overall exam architecture remains the same, there are a number of differences in administration and staffing costs, exam content, security concerns, and the tools used to give the…

Credentialing examination developers rely on task (job) analyses for establishing inventories of task and knowledge areas in which competency is required for safe and successful practice in target occupations. There are many ways in which task-related information may be gathered from practitioner ratings, each with its own advantage and…

Standard setting is arguably one of the most subjective techniques in test development and psychometrics. The decisions when scores are compared to standards, however, are arguably the most consequential outcomes of testing. Providing licensure to practice in a profession has high stake consequences for the public. Denying graduation or forcing…

Unforeseen complications during the administration of large-scale testing programs are inevitable and can prevent examinees from accessing all test material. For classification tests in which the primary purpose is to yield a decision, such as a pass/fail result, the current study investigated a model-based standard error approach, Bayesian…

To mitigate security concerns and unfair score gains, credentialing programs routinely administer new test material to examinees retesting after an initial failing attempt. Counterintuitively, a small but growing body of recent research suggests that repeating the identical form does not create an unfair advantage. This study builds upon and…

This article addresses the issue of how to detect item preknowledge using item response time data in two computer-based large-scale licensure examinations. Item preknowledge is indicated by an unexpected short response time and a correct response. Two samples were used for detecting item preknowledge for each examination. The first sample was from…

This research evaluated the impact of a common modification to Angoff standard-setting exercises: the provision of examinee performance data. Data from 18 independent standard-setting panels across three different medical licensing examinations were examined to investigate whether and how the provision of performance information impacted judgments…

Practice analyses are routinely used in support of the development of occupational and professional certification and licensure examinations. These analyses usually survey incumbents to obtain importance ratings of (1) specific tasks and (2) knowledge, skill, and ability (KSA) statements deemed by subject matter experts as essential to safe and…

Offers recommendations for the conduct of practice analysis (i.e., job analysis) concerning these issues: (1) selecting a method of practice analysis; (2) developing rating scales; (3) determining the content of test plans; (4) using multi-variate procedures for structuring test plans; and (5) determining topic weights for test plans. (SLD)

"In his commentary to our paper on the use of knowledge, skill, and ability statements in developing credentialing examinations (Wang, Schnipke, & Witt, 2005 )," Dr. LaDuca set forth his concerns while commending our paper for providing helpful insights into the importance of careful delineation of KSAs. We believe that there is little substantive…

Summarizes conceptualizations of sound teacher certification practices espoused by the courts and psychometricians, compares these conceptualizations, and provides suggestions for teacher certification testing programs that are legally and psychometrically defensible. (SLD)

Explores certification and license testing and questions the extent to which scientific standards are actually governing policies and practices. Compares the practices of two states that have developed licensure programs for the same field and points out differences in these examinations. (SLD)