NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: EJ1489531
Record Type: Journal
Publication Date: 2025
Pages: 13
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: EISSN-2375-5636
Available Date: 0000-00-00
Blueprinting the Future: Automatic Item Categorisation Using Hierarchical Zero-Shot and Few-Shot Classifiers
Ting Wang; Keith Stelter; Thomas O’Neill; Nathaniel Hendrix; Andrew Bazemore; Kevin Rode; Warren P. Newton
Journal of Applied Testing Technology, v26 n1 p1-13 2025
Precise item categorisation is essential in aligning exam questions with content domains outlined in assessment blueprints. Traditional methods, such as manual classification or supervised machine learning, are often time-consuming, error-prone, or limited by the need for large training datasets. This study presents a novel approach using zero-shot and few-shot Generative Pretrained Transformer (GPT) models for hierarchical item categorisation. By leveraging human-readable language descriptions within a structured Python dictionary, the model navigates complex blueprint hierarchies without requiring extensive training data. An initial simulation with synthetic items demonstrated the method's effectiveness, achieving an average F1 score of 92.91%. The approach was then applied to 200 real exam items from the 2022 In-Training Examination (ITE) by the American Board of Family Medicine (ABFM), reclassifying them according to a newly developed blueprint in just 15 minutes--a process that would typically take several days of expert review. This technique offers rapid, consistent, and scalable item categorisation, minimises human bias, and allows for iterative refinement through simple adjustments to category definitions, enhancing both efficiency and sustainability in assessment design.
Association of Test Publishers. 601 Pennsylvania Avenue NW, South Building Suite 900, Washington DC 20004. Tel: 866-240-7909; Fax: 717-755-8962; e-mail: wgh.atp@att.net; Web site: https://jattjournal.net/index.php/atp/index
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A
Author Affiliations: N/A