ERIC Number: ED669040
Record Type: Non-Journal
Publication Date: 2021
Pages: 158
Abstractor: As Provided
ISBN: 979-8-5442-9323-1
ISSN: N/A
EISSN: N/A
Available Date: 0000-00-00
Variation in Assembling Assessments Using Automated Test Assembly Methodologies: Item-Pool Constraints and Response-Time Targets
Aaron McVay
ProQuest LLC, Ph.D. Dissertation, The University of Iowa
As assessments move towards computerized testing and making continuous testing available the need for rapid assembly of forms is increasing. The objective of this study was to investigate variability in assembled forms through the lens of first- and second-order equity properties of equating, by examining three factors and their interactions. Two of the three factors relate to item-pool properties, divergent item properties, and item-pool size, and the third is the use of three levels of estimated response time targets. Automated test assembly is implemented using the OPTMODEL procedure in base SAS® version 9.4, SAS/OR® version 15.2. Item data were simulated from a large-scale operational 5th grade Mathematics assessment. At the form level, variation in first- and second-order equity are evaluated along with the resulting observed-score distributions. At the optimization level, variation in solve time, iterations, and the absolute objective gap are evaluated. The study used a fully crossed 3x2x2 design, and results from two MANOVA analyses indicate that the three response time constraints used accounted for 22% of the variation in the optimal linear combination of solve time, iterations, and absolute objective gap as determined by the MANOVA. A follow-up test showed that only the iterations variable was significantly different across the response time constraints, however, solve time was not significantly different. A general lack of practically significant differences across the main effects and interactions, along with the successful assembly of 3,000 forms with the desired constraints, support the reliability of the implemented automated test assembly system in generating observed-score pre-equated forms. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]
Descriptors: Automation, Computer Assisted Testing, Test Items, Reaction Time, Evaluation Methods, Test Construction, Mathematics Tests, Grade 5, Reliability
ProQuest LLC. 789 East Eisenhower Parkway, P.O. Box 1346, Ann Arbor, MI 48106. Tel: 800-521-0600; Web site: http://www.proquest.com/en-US/products/dissertations/individuals.shtml
Publication Type: Dissertations/Theses - Doctoral Dissertations
Education Level: Elementary Education; Grade 5; Intermediate Grades; Middle Schools
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A
Author Affiliations: N/A