ERIC Number: ED385558
Record Type: Non-Journal
Publication Date: 1994-Jun
Pages: 41
Abstractor: N/A
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: N/A
Using the Free-Response Scoring Tool To Automatically Score the Formulating-Hypotheses Item. GRE Board Professional Report No. 90-02bP.
Kaplan, Randy M.; Bennett, Randy Elliot
This study explores the potential for using a computer-based scoring procedure for the formulating-hypotheses (F-H) item. This item type presents a situation and asks the examinee to generate explanations for it. Each explanation is judged right or wrong, and the number of creditable explanations is summed to produce an item score. Scores were generated for 30 examinees' responses to each of 8 items by a semantic pattern-matching program and independently by 5 human raters. On its initial scoring run, the program agreed highly with the raters' mean item scores for some questions and improved its concurrence substantially as modifications to the automatic scoring process were made. By the final run, correlations between the program and the raters on item scores ranged from .89 to .97, and mean human-machine discrepancies ran from .6 to 1.1 on a 16-point scale. At the individual hypothesis level, the proportion agreement, given the large disproportion of correct responses in the sample, was little better than chance. F-H items might be more effectively scored by a semiautomatic system that combines machine processing with a small number of human judges, and a preliminary configuration for such a process is presented. Appendix A discusses scoring iterations and modifications to the tool, and Appendix B presents changes to the scoring tool's interface. (Contains 5 figures, 9 tables, and 14 references.) (Author/SLD)
Descriptors: Automation, Computer Assisted Testing, Correlation, Higher Education, Hypothesis Testing, Responses, Scores, Scoring, Semantics, Test Items
Publication Type: Reports - Evaluative
Education Level: N/A
Audience: N/A
Language: English
Sponsor: Graduate Record Examinations Board, Princeton, NJ.
Authoring Institution: Educational Testing Service, Princeton, NJ.
Grant or Contract Numbers: N/A
Author Affiliations: N/A