ERIC Number: EJ1378572
Record Type: Journal
Publication Date: 2023-Jun
Pages: 14
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-0266-4909
EISSN: EISSN-1365-2729
Available Date: N/A
Semi-Automatic Coding of Open-Ended Text Responses in Large-Scale Assessments
Journal of Computer Assisted Learning, v39 n3 p841-854 Jun 2023
Background: In the context of large-scale educational assessments, the effort required to code open-ended text responses is considerably more expensive and time-consuming than the evaluation of multiple-choice responses because it requires trained personnel and long manual coding sessions. Aim: Our semi-supervised coding method eco (exploring coding assistant) dynamically supports human raters by automatically coding a subset of the responses. Method: We map normalized response texts into a semantic space and cluster response vectors based on their semantic similarity. Assuming that similar codes represent semantically similar responses, we propagate codes to responses in optimally homogeneous clusters. Cluster homogeneity is assessed by strategically querying informative responses and presenting them to a human rater. Following each manual coding, the method estimates the code distribution respecting a certainty interval and assumes a homogeneous distribution if certainty exceeds a predefined threshold. If a cluster is determined to certainly comprise homogeneous responses, all remaining responses are coded accordingly automatically. We evaluated the method in a simulation using different data sets. Results: With an average miscoding of about 3%, the method reduced the manual coding effort by an average of about 52%. Conclusion: Combining the advantages of automatic and manual coding produces considerable coding accuracy and reduces the required manual effort.
Descriptors: Foreign Countries, Achievement Tests, International Assessment, Secondary School Students, Educational Assessment, Coding, Simulation, Accuracy, Semantics
Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Publication Type: Journal Articles; Reports - Evaluative
Education Level: Secondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Identifiers - Assessments and Surveys: Program for International Student Assessment
Grant or Contract Numbers: N/A
Author Affiliations: N/A