ERIC Number: EJ1459602
Record Type: Journal
Publication Date: 2024-Dec
Pages: 64
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: EISSN-2330-8516
Available Date: 0000-00-00
Modeling Writing Traits in a Formative Essay Corpus. Research Report. ETS RR-24-02
Paul Deane; Duanli Yan; Katherine Castellano; Yigal Attali; Michelle Lamar; Mo Zhang; Ian Blood; James V. Bruno; Chen Li; Wenju Cui; Chunyi Ruan; Colleen Appel; Kofi James; Rodolfo Long; Farah Qureshi
ETS Research Report Series, Dec 2024
This paper presents a multidimensional model of variation in writing quality, register, and genre in student essays, trained and tested via confirmatory factor analysis of 1.37 million essay submissions to ETS' digital writing service, Criterion®. The model was also validated with several other corpora, which indicated that it provides a reasonable fit for essay data from 4th grade to college. It includes an analysis of the test-retest reliability of each trait, longitudinal trends by trait, both within the school year and from 4th to 12th grades, and analysis of genre differences by trait, using prompts from the Criterion topic library aligned with the major modes of writing (exposition, argumentation, narrative, description, process, comparison and contrast, and cause and effect). It demonstrates that many of the traits are about as reliable as overall e-rater® scores, that the trait model can be used to build models somewhat more closely aligned with human scores than standard e-rater models, and that there are large, significant trait differences by genre, consistent with genre differences in trait patterns described in the larger literature. Some of the traits demonstrated clear trends between successive revisions. Students using Criterion appear to have consistently improved grammar, usage, and spelling after getting Criterion feedback and to have marginally improved essay organization. Many of the traits also demonstrated clear grade level trends. These features indicate that the trait model could be used to support more detailed scoring and reporting for writing assessments and learning tools.
Descriptors: Writing (Composition), Essays, Models, Elementary School Students, Secondary School Students, Validity, Reliability, Natural Language Processing, Artificial Intelligence, Writing Evaluation, Institutional Characteristics, Student Characteristics, Scoring, Automation
ETS. Rosedale Road, Mailstop 19R, Princeton, NJ 08541. Tel: 609-921-9000; Fax: 609-734-5410; e-mail: RDweb@ets.org; Web site: https://www.ets.org/
Publication Type: Journal Articles; Reports - Research
Education Level: Elementary Education; Secondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A
Author Affiliations: N/A