ERIC Number: EJ1474397
Record Type: Journal
Publication Date: 2025-Jul
Pages: 7
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-0098-6283
EISSN: EISSN-1532-8023
Available Date: 0000-00-00
Grading the Graders: Comparing Generative AI and Human Assessment in Essay Evaluation
Elizabeth L. Wetzler1; Kenneth S. Cassidy1; Margaret J. Jones1; Chelsea R. Frazier1; Nickalous A. Korbut1; Chelsea M. Sims1; Shari S. Bowen1; Michael Wood1
Teaching of Psychology, v52 n3 p298-304 2025
Background: Generative artificial intelligence (AI) represents a potentially powerful, time-saving tool for grading student essays. However, little is known about how AI-generated essay scores compare to human instructor scores. Objective: The purpose of this study was to compare the essay grading scores produced by AI with those of human instructors to explore similarities and differences. Method: Eight human instructors and two versions of OpenAI's ChatGPT (3.5 and 4o) independently graded 186 deidentified student essays from an introductory psychology course using a detailed rubric. Scoring consistency was analyzed using Bland-Altman and regression analyses. Results: AI scores for ChatGPT3.5 were, on average, higher than human scores, although average scores for ChatGPT 4o and human scores were more similar. Notably, AI grading for both versions was more lenient than human instructors at lower performance levels and stricter at higher levels, reflecting proportional bias. Conclusion: Although AI may offer potential for supporting grading processes, the pattern of results suggests that AI and human instructors differ in how they score using the same rubric. Teaching Implications: Results suggest that educators should be aware that AI grading of psychology writing assignments that require reflection or critical thinking may differ markedly from scores generated by human instructors.
Descriptors: Essays, Writing Evaluation, Scores, Evaluators, Writing Instruction, Introductory Courses, Grading, Comparative Analysis, Psychology, Artificial Intelligence, Computer Software, Technology Integration, Scoring Rubrics, Writing Assignments
SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A
Data File: URL: https://osf.io/ystb4/?view_only=6cb52e885d254822856e699dff054153
Author Affiliations: 1Department of Behavioral Sciences and Leadership, United States Military Academy, West Point, New York, USA