ERIC Number: ED599252
Record Type: Non-Journal
Publication Date: 2019-Jul
Pages: 6
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: N/A
Improving Peer Assessment Accuracy by Incorporating Relative Peer Grades
Wang, Tianqi; Jing, Xia; Li, Qi; Gao, Jing; Tang, Jie
International Educational Data Mining Society, Paper presented at the International Conference on Educational Data Mining (EDM) (12th, Montreal, Canada, Jul 2-5, 2019)
Massive Open Online Courses (MOOCs) have become more and more popular recently. These courses have attracted a large number of students world-wide. In a popular course, there may be thousands of students. Such a large number of students in one course makes it infeasible for the instructors to grade all the submissions. Peer assessment is thus an effective paradigm that can help grade the submissions at a large scale. However, due to the variance in the ability and standard of the student graders, peer grades may be noisy and biased. Aggregating peer grades to have an accurate and fair final grade for a submission is a challenging problem because the reliability and bias degrees of graders are usually unknown in practice. To address this issue, some probabilistic models considering the graders' reliability and bias are proposed. However, due to the sparsity of peer grade observations, it is difficult for these models to estimate the accurate reliability and bias of the graders as well as the true grades of the submissions. Compared with absolute peer grades, the relative peer grades, derived from the difference between the peer grades of two submissions graded by the same grader, are less sparse and more robust to the grader's bias. Thus relative peer grades are informative and helpful in cardinal peer grading estimation whose goal is to estimate the absolute numeric grades of submissions. In this paper, we propose two new probabilistic models to help improve the accuracy of cardinal peer grading estimation using the observed relative grades among submissions. In this way, the relation between the true grades among submissions is taken into consideration when deriving the final grades. Experimental results on real MOOC peer grading datasets show that the proposed models outperform baselines and the relation of true grades among submissions indeed contributes to the improvement in the grade estimation. [For the full proceedings, see ED599096.]
Descriptors: Peer Evaluation, Accuracy, Grades (Scholastic), Grading, Online Courses, Evaluators, Reliability, True Scores, Foreign Countries
International Educational Data Mining Society. e-mail: admin@educationaldatamining.org; Web site: http://www.educationaldatamining.org
Publication Type: Speeches/Meeting Papers; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: National Science Foundation (NSF)
Authoring Institution: N/A
Identifiers - Location: China
Grant or Contract Numbers: IIS1553411
Author Affiliations: N/A