Despite the demonstrated learning benefits of peer evaluation, fears of teachers about its low reliability may restrict its use. In this study, the validity of peer assessment, in terms of agreement with the ratings of the teacher, has been tested in an organic chemistry course. The students were organized into small groups and commissioned to produce a screencast video on a molecule. Both students and teachers assessed the screencasts on five different dimensions. The internal consistency of the rating scale was confirmed. Comparing both datasets revealed fair correlations in all cases but statistically significant differences in four dimensions. The grades awarded by peers were lower than those granted by the teacher, which contradicts most of the results found in the literature. Statistically significant differences (p < 0.05) are compatible with the good agreement as reported by the Intraclass Correlation Coefficient and vice versa. Further research is necessary to elucidate the effect of diverse variables on the raters' agreement, improving the validity of peer evaluation.