Background and Context: Auto-graders are praised by novice students learning to program, as they provide them with automatic feedback about their problem-solving process. However, some students often make random changes when they have errors in their code, without engaging in deliberate thinking about the cause of the error. Objective: To investigate whether requiring students using an auto-grading system to reflect on the errors in their code would reduce trial and error behavior often seen in such systems. Method: The paper analyzes the impact of reflection per student and per problem using paired t-tests. Findings: Students took fewer steps to solve the problem in reflective sessions than in Usual Debugging Sessions (4.33 vs 8.04) and they made half as many syntax errors, logic errors, and runtime errors. However, they took more time between runs. Implications: This paper provides evidence that requiring reflection in autograding systems can improve student debugging skills.