A modified essay is an open-ended question about clinical presentations, with the goal to test application of basic science principles to novel scenarios. Due to high reliability and validity, multiple-choice question (MCQ) examinations are more commonly used. We have employed a combination of MCQ and modified essay examinations in medical school. The correlation between MCQ and modified essay grades is positive but small, suggesting they may be testing different cognitive levels/skills. In this paper, we evaluate the impact of a non-graded mock modified essay on student's performance and satisfaction, and outline our best practices. We have found that it is important to involve faculty from multiple disciplines to construct robust cases, and to review examinations and rubrics. The mock modified essay resulted in higher performance and student satisfaction, and fewer students' comments related to subjectivity, frustration and anxiety. Yet, there was not an improvement in confidence. Moreover, there was no correlation between grades in the mock and graded modified essays, suggesting that the mock modified essay may not be useful to predict academic performance. Yet, the incorporation of a mock modified essay was useful to clarify expectations and provide guidance, potentially reducing the anxiety induced by an unfamiliar assessment.