This paper describes the implementation of an inter-rater reliability measure for assessing portfolio scores in a teacher education program. The reliability coefficient for the portfolio scores from completers of a newly revised program were compared with the reliability coefficient of the scores from a second set of reviewers who discussed the scoring rubric in depth prior to their review. The results indicated low reliability between the first set of reviewers and moderate reliability between the second set. This project serves as a model for using a straightforward measure of inter-rater reliability to encourage practitioner reflection in teacher education programs by providing an objective tool to draw attention to the need for dialogue about the portfolio scoring process.