The purpose of this study was to examine the effectiveness of using a Competency-based Achievement System (CBAS) (a formative tool) in the practice ready assessment (PRA) of internationally-trained physicians seeking licensure to practice in Canada (summative assessment). The study was a retrospective analysis of over 5,000 FieldNotes completed for 68 candidates in a PRA program. The results indicate that CBAS is a useful tool for PRA. It provides a framework for integrating assessments of clinical competencies and documenting assessment evidence based on direct observations of candidate's performance in multiple clinical settings, case discussions, chart reviews and informal feedback from various sources. The documented evidence can be used to support summative decision-making on candidates' practice readiness as well as providing formative feedback to candidates to facilitate their professional development. The study also highlighted the areas in need of improvement for effective utilization of the CBAS.