This study investigates the impact of the language of instruction on assessment outcomes by comparing students' test performance in English (EMI) and Japanese Medium Instruction (JMI) courses. Data were collected from a university in Japan offering parallel Chemistry courses in Japanese and English as part of the same undergraduate curriculum. A mixed-methods approach was employed, with the quantitative analysis of pre-test and post-test scores of 28 EMI and 29 JMI students and the qualitative analysis of interviews conducted with eight EMI and six JMI students. The results revealed that EMI students underperformed significantly on explanation-oriented questions compared to JMI students, while there was no difference between the groups for calculation-oriented questions. When controlling for students' pre-test scores, the group differences remained significant for explanation-oriented questions but not for calculation-oriented questions. Findings also demonstrated a significant interaction between question type and group, indicating a substantial impact of the language of instruction on the test performance of EMI students compared to JMI students. The qualitative data highlighted the unique language-related challenges EMI students encountered when learning Chemistry. These findings highlight the need for diverse and fair assessment methods in EMI programs to ensure language proficiency does not disproportionately influence students' test performance.