Computer-based, pop-up glossaries are perhaps the most promising accommodation aimed at mitigating the influence of linguistic structure and cultural bias on the performance of English Learner (EL) students on statewide assessments. To date, there is no established procedure for identifying the words that require a glossary for EL students that is sufficiently reliable. In the coding procedure, we developed a method to reliably identify words and phrases that require a glossary. The method developed in the coding procedure was used to provide glossaries for the field-test items of statewide English language arts (ELA) and mathematics assessments across grades 3-11 (Current Study). In the Current Study, we assess the effectiveness and influence on construct validity of a pop-up glossary of the words identified in the coding procedure in a large scale, randomized controlled trial. The results demonstrated that generally the pop-up glossary accommodation was effective for both the ELA and mathematics assessments and did not harm the construct being measured.