This study is part of a larger effort to investigate linguistic features of assessment items that might explain differential performance of English learners on science assessments. An analysis of a large set of multiple-choice items suggested two features that held promise in reducing the performance gap: the presence of contrast words and the reduction of 1+ clauses. A subset of 48 items were modified and used in a study with 638 students to investigate the effect of these features. For half of those items, we reduced the number of 1+ clauses; for the other half, we removed contrast words. We analyzed the results from two subsets of students: 358 9th grade native-English and native-Spanish speaking students from across the U.S. and 138 8th grade students from one teacher at a school with 50% Spanish speaking students who agreed to participate in follow-up focus group discussions about the items. We compared student performance on the modified and original items and found little to no support for the selected linguistic features as affecting English learners' scores differentially, suggesting that performance was more strongly dependent on content knowledge than on the linguistic features of the items. The follow-up discussions also supported the idea that students' performance was strongly related to their content knowledge level.