The COVID-19 pandemic caused severe disruptions in living and learning to millions of college students. Here we investigated using mediation analysis two dimensions of anxiety that were specific to the pandemic -- COVID-19 related anxiety and COVID-19 vaccine anxiety -- to evaluate their relationship to college adjustment during the pandemic. Using cross-sectional survey data across three semester waves (Spring 2021, Fall 2021, and Spring 2022) we probed whether anxiety functioned as a challenge or hindrance stressor on adjustment. We found that although anxiety decreased in both COVID-19 dimensions across semesters, student adjustment to college remained consistently low. Our mediation analysis revealed that both COVID-19 related anxiety and COVID-19 vaccine-related anxiety functioned as challenge stressors, elevating academic, social, personal-emotional, and institutional adjustment during the pandemic. We discuss the role of positive COVID impacts on college adjustment, including enhanced social support.