Course-based undergraduate research experiences (CUREs) have grown in popularity, particularly within introductory biology courses, to provide more students with authentic research experiences. CUREs have been shown to have many of the same positive effects as conventional research experiences; however, most assessments of CUREs lack an appropriate comparison group to evaluate their effectiveness. Here, we introduced a CURE into an introductory biology lab at a regional public university but maintained a traditional, "cookie-cutter" lab in half the lab sections over a 3-year period. We compared changes in test scores and survey responses, final lab and lecture scores, and D, F, withdrawal, and incomplete (DFWI) and retention rates between non-honors biology students in each group. While both groups showed significant improvement in test scores, only transfer students had significantly greater improvement in the CURE vs traditional lab. Students in both groups showed significant increases in self-confidence in some lab-related tasks, but differences in these changes were generally not significant between groups. There were no significant differences in final lecture score, lab score, DFWI rate, or retention rate. Factors affecting the lack of measured CURE success may include the type of CURE chosen, student career interests, and COVID-19; other positive impacts of the CURE may not have been captured by our measurements. This study demonstrates the importance of carefully choosing a CURE to match the student population, as well as assessing the CURE's impact against a comparison group.