Joining a research group is one of the most important events on a graduate student's path to becoming an independent physics researcher and earning a Ph.D. However, graduate students' perspectives on the experience of finding a research group are not well documented in the literature. Understanding these perspectives is crucial for evaluating whether departments are providing students with adequate support while they search for a research group, and how difficulties during this process contribute to attrition. Semistructured interviews with N=20 first and second year physics Ph.D. students reveal that incoming graduate students see joining a research group as a significant decision, and recognize that it may impact whether they will be able to complete the program. We found that students who struggled to find a group felt isolated and worried about falling behind their peers, whereas students who were able to immerse themselves in a positive group environment reported increased sense of belonging in their programs. The process of finding a research group often held differential importance for students identifying as women and nonbinary, who at times reported having to deprioritize their preferred research topic in order to be part of a more inclusive working environment. Although incoming graduate students characterized joining a research group as a significant decision, they often felt unprepared to make it. Moreover, they perceived an overall lack of guidance and structure from their departments, and characterized coursework as a barrier to searching for a group. Our findings suggest that providing students with better support during their group search process could help improve retention, particularly for traditionally underrepresented students, and improve students' overall satisfaction in their graduate programs.