In psychology, researchers are often interested in testing hypotheses about mediation, such as testing the presence of a mediation effect of a treatment (e.g., intervention assignment) on an outcome via a mediator. An increasingly popular approach to testing hypotheses is the Bayesian testing approach with Bayes factors (BFs). Despite the growing popularity of BFs, the performance of using BFs to test mediation effects is understudied. In this article, we investigate the performance of the BF for testing the presence versus absence of a mediation effect, in terms of the true positive rates and false positive rates (i.e., the probabilities that the BF supports the presence of mediation when the mediation effect is present and when the mediation effect is absent, respectively). Our simulation results showed that the true positive and false positive rates of the BF for testing mediation can be impacted by the prior specification, including the prior odds specified for each path (treatment-mediator path, or mediator-outcome path) in the design stage and analysis stage. Based on our investigation, we developed an R function and a web application to help determine sample sizes for detecting mediation effects with the BF. Our study provides insights into the performance of using BFs for testing mediation effects and adds to researchers' toolbox for determining sample sizes in mediation studies. [This is the online first version of an article published in "Psychological Methods" (ISSN 1082-989X).]