Many previous studies observed that higher retrospective confidence ratings about memory performance were associated with shorter response times in memory test. Researchers often interpret response time as a measure of retrieval fluency which is an important cue utilized in confidence formation process. However, the drift diffusion model (DDM) indicates that response time in recognition memory test includes both a decision component representing memory retrieval, and a non-decision component unrelated to retrieval process. Few previous studies have investigated whether retrospective confidence in recognition test is related to the speed of both retrieval and non-decision processes. To address this question, the current study first analyzed data from six published experiments, and found that higher retrospective confidence ratings were associated with both higher drift rate (indicating retrieval fluency) and shorter non-decision time in DDM. Then we manipulated the ease of perception in two new experiments, and the results consistently indicated that difficulty in stimulus perception increased non-decision time in recognition test, which affected retrospective confidence. Furthermore, the documented results also suggest drift rate could partly account for the positive relationship between confidence and memory performance, while the reliance of confidence on non-decision time negatively affected confidence accuracy.