Understanding how non-numerical visual features systematically distort numerosity perception holds promise for unveiling the processes that give rise to our visual number sense. Recent studies show that increasing visual coherence systematically increases perceived numerosity, with this effect strengthening over development (DeWind et al., 2020; Qu, Bonner, et al., 2024; Qu et al., 2022). Here, we investigate the cognitive mechanisms underlying the coherence illusion from a view of perceptual decision processes. Specifically, we applied a drift diffusion model (DDM) to a previously described dataset from participants aged 5-30 tested in an ordinal numerical comparison task with color entropy systematically manipulated (Qu et al., 2022). By jointly modeling choice data and response times, we decomposed numerical discrimination performance into distinct decision components: the speed of numerical evidence accumulation (drift rate), the amount of evidence required for a decision (boundary separation), and the response bias reflecting a prior tendency of selecting one side over the other. We found that color coherence affected only the drift rate but not response bias or boundary separation, demonstrating that color coherence distorts numerical calculation through biased accumulation of evidence of quantity. Moreover, the impact of coherence on the drift rate coefficient increased with age as quantitative information is accumulated more efficiently over development. Our results offer a framework for understanding how numerical illusions arise from perceptual decision-making dynamics.