Although AR technology seems to be promising for helping students learn complicated and abstract concepts, whether the positive effects of this new technology can be applied to the learning of three-dimensional geometric concepts, and what factors might mediate its effectiveness are questions worth further investigation. This study used the eye-tracking method to examine middle school students' learning of three-dimensional geometric concepts with AR-based instruction in comparison with a group of students who learned via paper-based instruction. Motivation and conceptual achievements were assessed using the Instructional Materials Motivation Survey (IMMS) questionnaire and Geometry Achievement Test (GAT), respectively. The results showed that while AR learners performed the same as the paper-based learners in the post concept test, the two groups differed significantly in terms of their motivation orientations and visual attention patterns. AR students were more satisfied with and engaged more in the corresponding instruction. To explore in-depth the effectiveness of the AR system, the AR participants were divided into improved and unimproved learners. The former displayed higher confidence in and perceived greater relevance of the AR instruction. Their attention to and engagement in the simple geometric concepts was higher. Correlation and regression analyses indicated that the performance of paper-based learners was affected by their visual attention, while the performance of the AR learners was predicted by their beliefs in the relevance of the material to the learning goals. Noticeably, for the AR learners, their motivation was significantly associated with their visual attention.