With the widespread application of online education platforms, the necessity for identifying learners' mental states from webcam videos is increasing as it can be potentially applied to artificial intelligence-based automatic identification of learner states. However, the behaviors that elementary school students frequently exhibit during online learning particularly when they are in a low attention state have rarely been investigated. This study employed electroencephalography (EEG) to continuously track changes in the learner's attention state during online learning. A new EEG index reflecting elementary students' attention level was developed using an EEG dataset acquired from 34th graders during a computerized d2 test of attention. Characteristic behaviors of 24 elementary students in a low attention state were then identified from the webcam videos showing their upper bodies captured during 40-min online lectures, with the proposed EEG index being used as a reference to determine their attention level at the time. Various characteristic behaviors were identified regarding the participant's mouth, head, arms, and torso. For example, opening the mouth or leaning back was observed more frequently in a low-attention state than in a high-attention state. It is expected that the characteristic behaviors reflecting a learner's low attention state would be utilized as a useful reference in developing more interactive and effective online education systems.