Pointing to locations can either help or hinder people's ability to recall spatial information. Prior research has focused on two-dimensional spatial memory and pointing in real world tasks. The effect of pointing on three-dimensional spatial memory in virtual environments remains unexplored. We examine this effect by comparing participants' recall when they point to and memorize a series of three-dimensional locations using a gesture interface, and when they only passively view and memorize the spatial array. Thirty-three participants completed 50 trials in both conditions. The results reveal that pointing has a negative effect on memory in the three-dimensional virtual environment. These findings held across memory in the "x-y" plane of the computer screen as well as memory in the virtual third dimension. When they pointed during encoding, participants also responded more slowly, suggesting a higher cognitive load. Thus, active pointing may cause interference when users are simultaneously performing spatial memorization tasks.