We report on an evaluation of the effectiveness of Project ARIES, an astronomy-based physical science curriculum for upper elementary and middle school children. ARIES students use innovative, simple, and affordable apparatus to carry out a wide range of indoor and outdoor hands-on, discovery-based activities. Student journals and comprehensive teacher materials aid in making the science content accessible to students based on their shared experiences and observations. Approximately 750 Grades 3-6 students in ARIES (or treatment) classrooms are compared with approximately 650 Grades 4-6 students in control classrooms through a series of open-ended assessment measures, using a pretest and posttest format. A detailed analysis by item measures the gain in treatment and control groups. We identify concepts where the ARIES approach is more effective, where both are equally effective, and where neither results in much learning. (The ARIES approach was never less effective.) Although learning is in evidence for both control and treatment groups, overall, the ARIES students achieve roughly four times the gain of their control counterparts. In particular, ARIES students had much greater gains for the concepts that the control students found most difficult. (Contains 1 table and 2 figures.)