Various instruments for assessing metacognitive skills and strategy use exist. Off-line self-reports are questionnaires and interviews administered either before or after task performance, while on-line measures are gathered during task performance through thinking aloud or observation. Multi-method studies in reading have shown that off-line methods suffer from serious validity problems, whereas the validity of on-line methods is adequate. Little is known, however, about the validity of methods for assessing metacognition in mathematics. Five instruments were administered to 30 secondary-school students: two prospective questionnaires (MSLQ and ILS) before a mathematics task, two on-line methods (observation and thinking aloud) concurrent to the mathematics task, and a task-specific retrospective questionnaire after the mathematics task. Mathematics performance was assessed by a posttest and GPA. Results confirm that prospective questionnaires have poor convergent and predictive validity in mathematics. Although the retrospective questionnaire does slightly better than prospective questionnaires, the validity of both on-line methods stands out. It is concluded that on-line instruments should be preferred over off-line instruments for the assessment of metacognitive skillfulness in mathematics.