Data have long been considered a key factor in organizational decision-making (Simon, 1955; Lindblom & Cohen, 1979). Data offer perspective, guidance, and insights that inform policy and practice (Newell & Simon, 1972; Kennedy, 1984). Recently, education policymakers have invested in the use of data for organizational improvement in states…

E-learning researchers face considerable challenges in creating meaningful and generalizable studies due to the complex nature of this dynamic training medium. Our experience in conducting workplace e-learning research led us to create this guide for planning research on e-learning. We share the unanticipated complications we encountered in our…

Difficulties often encountered in conducting surveys in educational technology are described, and some guidelines are suggested for the design and implementation of a survey instrument. (RAO)

The National Assessment of Educational Progress (NAEP) has completed two assessments of mathematics, the first conducted in 1972-73 and the second during 1977-78. Each assessment surveyed the mathematics achievement of American 9-, 13-, and 17-year-olds, using a deeply stratified, multi-stage probability sample design. This report documents…

Three questions are discussed: (1) what approach will be taken to gain and maintain access to school districts and schools to administer assessment instruments? (2) what procedures should be employed in the difficult and complex task of gaining access to nonstudents (age 17)? and (3) what problems are likely to be encountered in doing this? Access…

In 1982, the Educational Testing Service (ETS) proposed to implement a new, complex design for the National Assessment of Educational Progress (NAEP). The major features of this design are described in "A New Design for a New Era" (Messick, Beaton, and Lord, 1983). The purpose of this document is to describe the actual implementation of…

The detrimental effects of nonresponse bias are particularly significant given the widespread use of the survey data collection method in educational surveys. Current methods for remediating nonresponse bias in educational surveys are explored and critiqued. (Author/LMO)

This final report summarizes Year 11 quality check activities for the National Assessment of Educational Progress (NAEP). A probability sample of 40 schools was selected for quality check purposes from all three age classes. One regular school was selected for each District Supervisor at each Age Class. Quality check activities were conducted in…

Procedures for implementing the Pennsylvania Plan (see TM 000 608) are detailed, and the method of achieving representative sampling is described. Functions of school representatives involved in the data collection are listed, and their training is outlined. Procedures for collecting school and teacher information, and administering tests and…

The sampling designs and procedures used by the National Assessment of Educational Progress in its second year assessment activities are documented. The report is organized in three parts. Project objectives and other background information are discussed in the first section. The details of the second year school and household samples are…

Since 1969 the National Assessment of Educational Progress (NAEP) has conducted annual assessments to determine, in the language of the current legislation, "the performance of children and young adults in the basic skills of reading, mathematics and communications" and to "report data periodically on changes in the knowledge and…

The purpose of this paper is to provide an overview of the analysis of data collected by the National Assessment of Educational Progress (NAEP). In simplest terms, the analysis can be characterized as establishing baseline estimates of the percentages of young Americans possessing certain skills, knowledge, understandings, and attitudes and…

Multiple matrix sampling (MMS) theory indicates how data may be gathered to most efficiently convey information about levels of attainment in a population, but standard analyses of these data require random sampling of items from a fixed pool of items. This assumption proscribes the retirement of flawed or obsolete items from the pool as well as…

The effects of certain characteristics (e.g., sample design) of National Assessment of Educational Progress (NAEP) data on statistical analysis techniques are considered. Ignoring special features of NAEP data and proceeding with a standard analysis can produce inferences that underestimate the true variability and overestimate the true degrees of…

The use of outcome assessment in special education is reviewed. Outcome assessments differ widely in design, measures, and data collection, but they share a common focus on outcomes as individual achievements, statuses, or behaviors. Assessments of special education outcomes often have an evaluative purpose in reflecting how well the special…