Educators and researchers strive to build policies and practices on data and evidence, especially on academic achievement scores. When assessment scores are inaccurate for specific student populations or when scores are inappropriately used, even data-driven decisions will be misinformed. To maximize the impact of the research-practice-policy…

Panahi and Mohebbi review James Dean Brown's 50-years of research in language testing, curriculum development and research statistics with reference to an impressionistic framework for analysis containing two components with their subcomponents: Annotations (i.e., briefing and implications) and main concepts and themes (i.e., testing and teaching…

There is significant controversy over what counts as evidence in the evaluation of social interventions. It is increasingly common to use methodological criteria to rank evidence types in a hierarchy, with Randomised Controlled Trials (RCTs) at or near the highest level. Because of numerous challenges to a hierarchical approach, this article…

This article proposes that sampling design effects have potentially huge unrecognized impacts on the results reported by large-scale district and state assessments in the United States. When design effects are unrecognized and unaccounted for they lead to underestimating the sampling error in item and test statistics. Underestimating the sampling…

This review presents a novel framework for evaluating evidence based on a set of parallel criteria that can be applied to both group and single-subject experimental design (SSED) studies. The authors illustrate use of this evaluation system in a systematic review of 67 articles investigating social skills interventions for preschoolers with autism…

The important but little understood problem that motivated this study was the lack of research on valid assessment methods to determine progress in higher-order learning in situations involving complex and ill-structured problems. Without a valid assessment method, little progress can occur in instructional design research with regard to designing…

This article provides two illustrations of some of the factors that can influence findings from pre- and post-test research designs in evaluation studies, including regression to the mean (RTM), maturation, history and test effects. The first illustration involves a re-analysis of data from a study by Marsden (2004), in which pre-test scores are…

D'Eon et al. concluded that change in performance self-assessment means from before to after a workshop can detect workshop success in their and other situations. In this commentary, their recommendation is refuted by showing that (a) self-assessments with balanced over- and underestimations are still biased and should not be used to evaluate…

This article on evaluation perspectives and procedures is divided into six sections. The first section briefly discusses qualitative and quantitative research and evaluation. In the second section there is an exploration of the utility and validity of a checklist that can be used to evaluate products, as an instrument for evaluating producers, for…

A discussion of the evaluation of bilingual education programs focuses on building a comprehensive framework for local efforts at evaluation. The discussion begins with an introduction to the legislative history of bilingual education programs and the evolution of their evaluation. This is followed by a review of the literature on current…

Threats to the validity of the Title I Evaluation and Reporting System are covered in two parts: an annotated bibliography of reports concerned with technical issues, and a discussion of threats to validity--from the reporting system in general and from each model in particular. Threats common to all three evaluation models are reported:…

This paper examines issues in designing post-school follow-up studies in special education. The examination focuses on survey research techniques, which are widely used in the investigation of post-school adjustment of former students with handicaps. In special education, survey research studies are used commonly to address many important…

In a study designed to validate oral language proficiency tests, it is planned to administer a series of tests to 100 native Mandarin Chinese-speaking subjects (foreign students and their spouses). The tests will measure communicative competence in speaking (ability to speak, exhibiting control of linguistic, sociolinguistic, and pragmatic rules;…

This handbook has been prepared to address one specific problem: How can the techniques of program evaluation be effectively applied to produce useful information about the operations and impact of moral education programs in secondary schools? The handbook is divided into five chapters. The introductory chapter describes the background and…

Traditionally, the use of control groups, regression discontinuity/projection, or norm referenced models has been advocated for educational program evaluation. The small numbers of and the heterogeneous characteristics of severely and profoundly handicapped make these evaluation methods inappropriate for this population subgroup. Furthermore, no…