The examination of cross-classified category data is common in evaluation and research, with Karl Pearson's family of chi-square tests representing one of the most utilized statistical analyses for answering questions about the association or difference between categorical variables. Unfortunately, these tests are also among the more commonly…

To date, evaluation research and policy analysis have focused mainly on average program impacts and paid little systematic attention to their variation. Recently, the growing number of multi-site randomized trials that are being planned and conducted make it increasingly feasible to study "cross-site" variation in impacts. Important…

Statistical modeling of school effectiveness data was originally motivated by the dissatisfaction with the analysis of (school-leaving) examination results that took no account of the background of the students or regarded each school as an isolated unit of analysis. The application of multilevel analysis was generally regarded as a breakthrough,…

This article investigates three types of stage-sequential growth mixture models in the structural equation modeling framework for the analysis of multiple-phase longitudinal data. These models can be important tools for situations in which a single-phase growth mixture model produces distorted results and can allow researchers to better understand…

The purpose of this study was to investigate the application of the parallel analysis (PA) method for choosing the number of factors in component analysis for situations in which data are dichotomous or ordinal. Although polychoric correlations are sometimes used as input for component analyses, the random data matrices generated for use in PA…

This document describes the breadth of the research that the ETS (Educational Testing Service) Research & Development division is conducting in 2010. This portfolio will be updated in early 2011 to reflect changes to existing projects and new projects that were added after this document was completed. The research described in this portfolio falls…

This document describes the breadth of the research being conducted in 2008 by the Research and Development Division at Educational Testing Service (ETS). The research described falls into three large categories: (1) Research supported by the ETS research allocation; (2) Research funded by testing programs at ETS; and (3) Research funded by…

Describes and demonstrates analytical techniques used in organizational psychology and contemporary multilevel analysis. Using these analytic techniques, examines the relationship between educational outcomes and the school environment. Finds that at least some indicators might be represented as school-level phenomena. Results imply that the…

Education researchers, practitioners, and policymakers alike are committed to identifying interventions that teach students more effectively. Increased emphasis on evaluation and accountability has increased desire for sound evaluations of these interventions; and at the same time, school-level data have become increasingly available. This article…

To assess the status and the trends of subject matters investigated and research methods/designs and data analysis procedures employed by educational researchers, this study surveyed articles published by the "American Educational Research Journal (AERJ)," "Journal of Experimental Education (JEE)" and "Journal of Educational Research (JER)" from…

This monograph critiques the many nonexperimental impact estimation approaches that have been based on external comparison groups. It proposes an approach to evaluating employment and training (E&T) programs that calls for using the group of individuals who apply to a program but then choose not to participate in that program as an…

Three trends that are projected for the future of evaluation are greater use of evaluation research in litigation; increasing integration of the social and natural sciences for certain kinds of evaluation research; and a move away from conventional statistics toward data analysis. (SLD)

Research practitioners have traditionally collected and analyzed data using experimental designs and inferential statistical procedures. In addition to being difficult to learn and apply, these quantitative procedures generally leave out such variables as attitudes, feelings, and emotions. This paper describes the nature and role of qualitative…

With the continual progress of computer technologies, computer automated scoring (CAS) has become a popular tool for evaluating writing assessments. Research of applications of these methodologies to new types of performance assessments is still emerging. While research has generally shown a high agreement of CAS system generated scores with those…

This research study sought to determine the impact of an agricultural leadership program on rural community development beyond self-report survey data typically collected for program evaluation. Two hundred-ninety graduates of a program that taught leadership skills to impact policy at local, state, and national levels to adults aged 25-45 who…