Large-scale randomized studies provide the best means of evaluating practical, replicable approaches to improving educational outcomes. This article discusses the advantages, problems, and pitfalls of these evaluations, focusing on alternative methods of randomization, recruitment, ensuring high-quality implementation, dealing with attrition, and…

Business analytics courses, such as marketing research, data mining, forecasting, and advanced financial modeling, have substantial predictive modeling components. The predictive modeling in these courses requires students to estimate and test many linear regressions. As a result, false positive variable selection ("type I errors") is…

With growing popularity of single subject design as a method to evaluate the efficacy of interventions, it is important to ensure that the analyses of these methods are rigorous and reliable. The purpose of this case study is to discuss the measures used to evaluate the efficacy of interventions in single subject design studies in the fields of…

This addendum to previously published results presents alternative analyses of data from large-scale effectiveness studies of Cognitive Tutor Algebra I in middle schools and high schools. These alternative analyses produce results that are substantively the same as previously reported. We find a significant positive effect of 0.21 standard…

This article provides a conceptual introduction to the issues surrounding the analysis of clustered (nested) data. We define the intraclass correlation coefficient (ICC) and the design effect, and we explain their effect on the standard error. When the ICC is greater than 0, then the design effect is greater than 1. In such a scenario, the…

The Illinois Initiative on Transparency in Learning and Teaching is a grassroots assessment project designed to promote students' conscious understanding of how they learn and to enable faculty to gather, share, and promptly benefit from data about students' learning by coordinating their efforts across disciplines, institutions, and countries.…

This article discusses issues associated with statistical testing conducted with data from clustered school samples. Empirical researchers often conduct tests of statistical inference on sample data to ascertain the extent to which differences exist within groups in the population. Typically, much school-related data are collected from students.…

This report provides a description of the student characteristics, persistence, and degree attainment of a nationally representative sample of students who began postsecondary education for the first time during the 2003-04 academic year. The report describes the background, academic preparation, and experience of these beginning students over 3…

Over the years, methodologists have been recommending that researchers use magnitude of effect estimates in result interpretation to highlight the distinction between statistical and practical significance (cf. R. Kirk, 1996). A magnitude of effect statistic (i.e., effect size) tells to what degree the dependent variable can be controlled,…

Outlines concerns that must be addressed by those who advocate replacing statistical hypothesis-testing with alternative data-analysis strategies. Suggests that commonly recommended alternatives are not perfect and that various hypothesis-testing modifications can be implemented to make the process and its conclusions more credible. Hypothesis…

This report summarizes the findings of an analysis of a series of action research projects conducted by Goshen Community Schools at the elementary, middle, and high school levels. During the 2008-2009 school year, 40 teachers participated in independent action research studies regarding the extent to which a six step approach to direct vocabulary…

This paper summarizes an experiment conducted to examine the counting performance of 7- and 8-year-olds. Analysis of variance was computed on counting errors produced when enumerating a set of squares on a computer screen. The factors included in the analysis were age, gender, array size, error type, proximity, and error form. The primary…

The various costs of Type I and Type II errors of inference from data are discussed. Six methods for minimizing each error type are presented, which may be employed even after data collection for Type I and which minimizes Type II errors by a study design and analytical means combination. (Author/CM)

Demonstrates a statistically valid method allowing students to explore randomization. Described are two examples: counting techniques for a small set of data and simulation for a large sample. (YP)