Educators and researchers strive to build policies and practices on data and evidence, especially on academic achievement scores. When assessment scores are inaccurate for specific student populations or when scores are inappropriately used, even data-driven decisions will be misinformed. To maximize the impact of the research-practice-policy…

This supplement concerns Appendix E of the "What Works Clearinghouse (WWC) Procedures Handbook, Version 4.1." The supplement extends the range of designs and analyses that can generate effect size and standard error estimates for the WWC. This supplement presents several new standard error formulas for cluster-level assignment studies,…

A valuable extension of the single-rating regression discontinuity design (RDD) is a multiple-rating RDD (MRRDD). To date, four main methods have been used to estimate average treatment effects at the multiple treatment frontiers of an MRRDD: the "surface" method, the "frontier" method, the "binding-score" method, and…

This article discusses the nonuse, misuse, and proper use of pilot studies in experimental evaluation research. The authors first show that there is little theoretical, practical, or empirical guidance available to researchers who seek to incorporate pilot studies into experimental evaluation research designs. The authors then discuss how pilot…

In some instances, intentionally or not, study designs are such that there is clustering in one group but not in the other. This paper describes methods for computing effect size estimates and their variances when there is clustering in only one group and the analysis has not taken that clustering into account. The authors provide the effect size…

This article proposes that sampling design effects have potentially huge unrecognized impacts on the results reported by large-scale district and state assessments in the United States. When design effects are unrecognized and unaccounted for they lead to underestimating the sampling error in item and test statistics. Underestimating the sampling…

Effective policymaking requires understanding the causal effects of competing proposals. Relevant causal quantities include proposals' expected effect on different groups of recipients, the impact of policies over time, the potential trade-offs between competing objectives, and, ultimately, the optimal policy. This dissertation studies causal…

The constraints of conducting evaluations in real-world settings often necessitate the implementation of less than ideal designs. Unfortunately, the standard method for estimating the precision of a result (i.e., confidence intervals [CI]) cannot be used for evaluative conclusions that are derived from multiple indicators, measures, and data…

The article summarizes the results of an evaluation of the accuracy of statewide industry and occupational employment projections for 20 states. The authors provide some recommendations, based on evaluation results, to improve subsequent rounds of statewide projections. (CH)

Effects of response guided experimentation in applied behavior analysis on Type I error rates are explored. Data from T. A. Matyas and K. M. Greenwood (1990) suggest that, when visual inspection is combined with response guided experimentation, Type I error rates can be as high as 25%. (SLD)

In examining considerations in determining sample sizes for factor analyses, attention was given to the effects of outliers; the standard error of correlations, and their effect on factor structure; sample heterogeneity; and the misuse of rules of thumb for sample sizes. (Author)

The AP Course Audit utilizes a criterion-based professional judgment method of analysis within a nested multi-step review process. The overall goal of the methodology is to yield a final judgment on each syllabus that is ultimately valid. While reviewer consistency is an important consideration, the most important goal is to reach a final judgment…

Sampling distributions for ten tests for comparing population variances in a two group design were generated for several combinations of equal and unequal sample sizes, population means, and group variances when distributional forms differed. The ten procedures included: (1) O'Brien's (OB); (2) O'Brien's with adjusted degrees of freedom; (3)…

Six models for evaluating change are examined via a Monte Carlo study. All six models show a lack of power. A modified analysis of variance procedure is suggested as an alternative. (JKS)

Presents a general framework for designing sampling programs that ensure cost effectiveness, and managed errors kept within known and acceptable limits. (LZ)