Educators who are urged to use evidence-based practices to improve instruction often end up disappointed at the results, which fall short of those touted in the research and by the What Works Clearinghouse. Stanley Pogrow explains how common strategies researchers use to demonstrate evidence of success, such as statistical significance of or…

A class of effect size indices are discussed that evaluate the degree to which two nested confirmatory factor analysis models differ from each other in terms of fit to a set of observed variables. These descriptive effect measures can be used to quantify the impact of parameter restrictions imposed in an initially considered model and are free…

This supplement concerns Appendix E of the "What Works Clearinghouse (WWC) Procedures Handbook, Version 4.1." The supplement extends the range of designs and analyses that can generate effect size and standard error estimates for the WWC. This supplement presents several new standard error formulas for cluster-level assignment studies,…

Assessing publication bias is a critical procedure in meta-analyses for rating the synthesized overall evidence. Because statistical tests for publication bias are usually not powerful and only give "P" values that inform either the presence or absence of the bias, examining the asymmetry of funnel plots has been popular to investigate…

In this article, a 2-way multigroup common factor model (MG-CFM) is presented. The MG-CFM can be used to estimate interaction effects between 2 grouping variables on 1 or more hypothesized latent variables. For testing the significance of such interactions, a likelihood ratio test is presented. In a simulation study, the robustness of the…

A new effect size representing the predictive power of an independent variable from a multiple regression model is presented. The index, denoted as r[subscript sp], is the semipartial correlation of the predictor with the outcome of interest. This effect size can be computed when multiple predictor variables are included in the regression model…

This methodological brief introduces the readers to the propensity score matching method, which can be used for enhancing the validity of causal inferences in research situations involving nonexperimental design or observational research, or in situations where the benefits of an experimental design are not fully realized because of reasons beyond…

The work reported in this paper reflects a collaborative effort of many individuals representing multiple organizations. It began during a session at the October 2008 meeting of TILSA when a representative of a member state asked the group if any of their programs had experienced unexpected fluctuations in the annual state assessment scores, and…

Most psychology journals now require authors to report a sample value of effect size along with hypothesis testing results. The sample effect size value can be misleading because it contains sampling error. Authors often incorrectly interpret the sample effect size as if it were the population effect size. A simple solution to this problem is to…

Standard least squares analysis of variance methods suffer from poor power under arbitrarily small departures from normality and fail to control the probability of a Type I error when standard assumptions are violated. This article describes a framework for robust estimation and testing that uses trimmed means with an approximate degrees of…

The authors describe and illustrate 6 factors that affect the size of a Pearson correlation: (a) the amount of variability in the data, (b) differences in the shapes of the 2 distributions, (c) lack of linearity, (d) the presence of 1 or more "outliers," (e) characteristics of the sample, and (f) measurement error. Also discussed are ways to…

One version of r-sub(equivalent), calculated from Fisher's exact test p values and recommended for small samples, is considered "a more realistic... [and] a more accurate estimate of the population correlation than... the sample correlation, r-sub(sample)" (R. Rosenthal & D. B. Rubin, 2003, p. 494). Small sample properties of r-sub(sample) and of…

The computation and reporting of effect size estimates is becoming the norm in many journals in psychology and related disciplines. Despite the increased importance of effect sizes, researchers may not report them or may report inaccurate values because of a lack of appropriate computational tools. For instance, Pierce, Block, and Aguinis (2004)…

Single studies, by themselves, rarely explain the effect of treatments or interventions definitively in the social sciences. Researchers created meta-analysis in the 1970s to address this need. Since then, meta-analytic techniques have been used to support certain treatment modalities and to influence policymakers. Although these techniques…

This paper discusses the impact of sampling error on the construction of confidence intervals around effect sizes. Sampling error affects the location and precision of confidence intervals. Meta-analytic resampling demonstrates that confidence intervals can haphazardly bounce around the true population parameter. Special software with graphical…