Good practice in primary research has evolved over many decades of research in applied linguistics to counter human fallibility and biases. Surprisingly, perhaps, synthesising such research in an entire field has only recently started to develop its own methodologies and recommendations. This paper outlines some of the issues involved, especially…

Problems inherent in relying solely on statistical significance testing as a means of data interpretation are reviewed. The biggest problem with statistical significance testing is that researchers have used the results of this testing to ascribe importance or meaning to their studies where such meaning often does not exist. Often researchers…

Researchers increasingly recognize that significance tests are limited in their ability to inform scientific practice. Common errors in interpreting significance tests and three strategies for augmenting the interpretation of significance test results are illustrated. The first strategy for augmenting the interpretation of significance tests…

The purpose of this study was to investigate the methodology of empirical counseling supervision articles published from 1981 to 1987. Seventy-five studies met the criteria for inclusion. Fifty-five articles included adequate information about at least one inferential statistical test to permit computing eta(sup 2), epsilon(sup 2), power, and…

As an extension of B. Thompson's 1998 invited address to the American Educational Research Association, this paper cites two additional common faux pas in research methodology and explores some research issues for the future. These two errors in methodology are the use of univariate analyses in the presence of multiple outcome variables (with the…

Monte Carlo studies explored the sampling characteristics of Cohen's d and three approximations to Cohen's d when used as average effect size measures in meta-analysis. Reviews of 10, 100, and 500 studies (M) were simulated, with degrees of freedom (df) varied in seven steps from 8 to 58. In a two independent groups design, samples were obtained…

Three criticisms of overreliance on results from statistical significance tests are noted. It is suggested that: (1) statistical significance tests are often tautological; (2) some uses can involve comparisons that are not completely sensible; and (3) using statistical significance tests to evaluate both methodological assumptions (e.g., the…

This paper examines the problems caused by relying solely on statistical significance tests to interpret results in contemporary social science. The place of significance testing in educational research has often been debated. Among the problems in reporting statistical significance are questions of definition and terminology. Problems are also…

A meta-analysis was conducted on 44 educational studies that used either a (labelled) Hawthorne control group, a manipulation of Hawthorne effects, or a group designed to control for the Hawthorne effect. The sample included published journal articles, ERIC documents or unpublished papers, and dissertations. The studies were coded on 20 variables,…

In the educational literature, responses to surveys commonly serve as the source of data for many empirical articles. Whenever a survey is used as a source of data, the response rate can greatly affect the potential generalizability of the findings. Using Monte Carlo methods, this study examined the effects on sample estimates of the population…

Problems with using stepwise analytic methods are discussed, and better alternatives are illustrated. To make the illustrations concrete, an actual data set, involving responses of 91 medical school admissions directors to 30 variables, was used. The 30 variables involved perceptions of barriers to medical school with respect to characteristics of…

Reported in this paper are the findings of five meta-analyses assessing the relationships among children's interpersonal cognitive problem-solving (ICPS) skills, training, and behavioral adjustment. Five hypotheses were examined: (1) Adjusted children score higher on ICPS measures than do nonadjusted children; (2) Children trained in ICPS skills…

The term "statistical significance" is often misunderstood or abused to imply a large effect size. A recent example is in the work of J. P. Rushton (1988, 1990) on differences between Negroids and Caucasoids. Rushton used brain size and cranial size as indicators of intelligence, using Pearson "r"s ranging from 0.03 to 0.35.…

This paper focuses on a critical analysis and reanalysis of two earlier multisection validity meta-analyses of student ratings of instruction. The reanalysis is based on a sharpening of criteria for: (1) including studies in the meta-analysis; (2) defining key study characteristics that influence results; and (3) calculating the correlational…

Because research synthesis enables one to determine either the overall effectiveness of a particular treatment or the relative effectiveness of different types of treatments, it is becoming increasingly popular as a tool in program evaluation. Numerous methodological problems arise, however, when research synthesis is applied to studies conducted…