This report presents a review of the statistical practices of 30 journals representative of the second language field. A review of 150 articles showed a number of prevalent statistical violations including incomplete reporting of reliability, validity, non-significant results, effect sizes, and assumption checks as well as making inferences from…

Selection bias is a persistent, and often hidden, problem in educational research. It is the primary obstacle standing in between increasingly available large education datasets and the ability to make valid causal inferences to inform policymaking, research, and practice (Stuart, 2010). This article provides an accessible discussion on the…

The purpose of this article is to clarify the discrepancy in the use of "queer" as a sexual identity classification in education survey research. This study extends the work completed by Dugan and Yurman (2011), who empirically demonstrated problems with treating LGB students as a homogenous population through collapsing all respondents…

Educational researchers commonly use the rule of thumb of "design effect smaller than 2" as the justification of not accounting for the multilevel or clustered structure in their data. The rule, however, has not yet been systematically studied in previous research. In the present study, we generated data from three different models…

This methodological note illustrates how a commonly used calculation of the Delta-p statistic is inappropriate for categorical independent variables, and this note provides users of logistic regression with a revised calculation of the Delta-p statistic that is more meaningful when studying the differences in the predicted probability of an…

Education research rarely lends itself to large scale experimental research and true randomization, leaving the researcher to quasi-experimental designs. The problem with quasi-experimental research is that underlying factors may impact group selection and lead to potentially biased results. One way to minimize the impact of non-randomization is…

Per definition, CSCL research deals with the data of individuals nested in groups, and the influence of a specific learning setting on the collaborative process of learning. Most well-established statistical methods are not able to analyze such nested data adequately. This article describes the problems which arise when standard methods are…

Among the varied analytical challenges institutional researchers face, examining faculty pay may be one of the most vexing. Although the literature on faculty compensation analysis dates back to the 1970s (Loeb and Ferber, 1971; Gordon, Morton, and Braden, 1974; Scott, 1977; Braskamp and Johnson, 1978; McLaughlin, Smart, and Montgomery, 1978),…

Researchers sometimes use stepwise methods to eliminate variables from analyses when the variables do not appreciably improve the ability to explain or predict inferences about the importance of various predictor variables. It is argued that stepwise methods are usually not appropriate for these purposes for three reasons. First, some researchers…

Argues that Gottlieb rejects population sampling and statistical analyses of distributions as he proposes that his experimental brand of mechanistic science is the only legitimate approach to developmental research. Maintains that Gottlieb exaggerates developmental uncertainty, based on his own research with extreme environmental manipulations.…

Researchers persist in using stepwise regression in spite of problems with this approach. As noted by B. Thompson (1995), three problems accompany the use of stepwise applications. The first is that computer packages may use incorrect degrees of freedom in their computations, resulting in a greater likelihood of obtaining a spurious statistical…

A meta-analysis of outcomes from 32 studies investigating pretest effects was conducted. For all outcomes the average effect size was +.22, indicating an elevating effect of pretest on posttest. Duration of time between pre- and posttesting was also related to effect size. Researchers should continue to include pretest as a design variable.…

Examines difficulties with citation analysis as it is used to study citation frequency, usually for the evaluation of scientists, publications, or institutions. Topics addressed include selection of a control set of papers, comparisons of different types of papers (e.g., methodological or theoretical), effects of independent variables, and use of…

It has been increasingly realized that (1) multivariate methods are essential in most quantitative studies (Fish, 1988; Thompson, 1992), and (2) all conventional parametric analytic methods are correlational and invoke least squares weights (e.g., the beta weights in regression) (Knapp, 1978; Thompson, 1991). The present paper reviews one very…

Discusses procedures and considerations involved with secondary analyses of longitudinal databases. Procedures involve (1) formulating questions; (2) creating a feasibility matrix; (3) reformulating questions; (4) creating derived variables; (5) performing data reduction; (6) analyzing data; and (7) interpreting results. Problems associated with…