Background: The generalizability of a study's results continues to be at the forefront of concerns in evaluation research in education (Tipton & Olsen, 2018). Over the past decade, statisticians have developed methods, mainly based on propensity scores, to improve generalizations in the absence of random sampling (Stuart et al., 2011; Tipton,…

Factorial surveys use a population of vignettes to elicit respondents' attitudes or beliefs about different hypothetical scenarios. However, the vignette population is frequently too large to be assessed by each respondent. Experimental designs such as randomized block confounded factorial (RBCF) designs, D-optimal designs, or random sampling…

Meta-analyses are often used to estimate the relative average values of a quantitative outcome in two groups (eg, control and experimental groups). However, they may also examine the relative variability (variance) of those groups. For such comparisons, two relatively new effect size statistics, the log-transformed "variability ratio"…

A multilevel meta-analysis can combine the results of several single-subject experimental design studies. However, the estimated effects are biased if the effect sizes are standardized and the number of measurement occasions is small. In this study, the authors investigated 4 approaches to correct for this bias. First, the standardized effect…

There is a long-standing debate as to whether nonexperimental estimators of causal effects of social programs can overcome selection bias. Most existing reviews either are inconclusive or point to significant selection biases in nonexperimental studies. However, many of the reviews, the so-called "between-studies," do not make direct…

Educational researchers commonly use the rule of thumb of "design effect smaller than 2" as the justification of not accounting for the multilevel or clustered structure in their data. The rule, however, has not yet been systematically studied in previous research. In the present study, we generated data from three different models…

Instrumental variable estimators hold the promise of enabling researchers to estimate the effects of educational treatments that are not (or cannot be) randomly assigned but that may be affected by randomly assigned interventions. Examples of the use of instrumental variables in such cases are increasingly common in educational and social science…

The Programme for International Assessment of Adult Competencies (PIAAC) will establish technical standards and guidelines to ensure that the survey design and implementation processes of PIAAC yield high-quality and internationally comparable data. This document provides a revised version of the technical standards and guidelines originally…

Humphreys' comments on double-median splits (TM 504 009) are essentially correct, but are not relevant to the original Kirby and Das article (EJ 182 444). His comments do not weaken our findings. (Author/RD)

Large scale surveys usually employ a complex sampling design and as a consequence, no standard methods for estimation of the standard errors associated with the estimates of population means are available. Resampling methods, such as jackknife or bootstrap, are often used, with reference to their properties of robustness and reduction of bias. A…

The unbiased estimate of a "treatment effect" reached by analysis of covariance in a nonrandomized experiment would often require that a different covariate be used in each treatment. A sufficient but unlikely condition for an unbiased estimate is that the covariate for each treatment is (1) the complete covariate that predicts the outcome as…

Evaluations involving nonrandom assignment to treatment or control groups are vulnerable to an accidental or intentional confounding of a selection effect with the treatment effect. Two techniques, discriminant analysis and base expectancy analysis, permit separate estimation of the selection and treatment effects in the final results. (Author/CTM)

The National Food and Agriculture Council of the Philippines regularly requires rapid feedback data for analysis, which will assist in monitoring programs to improve and increase the production of selected crops by small scale farmers. Since many other development programs in various subject matter areas also require similar statistical…

An investigation was conducted of the characteristics of two estimation procedures and corresponding test statistics used in the analysis of completely randomized factorial experiments when observations are lost at random. For one estimator, contrast coefficients for cell means did not involve the cell frequencies. For the other, contrast…

This report presents the findings of a research study, conducted by the College of the Mainland (COM) as a subcontractor for Project FOLLOW-UP, designed to test the accuracy of random sampling and to measure non-response bias in mail surveys. In 1975, a computer-generated random sample of 500 students was drawn from a population of 1,256 students…