For binary experimental data, we discuss randomization-based inferential procedures that do not need to invoke any modeling assumptions. We also introduce methods for likelihood and Bayesian inference based solely on the physical randomization without any hypothetical super population assumptions about the potential outcomes. These estimators have…

In this paper, we investigate two purported problems with Bayesian Knowledge Tracing (BKT), a popular statistical model of student learning: "identifiability" and "semantic model degeneracy." In 2007, Beck and Chang stated that BKT is susceptible to an "identifiability problem"--various models with different…

Inferring causal effects of treatments is a central goal in many disciplines. The potential outcomes framework is a main statistical approach to causal inference, in which a causal effect is defined as a comparison of the potential outcomes of the same units under different treatment conditions. Because for each unit at most one of the potential…

In education research, small samples are common because of financial limitations, logistical challenges, or exploratory studies. With small samples, statistical principles on which researchers rely do not hold, leading to trust issues with model estimates and possible replication issues when scaling up. Researchers are generally aware of such…

One of six summaries of workshop sessions (See TM 000 130), designed to strengthen the evaluation of costly programs and their effects, this handbook presents an analysis of both random and nonrandom sampling errors by application of the Bayesian model. This model attempts to formalize the process and procedures of inference from data through…

Concludes that for most researchers trained in classical statistics, the use of the Bayesian approach requires substantial retooling. Observes that the technical details of the Bayesian approach are formidable, and will require studying textbooks, applications, and computer packages, as well as consulting colleagues. (Author/JDH)

Statistical significance is not a sufficient condition for claiming a hypothesis has been supported. Constructive replications are more important. Statistically significant results may be meaningless while a sequence of nonsignificant results may be quite important. Gives advice on how to overcome some limitations of classifical statistical…

Trade-offs are implicit in choosing a subjective or objective method for evaluating social programs. The differences between Bayesian and traditional statistics, decision and cost-benefit analysis, and anthropological and traditional case systems illustrate trade-offs in choosing methods because of limited resources. (SLD)

Three separate papers are included in this report. The first describes a two-stage procedure for choosing from among several instructional programs the one which maximizes the probability of passing the test. The second gives the exact sample sizes required to determine whether a squared multiple correlation coefficient is above or below a known…

Four researchers at the Educational Testing Service describe what they consider some of the most vexing research problems they face. While these problems are not completely statistical, they all have major statistical components. Following the introduction (section 1), in section 2, "Problems with the Simultaneous Estimation of Many True…