This article introduces an innovative sampling scheme, the median sampling (MS), utilizing individual observations over time to efficiently estimate the mean of a process characterized by a symmetric (non-uniform) probability distribution. The mean estimator based on MS is not only unbiased but also boasts enhanced precision compared to its simple…

When scores are used to make decisions about respondents, it is of interest to estimate classification accuracy (CA), the probability of making a correct decision, and classification consistency (CC), the probability of making the same decision across two parallel administrations of the measure. Model-based estimates of CA and CC computed from the…

In this report, we analyze students' learning of compound probability by describing connections they generated while engaged with tasks involving two independent events. Several of their connections were compatible with the development of expertise, such as recognizing the need to determine sample spaces across a variety of situations and noting…

Background: The generalizability of a study's results continues to be at the forefront of concerns in evaluation research in education (Tipton & Olsen, 2018). Over the past decade, statisticians have developed methods, mainly based on propensity scores, to improve generalizations in the absence of random sampling (Stuart et al., 2011; Tipton,…

The American School District Panel (ASDP) is the newest addition to the RAND Corporation's American Educator Panels (AEP) and was designed to survey district leaders several times each school year. RAND recruits ASDP members using probabilistic sampling methods, which allow researchers to weight survey results to generalize to the national…

The current literature on test equating generally defines it as the process necessary to obtain score comparability between different test forms. The definition is in contrast with Lord's foundational paper which viewed equating as the process required to obtain comparability of measurement scale between forms. The distinction between the notions…

Not knowing the population size is a common problem in data-limited contexts. Drawing on work in Sierra Leone, this short take outlines a four-step solution to this problem: (1) estimate the population size using expert interviews; (2) verify estimates using interviews with participants sampled; (3) triangulate using secondary data; and (4)…

Expected yield rates are essential to a survey's data collection plan, as they inform requisite sample sizes to meet the survey's objectives. Given an overall expected yield rate for a self-administered mail survey, this short take describes a simple method for using the Census Planning Database to assign differential yield rates to lower-level…

When planning a cluster randomized trial, evaluators often have access to an enumerated cohort representing the target population of clusters. Practicalities of conducting the trial, such as the need to oversample clusters with certain characteristics in order to improve trial economy or support inferences about subgroups of clusters, may preclude…

This study investigates a range of non-normative ideas that pre-service teachers (PSTs) employ in reasoning about sampling variability. This issue was studied in the context of a content course on statistics and probability for pre-service middle grade teachers at a Midwestern American university. Analysis of seven PSTs' video and screen records…

Over the past ten years, propensity score methods have made an important contribution to improving generalizations from studies that do not select samples randomly from a population of inference. However, these methods require assumptions and recent work has considered the role of bounding approaches that provide a range of treatment impact…

Past studies have documented some pre-service teachers' (PSTs) difficulties in reasoning about sampling variability. This study adds to the body of literature by investigating the ideas that PSTs employ in reasoning about sampling variability, and by conjecturing what is behind the difficulties especially during the contextuality episodes. This…

Place-based initiatives often use resident surveys to inform and evaluate interventions. Sampling based on well-defined sampling frames is important but challenging for initiatives that target subpopulations. Databases that enumerate total population counts can produce overinclusive sampling frames, resulting in costly outreach to ineligible…

The American School District Panel (ASDP) is a nationally representative set of more than 1,000 school district leaders who agree to take surveys over time. RAND recruits ASDP members using probabilistic sampling methods, which allow researchers to weight survey results to generalize to the national population of school districts and charter…

When constructed-response items are administered for a second time, it is necessary to evaluate whether the current Time B administration's raters have drifted from the scoring of the original administration at Time A. To study this, Time A papers are sampled and rescored by Time B scorers. Commonly the scores are compared using the proportion of…