This article introduces an innovative sampling scheme, the median sampling (MS), utilizing individual observations over time to efficiently estimate the mean of a process characterized by a symmetric (non-uniform) probability distribution. The mean estimator based on MS is not only unbiased but also boasts enhanced precision compared to its simple…

When scores are used to make decisions about respondents, it is of interest to estimate classification accuracy (CA), the probability of making a correct decision, and classification consistency (CC), the probability of making the same decision across two parallel administrations of the measure. Model-based estimates of CA and CC computed from the…

In this report, we analyze students' learning of compound probability by describing connections they generated while engaged with tasks involving two independent events. Several of their connections were compatible with the development of expertise, such as recognizing the need to determine sample spaces across a variety of situations and noting…

Background: The generalizability of a study's results continues to be at the forefront of concerns in evaluation research in education (Tipton & Olsen, 2018). Over the past decade, statisticians have developed methods, mainly based on propensity scores, to improve generalizations in the absence of random sampling (Stuart et al., 2011; Tipton,…

The American School District Panel (ASDP) is the newest addition to the RAND Corporation's American Educator Panels (AEP) and was designed to survey district leaders several times each school year. RAND recruits ASDP members using probabilistic sampling methods, which allow researchers to weight survey results to generalize to the national…

Expected yield rates are essential to a survey's data collection plan, as they inform requisite sample sizes to meet the survey's objectives. Given an overall expected yield rate for a self-administered mail survey, this short take describes a simple method for using the Census Planning Database to assign differential yield rates to lower-level…

When planning a cluster randomized trial, evaluators often have access to an enumerated cohort representing the target population of clusters. Practicalities of conducting the trial, such as the need to oversample clusters with certain characteristics in order to improve trial economy or support inferences about subgroups of clusters, may preclude…

This study investigates a range of non-normative ideas that pre-service teachers (PSTs) employ in reasoning about sampling variability. This issue was studied in the context of a content course on statistics and probability for pre-service middle grade teachers at a Midwestern American university. Analysis of seven PSTs' video and screen records…

Past studies have documented some pre-service teachers' (PSTs) difficulties in reasoning about sampling variability. This study adds to the body of literature by investigating the ideas that PSTs employ in reasoning about sampling variability, and by conjecturing what is behind the difficulties especially during the contextuality episodes. This…

Place-based initiatives often use resident surveys to inform and evaluate interventions. Sampling based on well-defined sampling frames is important but challenging for initiatives that target subpopulations. Databases that enumerate total population counts can produce overinclusive sampling frames, resulting in costly outreach to ineligible…

The American School District Panel (ASDP) is a nationally representative set of more than 1,000 school district leaders who agree to take surveys over time. RAND recruits ASDP members using probabilistic sampling methods, which allow researchers to weight survey results to generalize to the national population of school districts and charter…

When constructed-response items are administered for a second time, it is necessary to evaluate whether the current Time B administration's raters have drifted from the scoring of the original administration at Time A. To study this, Time A papers are sampled and rescored by Time B scorers. Commonly the scores are compared using the proportion of…

In this article, we explore the use of Facebook targeted advertisements for the collection of survey data. We illustrate the potential of survey sampling and recruitment on Facebook through the example of building a large employee--employer linked data set as part of The Shift Project. We describe the workflow process of targeting, creating, and…

When analyzing large-scale assessments (LSAs) that use complex sampling designs, it is important to account for probability sampling using weights. However, the use of these weights in multilevel models has been widely debated, particularly regarding their application at different levels of the model. Yet, no consensus has been reached on the best…

Recent years have seen a growing number of studies investigating the accuracy of nonprobability online panels; however, response quality in nonprobability online panels has not yet received much attention. To fill this gap, we investigate response quality in a comprehensive study of seven nonprobability online panels and three probability-based…