ERIC Number: ED596086
Record Type: Non-Journal
Publication Date: 2019-Apr
Pages: 45
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-
EISSN: N/A
Available Date: N/A
Validation Methods for Aggregate-Level Test Scale Linking: A Case Study Mapping School District Test Score Distributions to a Common Scale. CEPA Working Paper No. 16-09
Reardon, Sean F.; Ho, Andrew D.; Kalogrides, Demetra
Stanford Center for Education Policy Analysis
Linking score scales across different tests is considered speculative and fraught, even at the aggregate level (Feuer et al., 1999; Thissen, 2007). We introduce and illustrate validation methods for aggregate linkages, using the challenge of linking U.S. school district average test scores across states as a motivating example. We show that aggregate linkages can be validated both directly and indirectly under certain conditions, such as when the scores for at least some target units (districts) are available on a common test (e.g., the National Assessment of Educational Progress). We introduce precision-adjusted random effects models to estimate linking error, for populations and for subpopulations, for averages and for progress over time. These models allow us to distinguish linking error from sampling variability and illustrate how linking error plays a larger role in aggregates with smaller sample sizes. Assuming that target districts generalize to the full population of districts, we can show that standard errors for district means are generally less than 0.2 standard deviation units, leading to reliabilities above 0.7 for roughly 90% of districts. We also show how sources of imprecision and linking error contribute to both within- and between-state district comparisons within vs. between states. This approach is applicable whenever the essential counterfactual question--"what would means/variance/progress for the aggregate units be, had students taken the other test?"--can be answered directly for at least some of the units.
Descriptors: Test Validity, Evaluation Methods, School Districts, Scores, National Competency Tests, Error of Measurement, Sample Size, Scaling, Hierarchical Linear Modeling, Achievement Tests
Stanford Center for Education Policy Analysis. 520 Galvez Mall, CERAS Building, 5th Floor, Stanford, CA 94305. Tel: 650-736-1258; Fax: 650-723-9931; e-mail: contactcepa@stanford.edu; Web site: http://cepa.stanford.edu
Publication Type: Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: Institute of Education Sciences (ED); Spencer Foundation; Russell Sage Foundation; Bill and Melinda Gates Foundation; Overdeck Family Foundation; William T. Grant Foundation
Authoring Institution: Stanford Center for Education Policy Analysis (CEPA)
Identifiers - Assessments and Surveys: National Assessment of Educational Progress
IES Funded: Yes
Grant or Contract Numbers: R305D110018
Author Affiliations: N/A