NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: EJ1475702
Record Type: Journal
Publication Date: 2025-Aug
Pages: 36
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-0049-1241
EISSN: EISSN-1552-8294
Available Date: 0000-00-00
The Mixed Subjects Design: Treating Large Language Models as Potentially Informative Observations
Sociological Methods & Research, v54 n3 p1074-1109 2025
Large language models (LLMs) provide cost-effective but possibly inaccurate predictions of human behavior. Despite growing evidence that predicted and observed behavior are often not "interchangeable," there is limited guidance on using LLMs to obtain valid estimates of causal effects and other parameters. We argue that LLM predictions should be treated as potentially informative observations, while human subjects serve as a gold standard in a "mixed subjects design." This paradigm preserves validity and offers more precise estimates at a lower cost than experiments relying exclusively on human subjects. We demonstrate--and extend--prediction-powered inference (PPI), a method that combines predictions and observations. We define the "PPI correlation" as a measure of interchangeability and derive the "effective sample size" for PPI. We also introduce a power analysis to optimally choose between "informative but costly" human subjects and "less informative but cheap" predictions of human behavior. Mixed subjects designs could enhance scientific productivity and reduce inequality in access to costly evidence.
SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A
Author Affiliations: 1Department of Sociology, Stanford University, Stanford, CA, USA; 2Department of Statistics, Stanford University, Stanford, CA, USA; 3MIT Sloan School of Management, Cambridge, MA, USA