ERIC Number: EJ1447304
Record Type: Journal
Publication Date: 2024-Nov
Pages: 27
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-1759-2879
EISSN: EISSN-1759-2887
Available Date: N/A
An Evaluation of the Performance of Stopping Rules in AI-Aided Screening for Psychological Meta-Analytical Research
Research Synthesis Methods, v15 n6 p1120-1146 2024
Several AI-aided screening tools have emerged to tackle the ever-expanding body of literature. These tools employ active learning, where algorithms sort abstracts based on human feedback. However, researchers using these tools face a crucial dilemma: When should they stop screening without knowing the proportion of relevant studies? Although numerous stopping rules have been proposed to guide users in this decision, they have yet to undergo comprehensive evaluation. In this study, we evaluated the performance of three stopping rules: the knee method, a data-driven heuristic, and a prevalence estimation technique. We measured performance via sensitivity, specificity, and screening cost and explored the influence of the prevalence of relevant studies and the choice of the learning algorithm. We curated a dataset of abstract collections from meta-analyses across five psychological research domains. Our findings revealed performance differences between stopping rules regarding all performance measures and variations in the performance of stopping rules across different prevalence ratios. Moreover, despite the relatively minor impact of the learning algorithm, we found that specific combinations of stopping rules and learning algorithms were most effective for certain prevalence ratios of relevant abstracts. Based on these results, we derived practical recommendations for users of AI-aided screening tools. Furthermore, we discuss possible implications and offer suggestions for future research.
Descriptors: Artificial Intelligence, Psychological Studies, Researchers, Screening Tests, Research Methodology, Experimenter Characteristics, Performance Based Assessment, Evaluation Criteria, Evaluation Methods
Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A
Data File: URL: https://osf.io/7yhrq/
Author Affiliations: N/A