Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research
Diego Campos, Tim Fütterer, Thomas Gfrörer, Rosa Lavelle-Hill, Kou Murayama, Lars König, Martin Hecht, Steffen Zitzmann, Ronny Scherer
Abstract
Systematic reviews and meta-analyses are crucial for advancing research; yet, they are time-consuming and resource-demanding. Although machine learning and natural language processing algorithms may reduce this time and these resources, their performance has not been tested in the field of education and educational psychology, and there is a lack of clear information on when researchers should stop the reviewing process. In this study, we conducted a retrospective screening simulation using 27 systematic reviews in education and educational psychology. We evaluated the recall, work saved over sampling, and the estimated time savings of several active learning screening algorithms and heuristic stopping criteria. The results showed on average a 50% (SD = 20%) reduction in screening workload when using active learning algorithms for abstract screening and an estimated time savings of 1.64 days (SD = 1.78). The learning algorithm Random Forests with Sentence Bidirectional Encoder Representations from Transformers outperformed other algorithms—a finding that emphasizes the importance of incorporating semantic and contextual information during feature extraction and modeling in the screening process. Furthermore, we found that 95% of all relevant abstracts within a given dataset can be retrieved using heuristic stopping rules. Specifically, an approach that stops the screening process after consecutively classifying 7% of irrelevant papers yielded the most significant gains in terms of works savings over sampling (M = 41%, SD = 26%). However, the performance of the heuristic stopping criteria depended on the active learning algorithm used, the length and the proportion of relevant papers in a database. Overall, our study provides empirical evidence on the performance of machine learning screening algorithms for abstract screening in systematic reviews in education and educational psychology.