Automation bias in AI-assisted detection of cerebral aneurysms on time-of-flight MR angiography
Su Hwan Kim, Severin Schramm, Evamaria Olga Riedel, Lena Schmitzer, Enrike Rosenkranz, Olivia Kertels, Jannis Bodden, Karolin J. Paprottka, Dominik Sepp, Martin Renz, Jan S. Kirschke, Thomas Baum, Christian Maegerlein, Tobias Boeckh‐Behrens, Claus Zimmer, Benedikt Wiestler, Dennis M. Hedderich
Abstract
PURPOSE: To determine how automation bias (inclination of humans to overly trust-automated decision-making systems) can affect radiologists when interpreting AI-detected cerebral aneurysm findings in time-of-flight magnetic resonance angiography (TOF-MRA) studies. MATERIAL AND METHODS: Nine radiologists with varying levels of experience evaluated twenty TOF-MRA examinations for the presence of cerebral aneurysms. Every case was evaluated with and without assistance by the AI software © mdbrain, with a washout period of at least four weeks in-between. Half of the cases included at least one false-positive AI finding. Aneurysm ratings, follow-up recommendations, and reading times were assessed using the Wilcoxon signed-rank test. RESULTS: False-positive AI results led to significantly higher suspicion of aneurysm findings (p = 0.01). Inexperienced readers further recommended significantly more intense follow-up examinations when presented with false-positive AI findings (p = 0.005). Reading times were significantly shorter with AI assistance in inexperienced (164.1 vs 228.2 s; p < 0.001), moderately experienced (126.2 vs 156.5 s; p < 0.009), and very experienced (117.9 vs 153.5 s; p < 0.001) readers alike. CONCLUSION: Our results demonstrate the susceptibility of radiology readers to automation bias in detecting cerebral aneurysms in TOF-MRA studies when encountering false-positive AI findings. While AI systems for cerebral aneurysm detection can provide benefits, challenges in human-AI interaction need to be mitigated to ensure safe and effective adoption.