Automation Bias in AI-Decision Support: Results from an Empirical Study
Florian Kücking, Ursula Hübner, Mareike Przysucha, Niels Hannemann, Jan-Oliver Kutza, Maurice Moelleken, Cornelia Erfurt‐Berge, Joachim Dissemond, Birgit Babitsch, Dorothee Busch
Abstract
INTRODUCTION: Automation bias poses a significant challenge to the effectiveness of Clinical Decision Support Systems (CDSS), potentially compromising diagnostic accuracy. Previous research highlights trust, self-confidence, and task difficulty as key determinants. With the increasing availability of AI-enabled CDSS, automation bias attains new attention. This study therefore aims to identify factors influencing automation bias in a diagnostic task. METHODS: A quantitative intervention study with participants from different backgrounds (n = 210) was conducted, employing regression analysis to analyze potential factors. Automation bias was measured as the agreement rate with wrong AI-enabled recommendations. RESULTS AND DISCUSSION: Diagnostic performance, certified wound care training, physician profession, and female gender significantly reduced false agreement rates. Higher perceived benefit of the system was significantly associated with promoting false agreement. Strategies like comprehensive diagnostic training are pivotal in the prevention of automation bias when implementing CDSS. CONCLUSION: Considering factors influencing automation bias when introducing a CDSS is critical to fully leverage the benefits of such a system. This study highlights that non-specialists, who stand to gain the most from CDSS, are also the most susceptible to automation bias, emphasizing the need for specialized training to mitigate this risk and ensure diagnostic accuracy and patient safety.