Simulating mismatch between calibration and target population in AI for mammography the retrospective VAIB study
Haiko Schurz, Klara Solander, Davida Åström, Fernando Cossío, Tae Yang Choi, Magnus Dustler, Claes Lundström, Håkan Gustafsson, Sophia Zackrisson, Fredrik Strand
Abstract
AI cancer detection models require calibration to attain the desired balance between cancer detection rate (CDR) and false positive rate. In this study, we simulate the impact of six types of mismatches between the calibration population and the clinical target population, by creating purposefully non-representative datasets to calibrate AI for clinical settings. Mismatching the acquisition year between healthy and cancer-diagnosed screening participants led to a distortion in CDR between -3% to +19%. Mismatching age led to a distortion in CDR between -0.2% to +27%. Mismatching breast density distribution led to a distortion in CDR between +1% to 16%. Mismatching mammography vendors lead to a distortion in CDR between -32% to + 33%. Mismatches between calibration population and target clinical population lead to clinically important deviations. It is vital for safe clinical AI integration to ensure that important aspects of the calibration population are representative of the target population.