Enhancing AI reliability: A foundation model with uncertainty estimation for optical coherence tomography-based retinal disease diagnosis
Yuanyuan Peng, Aidi Lin, Meng Wang, Tian Lin, Linna Liu, Jianhua Wu, Ke Zou, Tingkun Shi, Lixia Feng, Zhen Liang, Tao Li, Dan Liang, Shanshan Yu, Dawei Sun, Jing Luo, Ling Gao, Xinjian Chen, Binwei Huang, Chaoxin Zheng, Chuang Jin, Dezhi Zheng, Dingguo Huang, Dongjie Li, Guihua Zhang, Hanfu Wu, Honghe Xia, Hongjie Lin, Huiyu Liang, Jingsheng Yi, Jinqu Huang, Juntao Liu, Man Chen, Qin Zeng, Taiping Li, Weiqi Chen, Xia Huang, Xiaolin Chen, Xixuan Ke, Xulong Liao, Yifan Wang, Yin Huang, Yinglin Cheng, Yinling Zhang, Yongqun Xiong, Yuqiang Huang, Zhenggen Wu, Zijing Huang, Ching-Yu Cheng, Huazhu Fu, Haoyu Chen
Abstract
Inability to express the confidence level and detect unseen disease classes limits the clinical implementation of artificial intelligence in the real world. We develop a foundation model with uncertainty estimation (FMUE) to detect 16 retinal conditions on optical coherence tomography (OCT). In the internal test set, FMUE achieves a higher F1 score of 95.74% than other state-of-the-art algorithms (92.03%-93.66%) and improves to 97.44% with threshold strategy. The model achieves similar excellent performance on two external test sets from the same and different OCT machines. In human-model comparison, FMUE achieves a higher F1 score of 96.30% than retinal experts (86.95%, p = 0.004), senior doctors (82.71%, p < 0.001), junior doctors (66.55%, p < 0.001), and generative pretrained transformer 4 with vision (GPT-4V) (32.39%, p < 0.001). Besides, FMUE predicts high uncertainty scores for >85% images of non-target-category diseases or with low quality to prompt manual checks and prevent misdiagnosis. Our FMUE provides a trustworthy method for automatic retinal anomaly detection in a clinical open-set environment.