Responsible adoption of multimodal artificial intelligence in health care: promises and challenges
Ghazal Azarfar, Sara Naimimohasses, Sirisha Rambhatla, Matthieu Komorowski, Diana Ferro, Peter Lewis, Darren Gates, Nawar Shara, Gregg M. Gascon, Anthony Chang, Muhammad Mamdani, Mamatha Bhat
Abstract
Clinicians rely on various data modalities-such as patient history, clinical signs, imaging, and laboratory results-to improve decision making. Multimodal artificial intelligence (AI) systems are emerging as powerful tools to process these diverse data types; however, the clinical adoption of multimodal AI systems is challenging because of data heterogeneity and integration complexities. The 2024 Temerty Centre for AI Research and Education in Medicine symposium, held on June 17, 2024, in Toronto, Canada, explored the potential and challenges of implementing multimodal AI in health care. In this Review, we summarise insights from the symposium. We discuss current applications, such as those used in early diagnosis of sepsis and cardiology, and identify key barriers, including fusion techniques, model selection, generalisation, fairness, safety, security, and international considerations on the responsible deployment of multimodal AI in health care. We outline practical strategies to overcome these obstacles, emphasising technologies such as federated learning to reduce bias and promote equitable health care. By addressing these challenges, multimodal AI can transform clinical practice and improve patient outcomes worldwide.