The Synergy of Seeing and Saying: Revolutionary Advances in Multi-modality Medical Vision-Language Large Models
Xiang Li, Yu Sun, Jia Lin, Like Li, Ting Feng, Shen Yin
Abstract
The application of visual-language large models in the field of medical health has gradually become a research focus. The models combine the capability for image understanding and natural language processing, and can simultaneously process multi-modality data such as medical images and medical reports. These models can not only recognize images, but also understand the semantic relationship between images and texts, effectively realize the integration of medical information, and provide strong support for clinical decision-making and disease diagnosis. The visual-language large model has good performance for specific medical tasks, and also shows strong potential and high intelligence in the general task models. This paper provides a comprehensive review of the visual-language large model in the field of medical health. Specifically, this paper first introduces the basic theoretical basis and technical principles. Then, this paper introduces the specific application scenarios in the field of medical health, including modality fusion, semi-supervised learning, weakly supervised learning, unsupervised learning, cross-domain model and general models. Finally, the challenges including insufficient data, interpretability, and practical deployment are discussed. According to the existing challenges, four potential future development directions are given.