Fine-Tuning Multimodal Vision-Language Models for Brain CT Diagnosis via a Triple-Branch Framework
Xiong Luo
Abstract
Medical image analysis, particularly for brain CT scans, requires advanced techniques to improve diagnostic accuracy and efficiency. Existing multimodal models often struggle with the complexities of domain-specific medical images and their associated reports. This article presents TriMedTune, a novel triple-branch framework designed to fine-tune multimodal models for brain CT diagnosis. The proposed framework consists of three core components: Hierarchical Visual Prompt Injection (HVPI), Diagnostic Alignment for Terminology Accuracy (DATA), and Medical Knowledge Distillation with Uncertainty Regularization (MKD-UR). These modules enhance the model’s ability to process and interpret medical images by improving visual context understanding, aligning diagnostic terminology, and ensuring robust performance through knowledge distillation and uncertainty modeling. The framework utilizes efficient training strategies, including LoRA-based fine-tuning, dynamic prompt sampling, and mixed-precision optimization. Experimental evaluations show that TriMedTune outperforms state-of-the-art models across multiple performance metrics, demonstrating its effectiveness in medical image diagnosis and report generation.