DermaGPT a federated multimodal framework with a meta learned trust function for interpretable dermatology diagnostics
Nastaran Mehrabi Hashjin, Mohammad Hussein Amiri, Maryam Khanian Najafabadi
Abstract
Advances in generative and federated artificial intelligence enable privacy-aware diagnostic systems that integrate multimodal reasoning and explainability. This work introduces DermaGPT, a federated multimodal framework for dermatology decision support that emphasizes trustworthy use under heterogeneous, privacy-sensitive data. The system combines a PaLI-Gemma 2 vision–language backbone, fine-tuned with low-rank adaptation, with a retrieval-augmented large language model that generates clinically coherent and patient-friendly explanations. To improve robustness and calibration across sites, a meta-learned trust function (MLTF) dynamically re-weights client updates based on uncertainty, calibration, and domain-shift indicators. Evaluated on four institutional datasets and an external cohort of 4,452 biopsy-confirmed clinical and dermoscopic images, DermaGPT achieved 90.2% diagnostic accuracy across 11 lesion types and 93.3% accuracy in malignancy prediction, with well-calibrated outputs under federated training. Expert dermatologists rated its explanations as clear and clinically relevant; these ratings were obtained on class-level canonical exemplars rather than per-image reports. In our deployment threat model, images are processed locally by the vision module; when a third-party LLM is used, only text (a short diagnostic summary and the user question) is transmitted, which may still be considered sensitive health data. Taken together, these results indicate that a trust-aware, federated multimodal design can deliver interpretable, efficient, and privacy-aware dermatology decision support that is intended to augment rather than replace clinician judgment.