Multimodal generative AI for interpreting 3D medical images and videos
Jung-Oh Lee, Hong-Yu Zhou, Tyler M. Berzin, Daniel K. Sodickson, Pranav Rajpurkar
Abstract
This perspective proposes adapting video-text generative AI to 3D medical imaging (CT/MRI) and medical videos (endoscopy/laparoscopy) by treating 3D images as videos. The approach leverages modern video models to analyze multiple sequences simultaneously and provide real-time AI assistance during procedures. The paper examines medical imaging's unique characteristics (synergistic information, metadata, and world model), outlines applications in automated reporting, case retrieval, and education, and addresses challenges of limited datasets, benchmarks, and specialized training.
Topics & Concepts
MetadataComputer sciencePerspective (graphical)Generative grammarGenerative modelArtificial intelligenceMedical imagingMultimediaComputer visionInformation retrievalWorld Wide WebColorectal Cancer Screening and DetectionRadiomics and Machine Learning in Medical ImagingAI in cancer detection