The Performance of ChatGPT-4V in Interpreting Images and Tables in the Japanese Medical Licensing Exam

Abstract

The recent introduction of Chat Generative Pre-trained Transformer 4 Vision (ChatGPT-4V) has expanded the capabilities of language models to include image input features, potentially broadening their application in the medical field. This Research Letter evaluates the performance of ChatGPT-4V in interpreting clinical images and tables through the Japanese Medical Licensing Exam (JMLE). Employing the September 25, 2023, version of ChatGPT-4V, the study compared the program’s responses to the 117th JMLE against the passing criteria and the average scores of human examinees. While ChatGPT-4V surpassed the passing threshold with an 85.1% correct response rate in essential knowledge and 76.5% in other areas, it fell short in image-based (71.9%) and table-based questions (35.0%), indicating a significant gap compared to human performance. This suggests limitations in the model’s image and table interpretation, exacerbated by its lower proficiency in non-Latin characters and potential overreliance on text information. Despite its success in passing the JMLE, the study highlights the need for further development of ChatGPT-4V to enhance its reliability for medical applications, including diagnostics.

Topics & Concepts