OpenAI o1 Large Language Model Outperforms GPT-4o, Gemini 1.5 Flash, and Human Test Takers on Ophthalmology Board–Style Questions
Ryan Shean, Tathya Shah, Sina Sobhani, Alan Tang, A. R. Setayesh, Kyle Bolo, Văn Thành Nguyễn, Benjamin Y. Xu
Abstract
Purpose: To evaluate and compare the performance of human test takers and three AI models-OpenAI o1, ChatGPT-4o, and Gemini 1.5 Flash-on ophthalmology board-style questions, focusing on overall accuracy and performance stratified by ophthalmic subspecialty and cognitive complexity level.Design: Cross-sectional study.Subjects: 500 questions sourced from the Basic and Clinical Science Course (BCSC) and EyeQuiz question banks.Methods: Three large language models (LLMs) interpreted the questions using standardized prompting procedures.Subanalysis was performed stratifying the questions by subspecialty and complexity defined by the Buckwalter Taxonomic Schema.Statistical analysis, including the analysis of variance (ANOVA) and McNemar's test, was conducted to assess performance differences. Main Outcome Measures:Accuracy of responses for each model and human test takers, stratified by subspecialty and cognitive complexity.Results: OpenAI o1 achieved the highest overall accuracy (423/500, 84.6%), significantly outperforming GPT-4o (331/500, 66.2%; P < 0.001), Gemini (301/500, 60.2%; P < 0.001).o1 demonstrated superior performance on both BCSC (228/250, 91.2%) and EyeQuiz (195/250, 78.0%) questions compared to GPT-4o (BCSC: 183/250, 73.2%; EyeQuiz: 148/250, 59.2%) and Gemini (BCSC: 163/250, 65.2%; EyeQuiz: 137/250, 54.8%).On BCSC questions, human performance was lower (64.5%)than Gemini 1.5 Flash (65.2%),GPT-4o (73.2%), and OpenAI o1 (91.2%) (P < 0.001).OpenAI o1 outperformed other models in each of the nine ophthalmic subfields and three cognitive complexity levels.Conclusions: OpenAI o1 outperformed GPT-4o, Gemini, and human test takers in answering ophthalmology board-style questions from two question banks and across three complexity levels.These findings highlight advances in AI technology and OpenAI o1's growing potential as an adjunct in ophthalmic education and care.