Evaluation of the reliability and readability of answers given by chatbots to frequently asked questions about endophthalmitis: A cross-sectional study on chatbots
Süleyman Demir
Abstract
Objective: This study aimed to investigate the accuracy, reliability, and readability of A-Eye Consult, ChatGPT-4.0, Google Gemini and Copilot AI large language models (LLMs) in responding to patient questions about endophthalmitis. Methods: The LLMs’ responses to 25 questions about endophthalmitis, frequently asked by patients, were evaluated by two ophthalmologists using a five-point Likert scale, with scores ranging from 1–5. The DISCERN scale assessed the reliability of the LLMs’ responses, whereas the Flesch Reading Ease (FRE) and Flesch–Kincaid Grade Level (FKGL) indices assessed readability and text complexity, respectively. Results: A-Eye Consult and ChatGPT-4.0 outperformed Google Gemini and Copilot in providing comprehensive and precise responses. The Likert score significantly differed across all four LLMs ( p < .001), with A-Eye Consult scoring significantly higher than Google Gemini and Copilot ( p < .001). Conclusions: A-Eye Consult and ChatGPT-4.0 responses, while more complex than those of other LLMs, provided more reliable and accurate information.