Exploring the performance of ChatGPT on acute pancreatitis-related questions

Abstract

Acute pancreatitis (AP) is a serious gastrointestinal disease with an incidence rate of approximately 34 cases per 100,000 individuals annually, the overall burden of AP remains high with the aging population [1].There is a notable trend among the public to acknowledge APrelated information to improve awareness.Artificial intelligence (AI) is a large language model providing updated and useful information.The Chat Generative Pre-trained Transformer (ChatGPT, https:// openai.com), developed by OpenAI and launched on November 30, 2022, stands out in this field.Various studies have explored its utility in responding to medical questions.This study aims to evaluate and compare the capabilities of ChatGPT-3.5 and ChatGPT-4.0 in answering test questions about AP, employing both subjective and objective metrics. MethodsAs shown in Table S1, we conducted our study using 18 subjective test questions derived from the Atlanta AP classification consensus and the American Gastroenterological Association (AGA) guidelines (Strength of recommendation: Strong) [2-4].Additionally, we selected 73 objective questions with the highest number of tested times from the Chinese professional physician test database, categorizing them into four subfields (Table S2).These questions were submitted to ChatGPT in two separate sessions on February 1, 2024, and February 8, 2024, respectively.Two independent reviewers evaluated the subjective questions using a 5-point Likert Scale.Any discordance was resolved by the third author.The flowchart of overall study design is presented in Figure S1.The response accuracy was analyzed using the Chi-squared and Mann-Whitney U tests, with a P-value of < 0.05 indicating statistical significance. ResultsAs shown in Table 1, ChatGPT-3.5 correctly answered 80% of subjective questions, while ChatGPT-4.0achieved an accuracy rate of 94%.For objective questions, Chat-GPT-4.0 outperformed ChatGPT-3.5 with a 78.1% accuracy rate compared to 68.5% (P = 0.01) (Figure S2A).Across all questions tested in the study, the concordance rate between ChatGPT-3.5 and ChatGPT-4.0was 80.8% and 83.6% (Figure S2B), respectively, with the mean

Topics & Concepts