Exploring the performance of ChatGPT-3.5 in addressing dermatological queries: a research investigation into AI capabilities
Marcin Rojek, Jakub Kufel, Michał Bielówka, Adam Mitręga, Dominika Kaczyńska, Łukasz Czogalik, Dominika Kondoł, Kacper Palkij, Sylwia Mielcarska, Wiktoria Bartnikowska
Abstract
Introduction:In the 21 st century's era of rapid technological advancement, ChatGPT-3.5, an artificial intelligence (AI) language model, is scrutinized for its application in dermatology.Using 119 questions from the National Specialist Examination (PES), we assess ChatGPT-3.5'sperformance by comparing it to human skills and addressing ethical implications.Objective: Our primary aim is to evaluate ChatGPT-3.5'sproficiency in responding to 119 dermatology questions from the PES.The study emphasizes ethical considerations and compares the model's knowledge and skills to those of human dermatologists.Material and methods: Utilizing the 2023 PES question database, questions were categorized by Bloom's taxonomy and thematic content.ChatGPT-3.5,version of 3 August 2023, answered 119 questions in five sessions, allowing for a probabilistic evaluation.Statistical analyses, conducted using R Studio, assessed correctness, confidence, and difficulty.Results: ChatGPT-3.5 achieved a 49.58% correct response rate, below the 60% passing threshold.No significant differences in difficulty or correlations between difficulty and certainty were observed.Varied performance across question types highlighted strengths and weaknesses.Despite suboptimal results, ChatGPT-3.5'sdifferential performance offers insights, suggesting future improvements.The study advocates for ongoing research into AI integration in dermatology, envisioning a promising role for AI in assisting dermatologists.Conclusions: Ethical considerations are crucial for effective AI introduction, minimizing errors, and enhancing dermatological healthcare quality, fostering optimism for AI's evolving role in dermatology.