Evaluating interactions of patients with large language models for medical information
Nicolas Carl, Sarah Haggenmüller, Christoph Wies, Lisa Nguyen, Jana Theres Winterstein, Martin J. Hetz, Maurin Helen Mangold, Friedrich Otto Hartung, Britta Grüne, Tim Holland‐Letz, Maurice Stephan Michel, Titus J. Brinker, Frederik Wessels
Abstract
OBJECTIVES: To explore the interaction of real-world patients with a chatbot in a clinical setting, investigating key aspects of medical information provided by large language models (LLMs). PATIENTS AND METHODS: The study enrolled 300 patients seeking urological counselling between February and July 2024. First, participants voluntarily conversed with a Generative Pre-trained Transformer 4 (GPT-4) powered chatbot to ask questions related to their medical situation. In the following survey, patients rated the perceived utility, completeness, and understandability of the information provided during the simulated conversation as well as user-friendliness. Finally, patients were asked which, in their experience, best answered their questions: LLMs, urologists, or search engines. RESULTS: A total of 292 patients completed the study. The majority of patients perceived the chatbot as providing useful, complete, and understandable information, as well as being user-friendly. However, the ability of human urologists to answer medical questions in an understandable way was rated higher than of LLMs. Interestingly, 53% of participants rated the question-answering ability of LLMs higher than search engines. Age was not associated with preferences. Limitations include social desirability and sampling biases. DISCUSSION: This study highlights the potential of LLMs to enhance patient education and communication in clinical settings, with patients valuing their user-friendliness and comprehensiveness for medical information. By addressing preliminary questions, LLMs could potentially relieve time constraints on healthcare providers, enabling medical personnel to focus on complex inquiries and patient care.