ChatGPT vs. neurologists: a cross-sectional study investigating preference, satisfaction ratings and perceived empathy in responses among people living with multiple sclerosis
Elisabetta Maida, Marcello Moccia, Raffaele Palladino, Giovanna Borriello, Giuseppina Affinito, Marinella Clerico, Anna Maria Repice, Alessia Di Sapio, Rosa Iodice, Antonio Luca Spiezia, Maddalena Sparaco, Giuseppina Miele, Floriana Bile, Cristiano Scandurra, Diana Ferraro, Maria Laura Stromillo, Renato Docimo, Antonio Martino, Luca Mancinelli, Gianmarco Abbadessa, Krzysztof Smolik, Lorenzo Lorusso, Maurizio Leone, Elisa Leveraro, Francesca Di Lauro, Francesca Trojsi, Lidia Mislin Streito, Francesca Gabriele, Fabiana Marinelli, Antonio Ianniello, Federica De Santis, Matteo Foschi, Nicola De Stefano, Vincenzo Brescia Morra, Alvino Bisecco, Giancarlo Coghe, Eleonora Cocco, Michele Romoli, Francesco Corea, Letizia Leocani, Jessica Frau, Simona Sacco, Matilde Inglese, Antonio Carotenuto, Roberta Lanzillo, Alessandro Padovani, Maria Triassi, Simona Bonavita, Luigi Lavorgna, Digital Technologies, Web, Social Media Study Group of the Italian Society of Neurology (SIN)
Abstract
BACKGROUND: ChatGPT is an open-source natural language processing software that replies to users' queries. We conducted a cross-sectional study to assess people living with Multiple Sclerosis' (PwMS) preferences, satisfaction, and empathy toward two alternate responses to four frequently-asked questions, one authored by a group of neurologists, the other by ChatGPT. METHODS: An online form was sent through digital communication platforms. PwMS were blind to the author of each response and were asked to express their preference for each alternate response to the four questions. The overall satisfaction was assessed using a Likert scale (1-5); the Consultation and Relational Empathy scale was employed to assess perceived empathy. RESULTS: We included 1133 PwMS (age, 45.26 ± 11.50 years; females, 68.49%). ChatGPT's responses showed significantly higher empathy scores (Coeff = 1.38; 95% CI = 0.65, 2.11; p > z < 0.01), when compared with neurologists' responses. No association was found between ChatGPT' responses and mean satisfaction (Coeff = 0.03; 95% CI = - 0.01, 0.07; p = 0.157). College graduate, when compared with high school education responder, had significantly lower likelihood to prefer ChatGPT response (IRR = 0.87; 95% CI = 0.79, 0.95; p < 0.01). CONCLUSIONS: ChatGPT-authored responses provided higher empathy than neurologists. Although AI holds potential, physicians should prepare to interact with increasingly digitized patients and guide them on responsible AI use. Future development should consider tailoring AIs' responses to individual characteristics. Within the progressive digitalization of the population, ChatGPT could emerge as a helpful support in healthcare management rather than an alternative.