Large language models can effectively extract stroke and reperfusion audit data from medical free-text discharge summaries
Rudy Goh, Benjamin K. Cook, Brandon Stretton, Andrew Booth, Shrirajh Satheakeerthy, Sarah Howson, Joshua G. Kovoor, Aashray Gupta, Sheryn Tan, W. Taylor Kimberly, Andrew Moey, Wilson Vallat, John Maddison, Jarrod A. Marks, Samuel Gluck, Toby Gilbert, Jim Jannes, Timothy Kleinig, Stephen Bacchi
Abstract
INTRODUCTION: Audits are an integral part of effective modern healthcare. The collection of data for audits can be resource intensive. Large language models (LLM) may be able to assist. This pilot study aimed to assess the feasibility of using a LLM to extract stroke audit data from free-text medical documentation. METHOD: Discharge summaries from a one-month retrospective cohort of stroke admissions at a tertiary hospital were collected. A locally-deployed LLM, LLaMA3, was then used to extract a variety of routine stroke audit data from free-text discharge summaries. These data were compared to the previously collected human audit data in the statewide registry. Manual case note review was undertaken in cases of discordance. RESULTS: Overall, there was a total of 144 data points that were extracted (9 data points for each of the 16 patients). The LLM was correct in 135/144 (93.8%) of individual datapoints. This performance included binary categorical, multiple-option categorical, datetime, and free-text extraction fields. CONCLUSIONS: LLM may be able to assist with the efficient collection of stroke audit data. Such approaches may be pursued in other specialties. Future studies should seek to examine the most effective way to deploy such approaches in conjunction with human auditors and researchers.