Evaluating large language and large reasoning models as decision support tools in emergency internal medicine

Abstract

BACKGROUND: Large Language Models (LLMs) hold promise for clinical decision support, but their real-world performance varies. We compared three leading models (OpenAI's "o1" Large Reasoning Model (LRM), Anthropic's Claude-3.5-Sonnet, and Meta's Llama-3.2-70B) to human experts in an emergency internal medicine setting. METHODS: We conducted a prospective comparative study on 73 anonymized patient cases from the Emergency Internal Medicine ward of the University Hospital Split, Croatia (June-September 2024). Two independent internal medicine specialists, blinded to model identity, graded the LLM-generated reports in two steps: (1) they evaluated the relevance of recommended diagnostic tests based on the patient's signs, symptoms, and medical history; (2) after reviewing the actual diagnostic test results, they assessed each model's final diagnosis, therapy plan, and follow-up recommendations. The same evaluative framework was applied to human-authored reports. Likert scales (1-4 or 1-3) were used, and statistical comparisons included the Friedman and Wilcoxon signed-rank tests. RESULTS: The o1 model achieved a mean final rating (3.63) statistically indistinguishable from human physicians (3.67; p = 0.62). Claude-3.5-Sonnet (3.38) and Llama-3.2-70B (3.23) scored significantly lower (p < 0.01 vs. o1), largely due to errors in therapy planning and non-medication recommendations. Despite this gap, all three models demonstrated ≥90 % accuracy in final diagnoses and patient admission decisions. The o1 model correctly classified all abnormal lab values (100 %), while Claude-3.5-Sonnet and Llama-3.2-70B showed minor errors (99.5 % and 99 % accuracy, respectively). CONCLUSIONS: When evaluated on real-world emergency cases, an advanced LLM with enhanced reasoning (o1) can match expert-level clinical performance, underscoring its potential utility as a decision-support tool.

Topics & Concepts