A comparative evaluation of chain-of-thought-based prompt engineering techniques for medical question answering
Sohyeon Jeon, Hong‐Gee Kim
Abstract
BACKGROUND AND OBJECTIVE: Large language models (LLMs) hold transformative potential for clinical decision-making in the rapidly advancing field of AI in medicine. This study evaluates how Chain-of-Thought (CoT) prompting techniques affect medical reasoning performance with consideration for clinical applicability. METHODS: Five LLMs (GPT-4o-mini, GPT-3.5-turbo, o1-mini, Gemini-1.5-Flash, Gemini-1.0-pro) were assessed via API access using CoT prompting methods exhibiting distinct cognitive characteristics. Evaluations included clinical datasets and non-clinical datasets. An iterative QA system was constructed to ensure consistent and reproducible results. RESULTS: o1-mini achieved the highest performance with 88.4 % accuracy in EHRNoteQA using MIMIC-IV discharge summaries. Traditional CoT consistently provided stable results across clinical datasets, while Interactive CoT showed variability - underperforming in complex medical reasoning tasks like MedMCQA (61.7 %) but performing well in clinical notes (83.5 % in EHRNoteQA). Statistical analysis revealed no significant differences between prompting methods across datasets (p > .05), though effect size analysis showed notable variations in method effectiveness. CONCLUSION: Complex prompting techniques do not significantly enhance performance compared to simpler approaches. Dataset characteristics and model architecture have greater impact, suggesting simpler CoT methods may be more effective for clinical applications.