Will code one day run a code? Performance of language models on <scp>ACEM</scp> primary examinations and implications

Abstract

OBJECTIVE: Large language models (LLMs) have demonstrated mixed results in their ability to pass various specialist medical examination and their performance within the field of emergency medicine remains unknown. METHODS: We explored the performance of three prevalent LLMs (OpenAI's GPT series, Google's Bard, and Microsoft's Bing Chat) on a practice ACEM primary examination. RESULTS: All LLMs achieved a passing score, with scores with GPT 4.0 outperforming the average candidate. CONCLUSION: Large language models, by passing the ACEM primary examination, show potential as tools for medical education and practice. However, limitations exist and are discussed.

Topics & Concepts