The performance evaluation of artificial intelligence ERNIE bot in Chinese National Medical Licensing Examination

Abstract

Clinical medical students are often tasked with acquiring a vast amount of theoretical knowledge and clinical experience across various subdisciplines during their academic journey. With limited time and energy, it is impractical for students to complete clinical rotations in every department, making it crucial to prioritize theoretical studies. By focusing on theoretical learning, students can indirectly gain clinical experience and develop a comprehensive understanding of different disciplines. Evaluating students’ performance in theoretical studies is a key component in assessing their progress. As a result, some researchers [1–4] are exploring the use of artificial intelligence in the National Medical Licensing Examination to assess its potential applications in medical research, education, and beyond. The study by Richard C. Armitage et al. [1] highlighted the impressive performance of the large language model (LLM) GPT-4 in the Membership of the Royal College of General Practitioners examination in an English context. In a similar vein, U. Hin Lai et al. [2] demonstrated ChatGPT’s strong performance in the United Kingdom Medical Licensing Assessment. However, a comparison of these two studies [3, 4] revealed that while GPT-4 excelled in the UK exams, it did not perform well in the Chinese National Medical Licensing Examination. ERNIE Bot is a LLM developed by Baidu, a Chinese company. It is trained based on a vast amount of Chinese-language content, and its version 3.5 is currently available for free public use. Compared with LLMs primarily trained on English data, such as ChatGPT, New Bing, and Claude+, ERNIE Bot exhibits superior comprehension and generation capabilities in the Chinese context [5]. This test is conducted using exam questions from 2011 to 2021 (Units 1 to 4, totaling 6600 questions). All question types are single-choice, and these four units are categorized based on question formats into A1-type (single-sentence best choice questions), A2-type (case summary best choice questions), A3-type (case group best choice questions), A4-type (case series best choice questions), and B1-type (standard matching questions). Please refer to the online supplementary material for an example of the conversation flow with ERNIE Bot. Due to the vast workload and data volume, the daily dialog limit of ERNIE Bot 4.0 is restricted. Therefore, we completed this test using ERNIE Bot 3.5. In the analysis of scores (Fig. 1), ERNIE Bot consistently scored above the passing threshold of 360 points, surpassing the average score of Chinese examinees. Research data from Hui Zong’s team and Luxiang Shang [3, 4] clearly indicate that GPT-4’s overall score fell below the passing mark. Consequently, ERNIE Bot outperforms GPT-4 in the Chinese National Medical Licensing Examination. Score comparison. Note: In Fig. 1, the vertical axis represents fractions and the horizontal axis represents years Our study expands on previous research by analyzing National Medical Licensing Examination questions over an 11-year period, resulting in a more robust dataset and stronger conclusions. The findings indicate the potential utility of ERNIE Bot in medical education and clinical settings. However, caution is advised due to the complexity of the algorithms and processes involved in analyzing answers. We recommend that ERNIE Bot be used by individuals with a solid medical background for educational purposes and as a supplementary tool in clinical practice, rather than as a substitute for professional medical judgment. The safety and accuracy of ERNIE Bot in clinical applications remain uncertain, and there is a risk of generating incorrect information or erroneous references. Non-medical users should seek advice from healthcare professionals rather than relying solely on AI for medical guidance. While AI can offer valuable educational support for medical students, it should not replace human interaction [6]. The integration of AI in clinical practice presents various challenges, including ethical, legal, social, and technical considerations [7, 8]. While there are ongoing concerns about the intersection of artificial intelligence and healthcare, the potential for substantial advancements in medical research and education remains promising [9, 10]. As AI continues to evolve through iterative updates, its capabilities are expanding, enabling a broader range of applications. For example, GPT-4 now has the capability to access the internet, facilitating real-time data updates [1, 10]. Nonetheless, to maximize the benefits of AI, a critical approach should be adopted, ensuring manageable risks and appropriate regulation of its use [9]. All authors contributed to the conception and design of the study. Material preparation, data collection, and analysis were conducted by Leiyun Huang, Jinghan Hu, and Qingjin Cai, respectively. The initial draft of the manuscript was authored by Leiyun Huang, with all authors providing input and revisions to versions. Final approval of the manuscript was granted by all authors. Conflict of interest statement: The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. This research was supported by the Yunnan Provincial Health Commission’s Leading Medical Talent Training Program (L-218004), the Yunnan Provincial Xingdian Elite Support Program for Outstanding Doctors (XDYC-MY-2022-0027), the Yunnan Provincial Clinical Medical Research Center for Orthopedics and Sports Rehabilitation (202102AA310068), the Yunnan Provincial Clinical Medical Center for Spinal Cord and Spinal Diseases (ZX2022000101), and the Key Laboratory of Digital Orthopedics of Yunnan Province (202005AG070004). Special project for social development of Yunnan Provincial Science and Technology Department (202403AC100003). According to current guidelines and regulations, we hereby confirm that this study does not require ethical approval. It is based on publicly available data and does not involve human subjects, animal experiments, or interference with organisms. The data underlying this article are available in the article and in its online supplementary material.

Topics & Concepts