MedScaleRE-PF: a prompt-based framework with retrieval-augmented generation, chain-of-thought, and self-verification for scale-specific relation extraction in Chinese medical literature
Zhenli Chen, Jie Hao, Haixia Sun, Liang Zhao, Jiao Li, Qing Qian, Qin Peng, Xuwen Wang, Shan Cong, Liu Shen, Zhen Guo, Siyue Pu, Lin Yan
Abstract
Large language models have shown promise in biomedical natural language processing, yet their use in extracting structured knowledge from medical scales remains limited. This study introduces MedScaleRE-PF, a novel prompting framework designed for relation extraction in Chinese medical scale texts. The framework combines few-shot in-context learning with retrieval-augmented generation, chain-of-thought prompting, and self-verification strategies to improve contextual understanding and factual consistency. We constructed the CMedS-RE dataset, consisting of 606 full-text articles with 19,051 sentences, 29,359 annotated entities, and 7217 relation instances. Experiments were conducted on two tasks: relational triple extraction (RTE) and relation classification (RC). We evaluated both single-step and multi-step prompting, along with four self-verification strategies: direct (D-SV), stepwise (S-CoT-SV), relation-specific (R-CoT-SV), and stepwise relation-specific (SR-CoT-SV). The best results were achieved with single-step prompting and the R-CoT-SV strategy, yielding F1 scores of 42.58 % for RTE under the 32-shot setting and 65.42 % for RC under the 8-shot setting. Compared to a RAG-only baseline, this configuration improved F1 by 7.59 % on RTE and 1.07 % on RC. Additional experiments demonstrated strong performance under annotation-scarce conditions, achieving 46.99 % F1 on RTE with 20 training articles and 59.87 % on RC with 50 articles. Ablation and error analyses further confirmed that task-specific prompt structure and verification design significantly impact performance under few-shot conditions. MedScaleRE-PF also showed consistent results across multiple LLMs, confirming its stability and generalizability. These findings highlight the effectiveness of combining simple prompting and CoT-inspired verification in domain-specific information extraction. MedScaleRE-PF offers a flexible and structured approach for mining medical scale knowledge and supports prompt-based development in biomedical applications.