BERT-XML: Large Scale Automated ICD Coding Using BERT Pretraining
Zachariah Zhang, Jingshu Liu, Narges Razavian
Abstract
ICD coding is the task of classifying and coding all diagnoses, symptoms and procedures associated with a patient's visit. The process is often manual, extremely time-consuming and expensive for hospitals as clinical interactions are usually recorded in free text medical notes. In this paper, we propose a machine learning model, BERT-XML, for large scale automated ICD coding of EHR notes, utilizing recently developed unsupervised pretraining that have achieved state of the art performance on a variety of NLP tasks. We train a BERT model from scratch on EHR notes, learning with vocabulary better suited for EHR tasks and thus outperform off-the-shelf models. We further adapt the BERT architecture for ICD coding with multi-label attention. We demonstrate the effectiveness of BERT-based models on the large scale ICD code classification task using millions of EHR notes to predict thousands of unique codes.