Semantic aware representation learning for optimizing image retrieval systems in radiology
Zografoula Vagena, Xiaoyang Wei, Camille Kurtz, Florence Cloppet
Abstract
Content-based image retrieval (CBIR), which consists of ranking a set of images with respect to a query image based on visual similarity, can assist diagnostic radiologists in assessing medical images, by identifying similar digital images in large image databases. Despite the many recent advances and innovations in CBIR for general images, their adoption in radiology has been slow and limited. In the current paper we attempt to close the gap between the two domains and wisely adapt modern CBIR techniques to radiology images: by extending the latest representation learning techniques in a way that can overcome the unique challenges and at the same time take advantage of the specific opportunities that are present in radiology we were able to come up with novel and effective medical image retrieval methods. Our method achieves the highest CUI@5 scores (18.48, 15.95) on two widely used datasets (ROCO and MEDICAT respectively), showcasing the superiority of the proposed method in comparison with state-of-the-art relevant alternatives. • Metadata about image semantics is beneficial for representation learning. • External medical knowledge can be integrated for better semantics understanding. • Severity of mistakes can to be taken into account to improve image retrieval. • Differentiable average precision-based loss is suitable for representation learning.