An Attention-Based Mechanism to Combine Images and Metadata in Deep Learning Models Applied to Skin Cancer Classification
André G. C. Pacheco, Renato A. Krohling
Abstract
Computer-aided skin cancer classification systems built with deep neural networks usually yield predictions based only on images of skin lesions. Despite presenting promising results, it is possible to achieve higher performance by taking into account patient demographics, which are important clues that human experts consider during skin lesion screening. In this article, we deal with the problem of combining images and metadata features using deep learning models applied to skin cancer classification. We propose the Metadata Processing Block (MetaBlock), a novel algorithm that uses metadata to support data classification by enhancing the most relevant features extracted from the images throughout the classification pipeline. We compared the proposed method with two other combination approaches: the MetaNet and one based on features concatenation. Results obtained for two different skin lesion datasets show that our method improves classification for all tested models and performs better than the other combination approaches in 6 out of 10 scenarios.