Explainable Multimodal Learning in Remote Sensing: Challenges and Future Directions
Alexander Günther, Hiba Najjar, Andreas Dengel
Abstract
Earth observation applications effectively leverage deep learning models to harness the abundantly available remote sensing data. In order to use all the different modalities relevant to a specific task, the fusion of these data sources can be achieved using multi-modal learning techniques. This is especially helpful when the input dataset contains both images and tabular data, or when the temporal and spatial resolutions vary across the modalities of interest. Nevertheless, these fusion techniques typically increase in complexity as the disparities in the nature of the fused modalities increase. The resulting complex deep learning models suffer from a lack of explainability and transparency, which is crucial in many sensitive human-related applications. In this letter, we describe how the research community in remote sensing addresses the issue of model explainability in the context of multi-modal learning. We additionally review the practices used in other application fields and identify some of the most promising explainability methods tailored for multi-modal deep networks to be exploited in remote sensing applications.