MAT-Net: Multiscale Aggregation Transformer Network for Hyperspectral Unmixing

Abstract

Deep learning (DL) has recently shown considerable potential in the field of Hyperspectral unmixing (HU), thanks to its advanced capabilities in feature representation. Most joint spectral-spatial DL-based unmixing methods extract spatial features at a single scale, with basic feature fusion techniques. Given the heterogeneity in ground object sizes, these methods, with their singular receptive fields, may capture only incomplete features or blend features from disparate objects, leading to inadequate spatial comprehension. To tackle this challenge, we propose the multiscale aggregation transformer network (MAT-Net), which utilizes an encoder-decoder architecture designed to harness both spectral and spatial data comprehensively. Within this structure, we have devised a dual-stream, multibranch convolutional neural network (CNN) encoder to extract both spectral and multiscale spatial information. The spatial stream includes three CNN branches with different receptive fields. A block-by-block branching strategy is employed in this stream to maintain the continuity of multiscale spatial information and reduce computational costs. To effectively integrate intricate multiscale features, we introduce a transformer encoder that incorporates multihead self-scale-aggregation attention (MSsaA) blocks. These blocks are designed to adaptively modulate feature weights based on the scale characteristics of the image, enabling a more nuanced feature integration process. Experimental results on synthetic and real datasets demonstrate the effectiveness of MAT-Net.

Topics & Concepts