S<sup>2</sup>HM<sup>2</sup>: A Spectral–Spatial Hierarchical Masked Modeling Framework for Self-Supervised Feature Learning and Classification of Large-Scale Hyperspectral Images
Lilin Tu, Jiayi Li, Xin Huang, Jianya Gong, Xing Xie, Leiguang Wang
Abstract
Most of the existing deep learning-based hyperspectral image (HSI) classification algorithms are based on supervised learning, where large number of annotated labels with high acquisition cost are required. Self-supervised learning (SSL) methods can learn abundant representations using large amount of unlabeled data, thereby reducing the reliability of labels. Particularly, SSL based on Masked Image Modeling (MIM) can extract fine-grained features, which is well-suited for HSI classification as a pixel-level interpretation task. However, MIM has scarcely been investigated in HSI classification. Current algorithms lack a comprehensive consideration of the multiscale spectral-spatial characteristics of HSI when constructing the pre-training task, and there exists high computational cost and redundancy when applied to large-scale HSI. Therefore, this paper develops an SSL framework based on Spectral-Spatial Hierarchical Masked Modeling (S <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> HM <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ) for large-scale HSI classification. Considering the spectral-spatial characteristics of HSI, 3D masking strategy and spectral-spatial consistency loss are proposed to construct MIM task. To fully exploit features at each scale, hierarchical 3D Feature Pyramid Network (3D-FPN) is designed as decoder for both pre-text and downstream tasks in a “pixel-to-pixel” manner. In addition, Multi-Scale Masked Feature Modeling (MS-MFM) task is proposed to further facilitate the multiscale feature learning. The SSL pre-training is guided by both MIM and MS-MFM. Experimental results on two large-scale hyperspectral datasets, i.e., WHU-OHS and WHU-H <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> SR, demonstrate the superiority of the proposed method. Furthermore, transfer learning experiments are conducted on a variety of hyperspectral datasets, where classification accuracies are boosted in most of the scenarios. Source code will be made available at https://github.com/tulilin/S2HM2.