An Integration of Natural Language and Hyperspectral Imaging: A review

Abstract

The innovation of transformer architecture has propelled the growth of natural language algorithms and models, spanning language models, large language models (LLMs), and pretraining image–text foundation models. Integrating these advancements with state-of-the-art natural language processing (NLP) techniques has led to innovative applications with active use. Achieving artificial general intelligence in the hyperspectral imaging (HSI) domain necessitates extracting abstract features from images, emphasizing global features over pixel-level local features using spectral and spatial characteristics. Abstract features, extracted by leveraging the abstract nature of natural language, provide more relevant information. Language, with its consideration of sequence modeling, semantic information, and image context, proves instrumental in this process. The fusion of NLP’s language processing capabilities with HSI’s detailed spectral information opens avenues for holistic and context-aware applications. This article explores the convergence of NLP and HSI, highlighting their potential synergy in HSI tasks. It begins by elucidating the fundamental principles of integrating NLP and HSI, establishing a foundational understanding for readers unfamiliar with the integration. Subsequently, the synthesis of NLP and HSI is examined, with a focus on recent developments, challenges, and potential applications. The review addresses the integration of sequence models, language models, transformer-based architectures, and image–text models with HSI for various tasks, including HSI classification, feature extraction, semantic segmentation, image captioning, visual question answering (VQA), object detection, hyperspectral unmixing, text–image retrieval, change detection, image reconstruction, and hyperspectral image superresolution. It critically evaluates the advantages and future research perspective of using natural language-based architectures and concepts HSI applications and tasks.

Topics & Concepts