Cooperative Connection Transformer for Remote Sensing Image Captioning
Kai Zhao, Wei Xiong
Abstract
Feature extraction is fundamental for successful remote sensing image captioning (RSIC). The representation forms of grid features and region features differ significantly. Grid features can provide fine-grained information while retaining background information for subsequent processes but lack salient target information. Region features can provide object-level information but lack contextual information. Thus, to fully utilise the advantages of both and achieve RSIC tasks, we propose a Cooperative Connection Transformer (CCT). In the encoder, we specifically designed grid and region features. We used the features after global average pooling as grid features to provide global background information. We also extract region features to provide salient region information. To solve the information loss caused by region feature extraction, we propose mapping region features and fusing them with grid features using a fusion attention module that filters out the information redundancy and noise generated by fusion. Cooperative connection attention is added to the decoder to better utilise different feature types to improve its perception of different features by directly connecting them to each decoder layer. To address the lack of region feature annotations in the RSIC field, we provide region feature annotations based on published datasets and extract region features accordingly. Extensive experiments demonstrate the effectiveness and superiority of the proposed method. Labelled region annotations are available at https://github.com/zk-1019/region-annotations.