Attention-Based Multimodal Image Feature Fusion Module for Transmission Line Detection
Hyeyeon Choi, Jong Pil Yun, Bum Jun Kim, Hyeonah Jang, Sang Woo Kim
Abstract
Transmission line (TL) inspection is important for ensuring a stable supply of electricity to rural areas. Currently, there are several TL detection approaches based on computer vision; however, they have limitations owing to background clutter in visible light images. This article presents a novel multimodal image feature fusion module that utilizes both visible light and infrared images to enhance the TL-detection performance. The proposed module consists of a multibranch feature extraction (MFE) block followed by a channelwise attention (CA) block. The first block extracts the representative features of each modal input using multiple branches. The outputs of the MFE block are jointly aggregated into an attention vector in the CA block. Finally, the attention vector recalibrates each input feature of the proposed module. To reduce the number of additional parameters due to the insertion of the module, we introduced a channel-shrink factor in the MFE block and utilized a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$1\times {1}$</tex-math></inline-formula> convolution in the CA block. Comparison experiments with various augmented conditions of day, night, fog, and snow were conducted on a real-world dataset, which we constructed by visible light and infrared images. The results showed that the proposed module outperformed not only the case of single modal input but also the state-of-the-art fusion methods, regardless of the baseline networks. Additionally, the proposed module showed effectiveness in terms of capacity when the baseline network has a large number of weight parameters.