Disparity-Guided Light Field Image Super-Resolution via Feature Modulation and Recalibration
Gaosheng Liu, Huanjing Yue, Kun Li, Jingyu Yang
Abstract
The disparity information reflects pixel-wise inter-view correlations among sub-aperture images (SAIs) of a light field (LF) image. Existing CNN-based methods for LF spatial super-resolution (SR) rarely utilize the disparities to incorporate the underlying inter-view correlations due to the lack of ground-truth disparity labels. In this paper, we incorporate a self-supervised training strategy to estimate disparity maps, and simultaneously propose a disparity-guided network for LF spatial SR (i.e., LF-DGNet), in which the estimated disparity maps are utilized to modulate the LF features. Specifically, we introduce a disparity-guided feature modulation (DGFM) module where the disparity maps are utilized to produce affine transformation parameters for each view-wise feature. We also propose a multi-view feature recalibration (MFR) module which consists of a triple attention (TA, i.e., spatial, channel, and angular attention) to enhance the ability of feature representation and further explore the intra- and inter-view correlations. Experimental results on public LF datasets including large-disparity LF images demonstrate that our proposed method achieves state-of-the-art performance both in quantitative results and visual quality. Our codes are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/GaoshengLiu/LF-DGNet</uri>