Deep HyFeat Based Attention in Attention Model for Face Super-Resolution

Abstract

Face super-resolution (SR) is the task of generating high-resolution (HR) face images from the low-resolution (LR) inputs. Recently, deep learning-based methods have shown remarkable progress in the SR field. Most of the methods perform additional tasks such as face parsing, landmark, and attention to generate the HR images. However, parsing maps and landmark-guided models require the supplementary labeled dataset, which is difficult to obtain in real life. The attention mechanism does not require the datasets extra labeling and is also beneficial for face SR. These methods focus on a few critical features and ignore the remaining ones, which sometime causes to ignore the valuable features. Therefore, this article proposes a novel deep hybrid feature (HyFeat)-based Attention in Attention model for face SR. Moreover, the proposed model uses the coarse SR network and deep convolutional neural network (CNN) to generate the HR image. A coarse SR network is applied to upsample the LR image and generate the coarse super-resolved image, which is further sent to the deep CNN model. The proposed work incorporates the HyFeat attention in attention unit (HyFA <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$^{\mathrm{ 2}}\text{U}$ </tex-math></inline-formula> ), which consists of HyFeat block and attention in attention block (A <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$^{\mathrm{ 2}}\text{B}$ </tex-math></inline-formula> ) in the deep CNN model to improve the visual quality of the output face images. HyFeat block assists the model in extracting the coarse features and learning the enriched contextual information to enhance the details of coarse features. A <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$^{\mathrm{ 2}}\text{B}$ </tex-math></inline-formula> preserves the attentive and non-attentive beneficial features while suppressing the unwanted features. The attention branch focuses on specific facial features and ignores the rest of the features. The non-attention branch aims to learn the informative features that the attention branch ignores. The proposed model repeats the HyFA <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$^{\mathrm{ 2}}\text{U}$ </tex-math></inline-formula> units to focus on different facial components and enhance the features to improve the quality of resultant faces. Experimental outcomes exhibit that the proposed model gains state-of-the-art performance on the standard datasets, namely CelebAHQ, Helen, FFHQ, and LFW face. The proposed method achieves an improvement of more than 0.35 dB in peak signal-to-noise ratio (PSNR) and 0.012 in structure similarity (SSIM) on different datasets over the best models available in the literature.

Topics & Concepts