Deformable Convolution Alignment and Dynamic Scale-Aware Network for Continuous-Scale Satellite Video Super-Resolution
Ning Ni, Libao Zhang
Abstract
Recently, due to higher requirements for satellite video resolution, video super-resolution (VSR) has been extensively studied. However, the following problems have not been effectively resolved: 1) Previous satellite VSR methods cannot achieve continuous-scale (integer and non-integer scale) VSR with a single model. 2) Satellite video has complex ground and weak textures, which increases the difficulty of capturing motion information. In addition, existing methods adopt a unified alignment path, which leads to a drop in feature alignment accuracy. 3) During feature fusion, previous methods ignore the correlation of spatio-temporal information in satellite video and cannot make full use of the spatio-temporal information. To address the above problems, in this paper, we propose a novel network for continuous-scale satellite VSR (CSVSR). Specifically, first, for effective motion capture and accurate feature alignment, we design a residual-guided and time-aware dynamic routing alignment module, which can use feature residuals to lock motion areas and then dynamically select the corresponding alignment path based on the temporal distance. Second, we proposed a non-local mask-based feature fusion module to exploit the correlation of the spatio-temporal features and complete effective spatio-temporal feature fusion. Third, to make our network adapt to multi-task learning, we develop a scale-aware convolutional (SA-Conv) layer, which lets our network dynamically extract scale-adaptive features according to the input scale factors. Finally, we propose a continuous-scale upsampling module with a global feature implicit function (GFIF), which can achieve continuous-scale mapping from features to pixel values. In addition, we carefully design a novel training strategy to optimize our network. Comprehensive experiments verify that the proposed CSVSR has superior reconstruction performance on continfuous-scale factors. The code will be available at https://github.com/chongningni/CSVSR.