Super-Resolution Reconstruction for Stereoscopic Omnidirectional Display Systems via Dynamic Convolutions and Cross-View Transformer
Xiongli Chai, Feng Shao, Hangwei Chen, Baoyang Mu, Yo‐Sung Ho
Abstract
Stereoscopic OmniDirectional Images (SODIs) usually require recording very High-Resolution (HR) information whereby it is beneficial to exploit a Super-Resolution (SR) scheme to super-resolve Low-Resolution (LR) SODIs. Compared with traditional 2D SR approaches, the algorithms of SODI SR (SODI-SR) need to deal with two extra aspects: binocular information and panoramic characteristics. In this paper, we first build a synthetic-specific SODI-SR dataset with LR-HR image pairs. Then we propose a Dynamic convolutions and Transformer Network (Dconv-Trans-Net) for SODI-SR. Specifically, due to the non-uniform sampling of EquiRectangular Projection (ERP), we deploy dynamic convolutions with the structure of Atrous Spatial Pyramid Pooling (ASPP) to adaptively select content-aware and weight-aware kernels for patch-wise feature extraction. To capture diverse feature embeddings from left and right views, we utilize a symmetric bi-directional Parallax Attention Module (biPAM) to extract local features along epipolar lines and propose Cross-view Transformer (CvTrans) to mine global contextual features apart from the epipolar line. Finally, quantitative and qualitative experiments demonstrate that our proposed approach outperforms the state-of-the-art panoramic or stereoscopic SR methods on the constructed SODI-SR dataset with two upscaling factors.