360 Panorama Synthesis from a Sparse Set of Images on a Low-Power Device
Julius Surya Sumantri, In Kyu Park
Abstract
A full 360 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">o</sup> × 180 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">o</sup> image provides an unlimited field of view (FOV) and an immersive experience for the users without any loss of information of the surrounding. In this study, a deep learning based approach is proposed to synthesize a 360 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">o</sup> image from a sparse set of images captured with a limited FOV. The proposed network consists of a cascade of the FOV estimation network and the panorama synthesis network. We propose a hierarchical generative network to synthesize high quality 360 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">o</sup> panorama images. The design of progressive multi-scale generator and multiple discriminator reduces the high frequency artifact which is commonly observed in image synthesis using generative networks. The network is further compressed to run on a low-power device such as a smartphone using our proposed size and latency optimization. Experimental result demonstrates that the proposed method produces 360 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">o</sup> panorama with satisfactory image quality of up to 512 × 1024 resolution. It is also shown that the proposed method outperforms the alternative method and can be generalized for non-panoramic scenes and real images captured by a smartphone camera.