Efficient Volumetric Video Streaming Through Super Resolution
Anlan Zhang, Chendong Wang, Bo Han, Feng Qian
Abstract
Volumetric videos allow viewers to exercise 6-DoF (degrees of freedom) movement when consuming fully 3D content (e.g., point clouds). Due to their truly immersive nature, streaming volumetric videos is highly bandwidth-demanding. In this work, we present to our knowledge a first volumetric video streaming system that leverages 3D super resolution (SR) of point clouds to boost the video quality on commodity devices, and to facilitate the distribution of volumetric content over bandwidth-constrained wireless networks. However, directly applying off-the-shelf 3D SR models leads to unacceptably low performance (~ 0.1 FPS even on a powerful GPU). To overcome this limitation, we propose a series of optimizations to make SR efficient. Our preliminary results indicate that for an edge-assisted (standalone mobile) setup, a small subset of our proposed optimizations can already drastically improve the FPS by a factor of 131× (53×) and reduce GPU memory usage by 83% (76%), while maintaining the same or even better SR inference accuracy, compared to using an off-the-shelf SR model.