2.7 MetaVRain: A 133mW Real-Time Hyper-Realistic 3D-NeRF Processor with 1D-2D Hybrid-Neural Engines for Metaverse on Mobile Devices
Donghyeon Han, Junha Ryu, Sangyeob Kim, Sangjin Kim, Hoi‐Jun Yoo
Abstract
A neural radiance field (NeRF) [1] uses a deep neural network (DNN) to create 3D models by training the DNN to memorize 3D scene geometry from a few photos. Prior work uses conventional computer graphic algorithms, such as ray-tracing or SLAM, for the same purpose. With NeRF, the generated model can display hyper-realistic 3D content on the metaverse, with quality better than or the same as 3D images rendered by complicated ray-tracing. The 3D model can also be shared with other metaverse users via low-bandwidth communication because the transaction requires <1MB of parameters. NeRF is promising for 3D reconstruction, but also for a wide range of applications from depth estimation to 3D style transfer [2], however, its heavy computational demands stand in the way of its applicability for mobile and wearable applications.