Deep Learning Augmented Realistic Avatars for Social VR Human Representation
Matthijs van der Boon, Leonor Fermoselle, Frank ter Haar, Sylvie Dijkstra-Soudarissanane, Omar Niamut
Abstract
Virtual reality (VR) has created a new and rich medium for people to meet each other digitally. In VR, people can choose from a broad range of representations. In several cases, it is important to provide users with avatars that are a lifelike representation of themselves, to increase the user experience and effectiveness of communication. In this work, we propose a pipeline for generating a realistic and expressive avatar from a single reference image. The pipeline consists of a blendshape-based avatar combined with two deep learning improvements. The first improvement module runs offline and improves the texture map of the base avatar. The second module runs inference in real-time at the rendering stage and performs a style transfer to the avatar’s eyes. The deep learning modules effectively improve the visual representation of the avatar and show how AI techniques can be integrated with traditional animation methods to generate realistic human avatars for social VR.