A Bayesian Treatment of Real-to-Sim for Deformable Object Manipulation
Rika Antonova, Jingyun Yang, Priya Sundaresan, Dieter Fox, Fábio Ramos, Jeannette Bohg
Abstract
We consider the problem of inferring simulation parameters such that the behavior of an object in simulation and the real world look similar. This <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">real-to-sim</i> problem is particularly challenging for deformable objects, where conventional techniques fall short as they often rely on precise state estimation and accurate dynamics models. In this letter, we formulate the real-to-sim problem as probabilistic inference over simulation parameters of deformable objects. Our key idea is in how we define the state space of a deformable object. We view noisy keypoints extracted from an image of the object as samples from the distribution that captures object geometry. We then embed this distribution into a reproducing kernel Hilbert space. A sequence of images of a moving object can then be represented by a trajectory of distribution embeddings in this novel state space for deformables. This allows for a principled way to incorporate noisy state observations into modern Bayesian tools for simulation parameter inference. Despite only using a small set of real-world trajectories, we show how the proposed approach can estimate posterior distributions over simulation parameters, such as elasticity, friction and scale, even for highly deformable objects, such as cloth and ropes.