3D Reconstruction with Spatial Memory

Abstract

We present Spann3R, a novel approach for dense 3D reconstruction from ordered or unordered image collections. Built on the DUSt3R paradigm, Spann3R uses a transformer-based architecture to directly regress pointmaps from images without any prior knowledge of the scene or camera parameters. Unlike DUSt3R, which pre-dicts per image-pair pointmaps expressed in a local coordinate frame, Spann3R predicts per-image pointmaps expressed in a global coordinate system, thus eliminating the need for optimization-based global alignment. The key idea behind Spann3R is to manage an external spa-tial memory that learns to keep track of all previous relevant 3D information. Spann3R then queries this spatial memory to predict the 3D structure of the next frame in a global coordinate system. Taking advantage of DUSt3R's pre-trained weights, and further fine-tuning on a subset of datasets, Spann3R shows competitive performance and generalization ability on various unseen datasets and can process ordered image collections in real-time. Project page: https://hengyiwang.github.io/projects/spanner

Topics & Concepts