Region-Based Representations Revisited
Michal Shlapentokh-Rothman, Ansel Blume, Yao Xiao, Yuqun Wu, T V Sethuraman, Heyi Tao, Jaeyong Lee, Wilfredo Torres, Yu-Xiong Wang, Derek Hoiem
Abstract
We investigate whether region-based representations are effective for recognition. Regions were once a mainstay in recognition approaches, but pixel and patch-based features are now used almost exclusively. We show that recent class-agnostic segmenters like SAM can be effectively combined with strong self-supervised representations, like those from DINOv2, and used for a wide variety of tasks, including semantic segmentation, object-based image re-trieval, and multi-image analysis. Once the masks and features are extracted, these representations, even with linear decoders, enable competitive performance, making them well suited to applications that require custom queries. The representations' compactness also makes them well-suited to video analysis and other problems requiring inference across many images.