GeoPix: A multimodal large language model for pixel-level image understanding in remote sensing

Abstract

Multimodal (MM) large language models (MLLMs) have achieved remarkable success in image- and region-level remote sensing (RS) image understanding tasks, such as image captioning (IC), visual question answering (VQA), and visual grounding. However, existing RS MLLMs lack pixel-level dialogue capability, which involves responding to user instructions with segmentation masks for specific instances. In this article, we propose GeoPix, an RS MLLM that extends image understanding capabilities to the pixel level. This is achieved by equipping the MLLM with a mask predictor, which transforms visual features from the vision encoder into masks conditioned on the LLM’s segmentation token embeddings. To facilitate the segmentation of multiscale objects in RS imagery, a class-wise learnable memory (CLM) module is integrated into the mask predictor to capture and store class-wise geo-context at the instance level across the entire dataset. In addition, to address the absence of large-scale datasets for training pixel-level RS MLLMs, we construct the GeoPixInstruct dataset, comprising 65,463 images and 140,412 instances, with each instance annotated with text descriptions, bounding boxes, and masks. Furthermore, we develop a two-stage training strategy to balance the distinct requirements of text generation and mask prediction in MM multitask optimization. Extensive experiments verify the effectiveness and superiority of GeoPix in pixel-level segmentation tasks while also maintaining competitive performance in image- and region-level benchmarks. The models, dataset, and code are publicly available at <ext-link ext-link-type="uri" xlink:href="https://github.com/Norman-Ou/GeoPix" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Norman-Ou/GeoPix</ext-link>.

Topics & Concepts