DDFAV: Remote Sensing Large Vision Language Models Dataset and Evaluation Benchmark

Abstract

With the rapid development of large visual language models (LVLMs) and multimodal large language models (MLLMs), these models have demonstrated strong performance in various multimodal tasks. However, alleviating the generation of hallucinations remains a key challenge in LVLMs research. For remote sensing LVLMs, there are problems such as low quality, small number and unreliable datasets and evaluation methods. Therefore, when applied to remote sensing tasks, they are prone to hallucinations, resulting in unsatisfactory performance. This paper proposes a more reliable and effective instruction set production process for remote sensing LVLMs to address these issues. The process generates detailed and accurate instruction sets through strategies such as shallow-to-deep reasoning, internal and external considerations, and manual quality inspection. Based on this production process, we collect 1.6 GB of remote sensing images to create the DDFAV dataset, which covers a variety of remote sensing LVLMs tasks. Finally, we develop a closed binary classification polling evaluation method, RSPOPE, specifically designed to evaluate hallucinations in remote sensing LVLMs or MLLMs visual question-answering tasks. Using this method, we evaluate the zero-shot remote sensing visual question-answering capabilities of multiple mainstream LVLMs. Our proposed dataset images, corresponding instruction sets, and evaluation method files are all open source.

Topics & Concepts