Enhancing few-shot KB-VQA with panoramic image captions guided by Large Language Models
Pengpeng Qiang, Hongye Tan, Xiaoli Li, Dian Wang, Ru Li, Xinyi Sun, Hu Zhang, Jiye Liang
Abstract
Current state-of-the-art (SOTA) KB-VQA techniques involve transforming images into image captions as prompts to harness the potent reasoning capabilities of large language models (LLMs) for generating answers. However, generic image captions often fall short in capturing crucial visual details, essential for LLMs to deliver precise responses . To address this challenge, we propose an image captioning model that effectively utilizes a set of visual language models , such as BLIP2, GRiT, OCR , etc., to extract rich visual information from images. Subsequently, we employ the inferential and summarization capabilities of LLM to generate panoramic image descriptions enriched with intricate details . Simultaneously, we employ Contextual Constraint Examples and Constraint Instruction to mitigate the potential hallucination issues arising from LLM-generated image captions. Extensive experiments validate the superiority and scalability of our proposed method, achieving significant improvements over SOTA methods in challenging few-shot settings. For instance, on the challenging OK-VQA, our method outperforms PICa by 6.5%. On the VQAv2 dataset, our method surpasses the SOTA approach by 5.4%.