Diff-ZsVQA: Zero-shot Visual Question Answering with Frozen Large Language Models Using Diffusion Model
Quanxing Xu, Jian Li, Yuhao Tian, Ling Zhou, Feifei Zhang, Rubing Huang
Abstract
Visual Question Answering (VQA) methods leveraging Large Language Models (LLMs) aim to enhance performance in few/zero-shot scenarios. While the results attained by these approaches were outstanding, there remains scope for further enhancement. Given the remarkable capabilities demonstrated by Diffusion Models (DMs), we recognize that the DMs can potentially improve the performance of VQA by optimizing the generation of captions. Furthermore, existing approaches for prompt construction neglect the influence of non-original questions and generated question-answer (QA) pairs, which leads to adverse effects on the inference. This paper proposes a novel framework called Diff used Z ero- s hot VQA, shortly Diff-ZsVQA, which innovatively incorporates a powerful DM into the LLM-based VQA pipeline for image-to-text converting. Moreover, to reduce the impact of non-original questions and generated QA pairs, we devise an Original-Question-Centric (OQC) prompt whose examples’ questions are identical while contexts are diverse. We first construct initial prompts to formulate answer candidates, then the final answer is selected among options in answer heuristics via OQC prompting. Compared with previous LLM-based VQA methods, the proposed architecture is simpler and it brings a higher efficiency to predictions in zero-shot VQA. Extensive experiments demonstrate that Diff-ZsVQA with OQC prompt achieves competitive performance with higher inference speed than most existing methods.