.png){kind=icon}
.png){kind=icon}
Return
Diff-ZsVQA: Zero-shot Visual Question Answering with Frozen Large Language Models Using Diffusion Model
.png){kind=icon}
PRE
AI
DOI：10.1016/j.eswa.2025.126951.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Visual Question Answering (VQA) methods leveraging Large Language Models (LLMs) aim to enhance performance in few/zero-shot scenarios. While the results attained by these approaches were outstanding, there remains scope for further enhancement. Given the remarkable capabilities demonstrated by Diffusion Models (DMs), we recognize that the DMs can potentially improve the performance of VQA by optimizing the generation of captions. Furthermore, existing approaches for prompt construction neglect the influence of non- original questions and generated question-answer (QA) pairs, which leads to adverse effects on the inference. This paper proposes a novel framework called Diffused Zero-shot VQA, shortly Diff-ZsVQA, which innovatively incorporates a powerful DM into the LLM-based VQA pipeline for image-to-text converting. Moreover, to reduce the impact of non-original questions and generated QA pairs, we devise an Original-Question-Centric (OQC) prompt whose examples' questions are identical while contexts are diverse. We first construct initial prompts to formulate answer candidates, then the final answer is selected among options in answer heuristics via OQC prompting. Compared with previous LLM-based VQA methods, the proposed architecture is simpler and it brings a higher efficiency to predictions in zero-shot VQA. Extensive experiments demonstrate that Diff-ZsVQA with OQC prompt achieves competitive performance with higher inference speed than most existing methods.
Keywords:
Multi-modal
Visual question answering
Zero-shot
Diffusion models
Large language models
Prompt
Journal
IF:
7.5
Papers:
2.9W
Citations:
10.2W
Organization
No organization information available