.png){kind=icon}
.png){kind=icon}
Return
Enhancing Compositional Reasoning in Multimodal Large Language Models
.png){kind=icon}
PRE
AI
DOI：10.1007/978-981-95-5679-3_6.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Recent studies have highlighted the shortage in compositional reasoning capabilities within Vision-Language Models (VLMs) such as CLIP and SigLIP. While these deficiencies have been documented in VLMs, their transmission to Multimodal Large Language Models (MLLMs) which frequently rely on VLM derived visual encoders has not been thoroughly explored. This paper systematically investigates the compositional reasoning deficiencies in MLLMs and demonstrates that these models inherit the limitations of their underlying visual architectures. To address this challenge, we propose a vision-language compositionality enhancement strategy aimed at improving compositional reasoning while maintaining the MLLM's overall multimodal understanding and reasoning capabilities. Our approach integrates two key innovations: (1) adopting multi-layer visual features as inputs of MLLMs to enhance fine-grained visual information, which is essential for effective compositional reasoning, and (2) a contrastive learning (CL) stage intended to improve the compositional reasoning abilities of MLLMs. Extensive experimental evaluations demonstrate that our auxiliary training strategy significantly enhances the performance of LLaVA-1.5-7B on compositional reasoning benchmarks, achieving performance parity with GPT4V on specific tasks. Importantly, this strategy not only preserves but also improves performance across general multimodal benchmarks, highlighting its dual efficacy in both specialization and generalization.
Keywords:
Compositional Reasoning
Multimodal Large Language Models
Vision-Language Models
Journal
P
IF:
0
Papers:
31
Citations:
0