.png){kind=icon}
Return
Learning Compositionality from Multifaceted Synthetic Data for Language-based Object Detection
.png){kind=icon}
PRE
AI
DOI：10.1007/s11263-025-02554-9.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Language-based object detection aims to locate target objects from complex language queries. However, current vision-language detectors often struggle to understand complex representations of visual objects (e.g., attributes, shapes, and relationships), especially under complex queries. In this paper, we first conduct a thorough analysis of current language-based detectors to identify their specific weaknesses in compositional understanding. To this end, we propose a novel comprehensive evaluation framework that automatically categorizes test cases by the type and complexity of compositionality, leveraging large language models (LLMs). This reveals that detectors show significant performance drops with increased complexity and consistent failures in specific types, such as spatial and numerical reasoning. To effectively address this, we propose a multifaceted synthetic data consisting of (1) generative model-based synthetic triplets that inherited compositional knowledge from large generative models (e.g., LLMs, diffusion models) in the form of triplets (i.e., image-text-box data); and (2) weakness-targeted synthetic descriptions designed to enhance understanding in vulnerable types like spatial and numeracy concepts. We further introduce a compositional contrastive learning method to better leverage the proposed synthetic data while mitigating the common drawbacks of synthetic data. Consequently, our models trained on proposed multifaceted synthetic data exhibit a significant performance boost in the Omnilabel benchmark by up to +7.1AP and the $$\hbox {D}^{3}$$ benchmark by up to $$+8.4$$ AP upon existing baselines.
Keywords:
Multimodal Synthetic Datasets
Compositional Learning
Transfer Learning
Language-based Object Detection
Journal
IF:
9.3
Papers:
3.9K
Citations:
2.8W
Organization
Cited Papers
.png){kind=icon}
ShareSave
Weak-to-Strong Compositional Learning from Generative Models for Language-Based Object Detection
PREAI
ShareSaveShareSave
View more