.png){kind=icon}
.png){kind=icon}
Return
Multimodal query-guided object localization
.png){kind=icon}
PRE
AI
DOI：10.1007/s11042-023-15779-y.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Recent studies have demonstrated the effectiveness of using hand-drawn sketches of objects as queries for one-shot object localization. However, hand-drawn crude sketches alone can be ambiguous for object localization, which could result in misidentification, e.g., a sketch of a laptop could be confused for a sofa. To overcome this, we propose a novel multimodal approach to object localization that combines sketch queries with linguistic category definitions, allowing for a better representation of visual and semantic cues. Our approach employs a cross-modal attention scheme that guides the region proposal network to obtain relevant proposals. Further, we propose an orthogonal projection-based proposal scoring technique that effectively ranks proposals with respect to the query. We evaluated our method using hand-drawn sketches from the 'Quick, Draw!' dataset and glosses from 'WordNet' as queries on the widely-used MS-COCO dataset, and achieve superior performance compared to related baselines in both open- and closed-set settings.
Keywords:
Sketch
Open-set object localization
Gloss
Cross-modal localization
Cross-modal attention
Journal
IF:
3
Papers:
1.9W
Citations:
3.2W