.png){kind=icon}
返回
Enabling Visual Object Detection With Object Sounds via Visual Modality Recalling Memory
.png){kind=icon}
PRE
AI
DOI：10.1109/TNNLS.2023.3323560.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
When humans hear the sound of an object, they recall associated visual information and integrate the sound with recalled visual modality to detect the object. In this article, we present a novel sound-based object detector that mimics this process. We design a visual modality recalling (VMR) memory to recall information of a visual modality based on an audio modal input (i.e., sound). To achieve this goal, we propose a VMR loss and an audio-visual association loss to guide the VMR memory to memorize visual modal information by establishing associations between audio and visual modalities. With the visual modal information recalled through the VMR memory along with the original audio input, we perform audio-visual integration. In this step, we introduce an integrated feature contrastive loss that allows the integrated feature to be embedded as if it were encoded using both audio and visual modal inputs. This guidance enables our sound-based object detector to effectively perform visual object detection even when only sound is provided. We believe that our work is a cornerstone study that offers a new perspective to conventional object detection studies that solely rely on the visual modality. Comprehensive experimental results demonstrate the effectiveness of the proposed method with the VMR memory.
Keyword:
Memory network
modality recalling
object sound
visual object detection
期刊
IF:
8.9
论文数:
7.6K
被引数:
7.2W
机构
暂无机构信息
引用论文
LOCATING STATIONS OF PUBLIC TRANSPORTATION VEHICLES FOR IMPROVING TRANSIT ACCESSIBILITY.png)
TRANSPORT
OAAI
TRANSPORTIF0
OAAI分享收藏分享收藏Thermal comfort, perceived air quality, and cognitive performance when personally controlled air movement is used by tropically acclimatized persons当热带适应的人使用个人控制的空气运动时，热舒适性，感知的空气质量和认知表现Indoor Air
OAAI
Indoor AirIF0
OAAI分享收藏
查看更多内容