.png){kind=icon}
Return
Enabling Visual Object Detection With Object Sounds via Visual Modality Recalling Memory
.png){kind=icon}
PRE
AI
DOI：10.1109/TNNLS.2023.3323560.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
When humans hear the sound of an object, they recall associated visual information and integrate the sound with recalled visual modality to detect the object. In this article, we present a novel sound-based object detector that mimics this process. We design a visual modality recalling (VMR) memory to recall information of a visual modality based on an audio modal input (i.e., sound). To achieve this goal, we propose a VMR loss and an audio-visual association loss to guide the VMR memory to memorize visual modal information by establishing associations between audio and visual modalities. With the visual modal information recalled through the VMR memory along with the original audio input, we perform audio-visual integration. In this step, we introduce an integrated feature contrastive loss that allows the integrated feature to be embedded as if it were encoded using both audio and visual modal inputs. This guidance enables our sound-based object detector to effectively perform visual object detection even when only sound is provided. We believe that our work is a cornerstone study that offers a new perspective to conventional object detection studies that solely rely on the visual modality. Comprehensive experimental results demonstrate the effectiveness of the proposed method with the VMR memory.
Keywords:
Memory network
modality recalling
object sound
visual object detection
Journal
IF:
8.9
Papers:
7.6K
Citations:
7.2W
Organization
No organization information available
Cited Papers
LOCATING STATIONS OF PUBLIC TRANSPORTATION VEHICLES FOR IMPROVING TRANSIT ACCESSIBILITY.png)
TRANSPORT
OAAI
TRANSPORTIF0
OAAIShareSaveShareSaveShareSaveThermal comfort, perceived air quality, and cognitive performance when personally controlled air movement is used by tropically acclimatized personsIndoor Air
OAAI
Indoor AirIF0
OAAIShareSave
View more