.png){kind=icon}
返回
Visually-Guided Audio Spatialization in Video with Geometry-Aware Multi-task Learning
.png){kind=icon}
PRE
AI
DOI：10.1007/s11263-023-01816-8.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Binaural audio provides human listeners with an immersive spatial sound experience, but most existing videos lack binaural audio recordings. We propose an audio spatialization method that draws on visual information in videos to convert their monaural (single-channel) audio to binaural audio. Whereas existing approaches leverage visual features extracted directly from video frames, our approach explicitly disentangles the geometric cues present in the visual stream to guide the learning process. In particular, we develop a multi-task framework that learns geometry-aware features for binaural audio generation by accounting for the underlying room impulse response, the visual stream's coherence with the sound source(s) positions, and the consistency in geometry of the sounding objects over time. Furthermore, we introduce two new large video datasets: one with realistic binaural audio simulated for real-world scanned environments, and the other with pseudo-binaural audio obtained from ambisonic sounds in YouTube 360(?) videos. On three datasets, we demonstrate the efficacy of our method, which achieves state-of-the-art results.
Keyword:
Audio spatialization
Binaural audio generation
Video
Audio-visual
Multi-task learning
期刊
IF:
9.3
论文数:
3.9K
被引数:
2.8W
机构
引用论文
Bursting Oscillations and the Mechanism with Sliding Bifurcations in a Filippov Dynamical System
PREAI
分享收藏Characterization of the Corrosion Products Formed on Carbon Steel in Qinghai Salt Lake Atmosphere.png)
Corrosion
CorrosionIF0
PREAI
分享收藏
分享收藏分享收藏
查看更多内容