.png){kind=icon}
返回
Sci-Phi: A Large Language Model Spatial Audio Descriptor
.png){kind=icon}
PRE
AI
DOI：10.1109/OJSP.2026.3657300.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
声景感知涉及对声音类型、时间、方向和距离，以及响度和混响的描述。尽管音频语言模型在声音识别方面表现出色，但单声道输入从根本上限制了空间理解能力。本研究提出Sci-Phi，一种具有双空间和频谱编码器的空间音频大型语言模型，可估算所有声源和周围环境的完整参数集。通过学习超过4,000小时的包含元数据的合成一阶全向声录音，Sci-Phi能够在一次处理中枚举和描述最多四个方向性声源，同时处理非方向性背景声音和房间特征。我们采用置换不变协议和15项涵盖内容、位置、时间、响度和混响的指标对模型进行评估，并分析其在声源数量、信噪比、混响水平以及声学、空间或时间上相似源的复杂混合物方面的鲁棒性。值得注意的是，Sci-Phi仅需轻微性能下降即可泛化到真实房间脉冲响应。总体而言，本研究建立了首个能够进行完整空间场景描述的音频大型语言模型，具有实际部署的巨大潜力。演示：https://sci-phi-audio.github.io/demo
Keyword:
Spatial audio
Reverberation
Metadata
Vectors
Three-dimensional displays
Location awareness
Training
Spectrogram
Large language models
Computer architecture
large language model
acoustic scene understanding
期刊
IF:
2.7
论文数:
156
被引数:
535
机构
引用论文
Echo-Aware Adaptation of Sound Event Localization and Detection in Unknown Environments.png)
ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
OAAI
ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)IF0
OAAI分享收藏Multi-ACCDOA: Localizing And Detecting Overlapping Sounds From The Same Class With Auxiliary Duplicating Permutation Invariant TrainingICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)OAAI
ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)IF0
OAAI分享收藏Exploring Self-supervised Contrastive Learning of Spatial Sound Event Representation探索空间声音事件表示的自监督对比学习ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)OAAI
ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)IF0
OAAI分享收藏Sound Event Localization Based on Sound Intensity Vector Refined by Dnn-Based Denoising and Source SeparationICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
OAAI
ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)IF0
OAAI分享收藏Natural Language Supervision For General-Purpose Audio Representations通用音频表示的自然语言监督ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)OAAI
ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)IF0
OAAI分享收藏
查看更多内容