.png){kind=icon}
Return
Sci-Phi: A Large Language Model Spatial Audio Descriptor
.png){kind=icon}
PRE
AI
DOI：10.1109/OJSP.2026.3657300.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Acoustic scene perception involves describing the type of sounds, their timing, their direction and distance, as well as their loudness and reverberation. While audio language models excel in sound recognition, single-channel input fundamentally limits spatial understanding. This work presents Sci-Phi, a spatial audio large language model with dual spatial and spectral encoders that estimates a complete parameter set for all sound sources and the surrounding environment. Learning from over 4,000 hours of synthetic first-order Ambisonics recordings including metadata, Sci-Phi enumerates and describes up to four directional sound sources in one pass, alongside non-directional background sounds and room characteristics. We evaluate the model with a permutation-invariant protocol and 15 metrics covering content, location, timing, loudness, and reverberation, and analyze its robustness across source counts, signal-to-noise ratios, reverberation levels, and challenging mixtures of acoustically, spatially, or temporally similar sources. Notably, Sci-Phi generalizes to real room impulse responses with only minor performance degradation. Overall, this work establishes the first audio LLM capable of full spatial-scene description, with strong potential for real-world deployment. Demo: https://sci-phi-audio.github.io/demo
Keywords:
Spatial audio
Reverberation
Metadata
Vectors
Three-dimensional displays
Location awareness
Training
Spectrogram
Large language models
Computer architecture
large language model
acoustic scene understanding
Journal
IF:
2.7
Papers:
156
Citations:
535
Organization
Cited Papers
ShareSaveEcho-Aware Adaptation of Sound Event Localization and Detection in Unknown Environments.png)
ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
OAAI
ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)IF0
OAAIShareSaveMulti-ACCDOA: Localizing And Detecting Overlapping Sounds From The Same Class With Auxiliary Duplicating Permutation Invariant TrainingICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)OAAI
ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)IF0
OAAIShareSaveExploring Self-supervised Contrastive Learning of Spatial Sound Event RepresentationICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)OAAI
ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)IF0
OAAIShareSaveSound Event Localization Based on Sound Intensity Vector Refined by Dnn-Based Denoising and Source SeparationICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
OAAI
ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)IF0
OAAIShareSaveNatural Language Supervision For General-Purpose Audio RepresentationsICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)OAAI
ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)IF0
OAAIShareSaveShareSaveShareSave
View more