.png){kind=icon}
.png){kind=icon}
Return
Low-resource video-conditioned music generation via reliability-aware visual–text–audio fusion
.png){kind=icon}
PRE
AI
Y
Y
DOI：10.1007/s00371-026-04615-7.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Video-conditioned background music generation under licensed-data constraints requires models to use sparse visual dynamics, dense textual intent, and acoustic context without destabilizing a pretrained audio prior. We present Adaptive Multimodal Composition (AMC), a parameter-efficient visual–text–audio adaptation framework that freezes the AudioLDM acoustic backbone and trains lightweight Temporal Alignment (TAM), Adaptive Multimodal Fusion (AMF), Cross-Modal Transformer (CMT), and Temporal Semantic Fusion (TSF) components. TAM places visual and acoustic features on a compatible one-second temporal grid; AMF estimates block-wise condition-specific usefulness; CMT models non-local structure in the aligned fused sequence; and TSF supplies a training-only, one-way multimodal-to-text semantic anchor. Under a controlled 100-h licensed-data protocol, AMC obtains FAD/KL scores of 1.66/0.19 on the complete in-domain evaluation while updating 4.8% of system parameters. On MusicCaps, where no paired video exists, masked visual–text–audio inference obtains 2.01/0.21 and is interpreted only as out-of-domain semantic transfer. Learned reliability weights correlate significantly with empirical marginal utility, with Spearman correlations of 0.52, 0.60, and 0.46 for the visual, text, and audio-context modalities, respectively (all p < 0.001), and 30/60-s evaluations provide evidence that CMT improves tempo stability, harmonic continuity, and recurrence organization. Code, configuration files, split metadata, evaluation scripts, and representative generated examples are available at: https://github.com/Music-aitech/AMC-main.git .
Keywords:
AI-assisted visual-content creation
Video-to-music generation
Multimodal generative AI
Reliability-aware fusion
Low-resource adaptation
Cross-modal alignment
Journal
T
IF:
0
Papers:
369
Citations:
0