.png){kind=icon}
返回
ACTUAL: Audio Captioning With Caption Feature Space Regularization
.png){kind=icon}
PRE
AI
DOI：10.1109/TASLP.2023.3293015.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Audio captioning aims at describing the content of audio clips with human language. Due to the ambiguity of audio content, different people may perceive the same audio clip differently, resulting in caption disparities (i.e., the same audio clip may be described by several captions with diverse semantics). In the literature, the one-to-many strategy is often employed to train the audio captioning models, where a related caption is randomly selected as the optimization target for each audio clip at each training iteration. However, we observe that this can lead to significant variations during the optimization process and adversely affect the performance of the model. In this article, we address this issue by proposing an audio captioning method, named ACTUAL (Audio Captioning with capTion featUre spAce reguLarization). ACTUAL involves a two-stage training process: (i) in the first stage, we use contrastive learning to construct a proxy feature space where the similarities between captions at the audio level are explored, and (ii) in the second stage, the proxy feature space is utilized as additional supervision to improve the optimization of the model in a more stable direction. We conduct extensive experiments to demonstrate the effectiveness of the proposed ACTUAL method. The results show that proxy caption embedding can significantly improve the performance of the baseline model and the proposed ACTUAL method offers competitive performance on two datasets compared to state-of-the-art methods.
Keyword:
Audio captioning
contrastive learning
cross-modal task
caption consistency regularization
期刊
I
IF:
5.1
论文数:
2.6K
被引数:
1.1W
机构
引用论文
Sodium channel kinetic changes that produce Brugada syndrome or progressive cardiac conduction system disease产生Brugada综合征或进行性心脏传导系统疾病的钠通道动力学变化
.png){kind=icon}
PREAI
分享收藏The shadow economy in three Mediterranean countries: France, Spain and Greece. A MIMIC approach
PREAI
分享收藏分享收藏分享收藏Genetic diversity and relationship of pummelo germplasms in Hunan province based on morphological traits and SRAP molecular markers基于形态性状和SRAP分子标记的湖南省柚类种质资源遗传多样性与亲缘关系
PREAI
分享收藏
分享收藏
查看更多内容