.png){kind=icon}
返回
Whose emotion matters? Speaking activity localisation without prior knowledge
.png){kind=icon}
OA
AI
DOI：10.1016/j.neucom.2023.126271.png)
.png){kind=icon}
.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
The task of emotion recognition in conversations (ERC) benefits from the availability of multiple modal-ities, as provided, for example, in the video-based Multimodal EmotionLines Dataset (MELD). However, only a few research approaches use both acoustic and visual information from the MELD videos. There are two reasons for this: First, label-to-video alignments in MELD are noisy, making those videos an unre-liable source of emotional speech data. Second, conversations can involve several people in the same scene, which requires the localisation of the utterance source. In this paper, we introduce MELD with Fixed Audiovisual Information via Realignment (MELD-FAIR) by using recent active speaker detection and automatic speech recognition models, we are able to realign the videos of MELD and capture the facial expressions from speakers in 96.92% of the utterances provided in MELD. Experiments with a self-supervised voice recognition model indicate that the realigned MELD-FAIR videos more closely match the transcribed utterances given in the MELD dataset. Finally, we devise a model for emotion recognition in conversations trained on the realigned MELD-FAIR videos, which outperforms state-of -the-art models for ERC based on vision alone. This indicates that localising the source of speaking activ-ities is indeed effective for extracting facial expressions from the uttering speakers and that faces provide more informative visual cues than the visual features state-of-the-art models have been using so far. The MELD-FAIR realignment data, and the code of the realignment procedure and of the emotional recogni-tion, are available at https://github.com/knowledgetechnologyuhh/MELD-FAIR.(c) 2023 The Author(s). Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
Keyword:
Multimodality
Active speaker detection
Emotion recognition
Forced alignment
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
6.5
论文数:
2.5W
被引数:
6.5W
机构
引用论文
分享收藏Synthesis, vibrational and luminescence studies on Eu3+:KY(WO4)2 red phosphorsEu3+:KY(WO4)2红色荧光粉的合成、振动及发光研究
PREAI
分享收藏EmbraceNet: A robust deep learning architecture for multimodal classificationEmbraceNet: 一种强大的多模态分类深度学习架构.png)
INFORMATION FUSIONOAAI
INFORMATION FUSIONIF15.5
OAAI分享收藏Cholinesterase as inflammatory markers in a experimental infection by Trypanosoma evansi in rabbitsOAAI
OAAI分享收藏分享收藏
查看更多内容