A Deep Multi-task Contextual Attention Framework for Multi-modal Affect Analysis

Md Shad Akhtar

doi:10.1145/3380744

Return

A Deep Multi-task Contextual Attention Framework for Multi-modal Affect Analysis

2020-05-13

PRE

Md Shad Akhtar *

Dushyant Singh Chauhan

Asif Ekbal

DOI：10.1145/3380744

Original

Original request for help

Save

Abstract

En 中文

Multi-modal affect analysis (e.g., sentiment and emotion analysis) is an interdisciplinary study and has been an emerging and prominent field in Natural Language Processing and Computer Vision. The effective fusion of multiple modalities (e.g., text, acoustic, or visual frames) is a non-trivial task, as these modalities, often, carry distinct and diverse information, and do not contribute equally. The issue further escalates when these data contain noise. In this article, we study the concept of multi-task learning for multi-modal affect analysis and explore a contextual inter-modal attention framework that aims to leverage the association among the neighboring utterances and their multi-modal information. In general, sentiments and emotions have inter-dependence on each other (e.g., anger. negative or happy. positive). In our current work, we exploit the relatedness among the participating tasks in the multi-task framework. We define three different multi-task setups, each having two tasks, i.e., sentiment & emotion classification, sentiment classification & sentiment intensity prediction, and emotion classification & emotion intensity prediction. Our evaluation of the proposed system on the CMU-Multi-modal Opinion Sentiment and Emotion Intensity benchmark dataset suggests that, in comparison with the single-task learning framework, our multi-task framework yields better performance for the inter-related participating tasks. Further, comparative studies show that our proposed approach attains state-of-the-art performance for most of the cases.

Keywords:

Multi-task learning

multi-modal analysis

sentiment analysis

sentiment intensity prediction

emotion analysis

emotion intensity prediction

inter-modal attention

AI Summary

Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.