Multi-task learning framework using tri-encoder with caption prompt for multimodal aspect-based sentiment analysis

doi:10.1007/s11227-025-07252-4

Return

Multi-task learning framework using tri-encoder with caption prompt for multimodal aspect-based sentiment analysis

2025-04-29

PRE

张青川 (Qingchuan Zhang) *

Haitao Xiong

DOI：10.1007/s11227-025-07252-4

Original

Original request for help

Save

Abstract

En 中文

Multimodal aspect-based sentiment analysis (MABSA) is an advanced technology to identify all aspect terms together with their respective sentiment mentioned in the multimodal data. Most existing MABSA methods encounter two main challenges: (1) The representations used in MABSA are directly encoded by the general pre-trained models, which are insensitive to identifying aspect-level sentiment. (2) Latent visual semantic information is underutilized when representing the key aspects and their sentimental polarities. To address the mentioned challenges, we propose an optimized multi-task learning framework using a tri-encoder with caption prompt (TECP), including MABSA and two auxiliary unimodal tasks to jointly learn the aspect-aware and sentiment-aware multimodal representation. In TECP, the merged-attention fusion network is designed to obtain multimodal features, which enhances the intra-modal and cross-modal semantic interactions. Within the tri-encoder, the caption encoder is designed to further generate visual caption feature as important clue to enrich the multimodal semantic and sentimental information. Moreover, within the caption encoder, the dependency weight attention network is proposed to focus on the aspect-level feature in the caption sentence. We conduct elaborate experiments and evaluate the performance of TECP with respect to Precision, Recall, and F1-score. Our TECP achieves SOTA results on two benchmark Twitter datasets in comparison with previous baseline models.

Keywords:

Multimodal aspect-based sentiment analysis

Multi-task learning

Image caption prompt

Dependency weight attention

Merged-attention fusion network