.png){kind=icon}
.png){kind=icon}
Return
Multimodal summarization with modality features alignment and features filtering
.png){kind=icon}
PRE
AI
DOI：10.1016/j.neucom.2024.128270.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Previous studies about M ulti M odal S ummarization (MMS) mainly focus on effective selection and filtering of visual features to assist in cross-modal fusion and text-based generation. However, there exists a natural disparity between the distributions of features from different modalities which limits a more comprehensive cross-modal fusion for MMS models. In this paper, we propose to utilize M aximum M ean D iscrepancy (MMD) to align the features from two modalities, design a filter to further denoise the visual features, and conduct cross-modal fusion based on generative pre-trained language models for better cross-modal fusion and text generation. Moreover, we notice the presence of some special tokens in the MMS dataset which are introduced in prior data preprocessing. This phenomenon could limit the performance of contemporary generative models. Thus we adopt the powerful L arge L anguage M odel (LLM) to preprocess the dataset to enhance MMS models. Experimental results on the original MMS dataset demonstrate that our proposed method is effective and outperforms previous strong baselines. Experimental results on the preprocessed MMS dataset also demonstrate the feasibility of incorporating LLM in the data preprocessing to enhance MMS models.
Keywords:
Multimodal summarization
Maximum mean discrepancy
Cross-modal fusion
Journal
IF:
6.5
Papers:
2.5W
Citations:
6.5W