Multi-semantic depth collaborative multi-modality image fusion network

Rui Chen

doi:10.1016/j.knosys.2026.115756

Multi-semantic depth collaborative multi-modality image fusion network

2026-03-10

0

PRE

AI

DOI：10.1016/j.knosys.2026.115756

Original

Original request for help

Save

Abstract

En 中文

Multi-modality image fusion aims to integrate complementary data from diverse sensors to achieve a more comprehensive representation of the imaging scene. Existing methods primarily focus on surface-level interactions between pixel-level features and visual or textual semantics, and do not fully integrate the global consistency between text modalities, image fusion, and downstream tasks. The lack of this deep collaboration between multiple semantics cannot effectively capture multi-modal information and the fine-grained association between fusion and downstream tasks. Therefore, we propose a Multi-Semantic depth collaborative Image Fusion (MSIF) network. Firstly, different from the traditional cascaded network structure, MSIF adopts a dual-task parallel structure that efficiently establishes deep correlations between the fusion and segmentation tasks. Second, MSIF enhances the semantic consistency between fusion and downstream tasks by leveraging deeper and detailed semantic information in text modalities to guide image fusion and segmentation. Additionally, MSIF uses the designed Multi-Semantic Deep Synergistic Module (MSDSM) to deeply integrate text features, pixel-level features and segmentation semantic features, so as to realize the effective fusion of multi-modal information and meet the semantic requirements of downstream tasks. Finally, the combined semantic features are fed into a fusion decoder to yield the final fusion result. Extensive quantitative and qualitative experiments demonstrate that our approach achieves state-of-the-art (SOTA) performance on multimodal datasets.

Multi-semantic depth collaborative multi-modality image fusion network

Abstract

Journal

Organization