.png){kind=icon}
.png){kind=icon}
Return
Hyperspectral and multispectral image fusion via N-gram transformer and local adaptive fusion strategy
.png){kind=icon}
PRE
AI
Y
DOI：10.1016/j.image.2026.117552.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
The fusion of low-resolution hyperspectral images (LR-HSI) with high-resolution images (HR-MSI) can significantly enhance the spatial resolution of hyperspectral images (HSI), thereby improving its applicability in HSI tasks. Current convolutional neural network (CNN)-based HR-MSI and LR-HSI fusion networks primarily focus on local neighbourhood relations, often overlooking global feature mappings. Some approaches integrate transformers to explore global intrinsic feature relationships, albeit at the cost of requiring extensive data. Moreover, existing fusion methods predominantly concentrate on learning modality-shared features. To address these limitations, this study introduces a local adaptive HR-MSI and LR-HSI feature pyramid fusion network based on N-Gram transformer architecture. Inspired by N-Gram models in natural language processing, N-Gram models are incorporated with sliding window self-attention (WSA) within transformers to enhance the interaction of feature domain information. Group convolutions are employed to reduce channels and enhance computational efficiency. Additionally, a novel local adaptive fusion strategy is proposed to effectively integrate spatial and spectral information from both modalities while mitigating cross-modal disparities. This strategy carefully balances local specificity and global context through a designed global bias mechanism. Furthermore, the adoption of a feature pyramid structure as the core of the HSI-MSI fusion model facilitates multi-scale feature extraction and fusion. Extensive experimental validation demonstrates that our proposed framework significantly outperforms existing methods in enhancing the quality of fused images and in classification tasks.
Keywords:
N-Gram
Transformer
Hyperspectral and multispectral fusion
Adaptive fusion
Journal
S
IF:
2.7
Papers:
18
Citations:
0
Organization
Cited Papers