.png){kind=icon}
返回
Foundation Model-Based Spectral-Spatial Transformer for Hyperspectral Image Classification
.png){kind=icon}
PRE
AI
DOI：10.1109/TGRS.2024.3456129.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Recently, deep learning models have dominated hyperspectral image (HSI) classification. Nowadays, deep learning is undergoing a paradigm shift with the rise of transformer-based foundation models. In this study, the potential of transformer-based foundation models, including the vision foundation model (VFM) and language foundation model (LFM), for HSI classification are investigated. First, to improve the performance of traditional HSI classification tasks, a spectral-spatial VFM-based transformer (SS-VFMT) is proposed, which inserts spectral-spatial information into the pretrained foundation transformer. Specifically, a given pretrained transformer receives HSI patch tokens for long-range feature extraction benefiting from the prelearned weights. Meanwhile, two enhancement modules, i.e., spatial and spectral enhancement modules (SpaEMs/ SpeEMs), utilize spectral and spatial information for steering the behavior of the transformer. Besides, an additional patch relationship distillation strategy is designed for SS-VFMT to exploit the pretrained knowledge better, leading to the proposed SS-VFMT-D. Second, based on SS-VFMT, to address a new HSI classification task, i.e., generalized zero-shot classification, a spectral-spatial vision-language-based transformer (SS-VLFMT) is proposed. This task is to recognize novel classes not seen during training, which is more meaningful as the real world is usually open. The SS-VLFMT leverages SS-VFMT to extract spectral-spatial features and corresponding hash codes while integrating a pretrained language model to extract text features from class names. Experimental results on HSI datasets reveal that the proposed methods are competitive compared to the state-of-the-art methods. Moreover, the foundation model-based methods open a new window for HSI classification tasks, especially for HSI zero-shot classification.
Keyword:
Deep learning
foundation model
hyperspectral image (HSI) classification
transformer
zero-shot learning (ZSL)
期刊
IF:
8.6
论文数:
2.2W
被引数:
10.7W
机构
引用论文
Investigation on the genotoxicity of different sizes of gold nanoparticles administered to the lungs of rats不同粒径纳米金对大鼠肺的遗传毒性研究
.png){kind=icon}
PREAI
分享收藏Consistency of Measurements of Wavelength Position From Hyperspectral Imagery: Use of the Ferric Iron Crystal Field Absorption at ∼900 nm as an Indicator of Mineralogy高光谱图像波长位置测量的一致性: 使用 ~ 900 nm的三价铁晶体场吸收作为矿物学指标
OAAI
OAAI分享收藏Deep Pyramidal Residual Networks for Spectral-Spatial Hyperspectral Image Classification用于光谱-空间高光谱图像分类的深度金字塔残差网络OAAI
OAAI分享收藏Spectral-Spatial MLP-Like Network With Reciprocal Points Learning for Open-Set Hyperspectral Image Classification具有互易点学习的光谱空间类MLP网络，用于开放集高光谱图像分类
PREAI
分享收藏Hyperspectral Image Classification Using Dictionary-Based Sparse Representation基于字典稀疏表示的高光谱图像分类
PREAI
分享收藏分享收藏
查看更多内容