Fractional Fourier Image Transformer for Multimodal Remote Sensing Data Classification

doi:10.1109/TNNLS.2022.3189994

Fractional Fourier Image Transformer for Multimodal Remote Sensing Data Classification

2024-02-01

49

PRE

AI

DOI：10.1109/TNNLS.2022.3189994

Original

Original request for help

Save

Abstract

En 中文

With the recent development of the joint classification of hyperspectral image (HSI) and light detection and ranging (LiDAR) data, deep learning methods have achieved promising performance owing to their locally sematic feature extracting ability. Nonetheless, the limited receptive field restricted the convolutional neural networks (CNNs) to represent global contextual and sequential attributes, while visual image transformers (VITs) lose local semantic information. Focusing on these issues, we propose a fractional Fourier image transformer (FrIT) as a backbone network to extract both global and local contexts effectively. In the proposed FrIT framework, HSI and LiDAR data are first fused at the pixel level, and both multisource feature and HSI feature extractors are utilized to capture local contexts. Then, a plug-and-play image transformer FrIT is explored for global contextual and sequential feature extraction. Unlike the attention-based representations in classic VIT, FrIT is capable of speeding up the transformer architectures massively and learning valuable contextual information effectively and efficiently. More significantly, to reduce redundancy and loss of information from shallow to deep layers, FrIT is devised to connect contextual features in multiple fractional domains. Five HSI and LiDAR scenes including one newly labeled benchmark are utilized for extensive experiments, showing improvement over both CNNs and VITs.

Keywords:

Feature extraction

Transformers

Laser radar

Data mining

Discrete Fourier transforms

Visualization

Semantics

Fractional Fourier image transformer (FrIT)

hyperspectral image (HSI)

light detection and ranging (LiDAR)

multimodal data

Fractional Fourier Image Transformer for Multimodal Remote Sensing Data Classification

Abstract

Journal

Organization