arrow
Return

Alliance: All-in-One Spectral-Spatial-Frequency Awareness Foundation Model

delete2025-12-03
delete0
PRE
AI
B
Boyu Zhao
李
李伟 (Wei Li)
王
王俊杰 (Junjie Wang)
Y
Yuxiang Zhang
H
Hong Yang
H
Haitao Zhao
R
Ran Tao
Q
Qian Du
DOI:10.1109/TPAMI.2025.3639595delete
deleteOriginal
deleteShare
deleteSave
Abstract

Abstract

En 中文
Frequency domain analysis reveals fundamental image patterns difficult to observe in raw pixel values, while avoiding redundant information in original image processing. Although recent remote sensing foundation models (FMs) have made progress in leveraging spatial and spectral information, they have limitations in fully utilizing frequency characteristics that capture hidden features. Existing FMs that incorporate frequency properties often struggle to maintain connections with the original image content, creating a semantic gap that affects downstream performance. To address these challenges, we propose the All-in-One Spectral-Spatial-Frequency Awareness Foundation Model (Alliance), a framework that effectively integrates information across all three domains. Alliance introduces several key innovations: (1) a progressive frequency decoding mechanism inspired by human visual cognition that minimizes multi-domain information gaps while preserving connections between general image information and frequency characteristics, progressively reconstructing from low to mid to high frequencies to extract patterns difficult to observe in raw pixel values; (2) a triple-domain fusion attention module that separately processes amplitude, phase, and spectral-spatial relationships for comprehensive feature integration; and (3) frequency embedding with frequency-aware Cls token initialization and frequency-specific mask token initialization that achieves fine-grained modeling of different frequency band information. Additionally, to evaluate FMs generalizability, we construct the Yellow River dataset, a large-scale multi-temporal collection that introduces challenging cross-domain tasks and establishes more rigorous standards for FMs assessment. Extensive experiments across six downstream tasks demonstrate Alliance’s superior performance.
Keywords:
Frequency domain
foundation model
human visual cognition
spatial domain
spectral domain
remote sensing

Journal

IEEE Transactions on Pattern Analysis and Machine Intelligence cover
IEEE Transactions on Pattern Analysis and Machine Intelligence
IF:
18.6
Papers:
2.1K
Citations:
9.8W

Organization

B
beijing institute of technology
Scholars:
5.5W
Papers: 4.0W
Citations: 63
C
Chinese Academy of Sciences
Scholars:
3.9W
Papers: 1.5W
Citations: 58.4W
M
mississippi state university
Scholars:
7.4K
Papers: 6.9K
Citations: 70
researcher View more organizations
Cited Papers

Cited Papers

Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Data
err2021-10-01
err0
errOAAI
errOscar Manas; Alexandre Lacoste; Xavier Giro-i-Nieto; David Vazquez; Pau Rodriguez
errShare
errSave
Prototypical Contrast Adaptation for Domain Adaptive Semantic Segmentation
err2022-10-22
err0
errOAAI
errZhengkai Jiang; Yuxi Li; Ceyuan Yang; Peng Gao; Yabiao Wang; Ying Tai; Chengjie Wang
errShare
errSave
SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery
err2024-06-16
err0
PREAI
errXin Guo; Jiangwei Lao; Bo Dang; Yingying Zhang; Lei Yu; Lixiang Ru; Liheng Zhong; Ziyuan Huang; Kang Wu; Dingxiang Hu; Huimei He; Jian Wang; Jingdong Chen; Ming Yang; Yongjun Zhang; Yansheng Li
errShare
errSave
errShare
errSave
SimPoolFormer: A two-stream vision transformer for hyperspectral image classification
err2025-01-01
err0
PREAI
errRoy, Swalpa Kumar; Jamali, Ali; Chanussot, Jocelyn; Ghamisi, Pedram; Ghaderpour, Ebrahim; Shahabi, Himan
errShare
errSave
Rethinking Transformers Pre-training for Multi-Spectral Satellite Imagery
err2024-06-16
err0
PREAI
errMubashir Noman; Muzammal Naseer; Hisham Cholakkal; Rao Muhammad Anwar; Salman Khan; Fahad Shahbaz Khan
errShare
errSave
researcher View more