.png){kind=icon}
返回
Factorization Vision Transformer: Modeling Long-Range Dependency With Local Window Cost
.png){kind=icon}
OA
AI
DOI：10.1109/TNNLS.2023.3342172.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Transformers have astounding representational power but typically consume considerable computation which is quadratic with image resolution. The prevailing Swin transformer reduces computational costs through a local window strategy. However, this strategy inevitably causes two drawbacks: 1) the local window-based self-attention (WSA) hinders global dependency modeling capability and 2) recent studies point out that local windows impair robustness. To overcome these challenges, we pursue a preferable trade-off between computational cost and performance. Accordingly, we propose a novel factorization self-attention (FaSA) mechanism that enjoys both the advantages of local window cost and long-range dependency modeling capability. By factorizing the conventional attention matrix into sparse subattention matrices, FaSA captures long-range dependencies, while aggregating mixed-grained information at a computational cost equivalent to the local WSA. Leveraging FaSA, we present the factorization vision transformer (FaViT) with a hierarchical structure. FaViT achieves high performance and robustness, with linear computational complexity concerning input image spatial resolution. Extensive experiments have shown FaViT's advanced performance in classification and downstream tasks. Furthermore, it also exhibits strong model robustness to corrupted and biased data and hence demonstrates benefits in favor of practical applications. In comparison to the baseline model Swin-T, our FaViT-B2 significantly improves classification accuracy by 1% and robustness by 7%, while reducing model parameters by 14%. Our code will soon be publicly available: at https://github.com/q2479036243/FaViT.
Keyword:
Transformers
Robustness
Computational modeling
Costs
Computational efficiency
Windows
Sparse matrices
Factorization
local window
long-range dependency
model robustness
transformer
期刊
IF:
8.9
论文数:
7.6K
被引数:
7.2W
机构
引用论文
Molecular and Pharmacological Characterization of GABAA Receptors in the Rat Pituitary大鼠垂体中GABAA 受体的分子和药理学特征
PREAI
分享收藏Inflexibility of mental planning: A characteristic disorder with prefrontal lobe lesions?心理计划的僵化: 前额叶病变的特征性障碍？
PREAI
分享收藏Candidate .gamma.-Secretases in the Generation of the Carboxyl Terminus of the Alzheimer's Disease .beta.A4 Amyloid: Possible Involvement of Cathepsin D候选人。Γ-分泌酶在阿尔茨海默氏病羧基末端的产生。Β.A4淀粉样蛋白: 组织蛋白酶D可能参与.png)
Biochemistry
BiochemistryIF0
PREAI
分享收藏
查看更多内容