.png){kind=icon}
返回
Rectify ViT Shortcut Learning by Visual Saliency
.png){kind=icon}
OA
AI
DOI：10.1109/TNNLS.2023.3310531.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Shortcut learning in deep learning models occurs when unintended features are prioritized, resulting in degenerated feature representations and reduced generalizability and interpretability. However, shortcut learning in the widely used vision transformer (ViT) framework is largely unknown. Meanwhile, introducing domain-specific knowledge is a major approach to rectifying the shortcuts that are predominated by background-related factors. For example, eye-gaze data from radiologists are effective human visual prior knowledge that has the great potential to guide the deep learning models to focus on meaningful foreground regions. However, obtaining eye-gaze data can still sometimes be time-consuming, labor-intensive, and even impractical. In this work, we propose a novel and effective saliency-guided ViT (SGT) model to rectify shortcut learning in ViT with the absence of eye-gaze data. Specifically, a computational visual saliency model (either pretrained or fine-tuned) is adopted to predict saliency maps for input image samples. Then, the saliency maps are used to filter the most informative image patches. Considering that this filter operation may lead to global information loss, we further introduce a residual connection that calculates the self-attention across all the image patches. The experiment results on natural and medical image datasets show that our SGT framework can effectively learn and leverage human prior knowledge without eye-gaze data and achieves much better performance than baselines. Meanwhile, it successfully rectifies the harmful shortcut learning and significantly improves the interpretability of the ViT model, demonstrating the promise of transferring human prior knowledge derived visual saliency in rectifying shortcut learning.
Keyword:
Interpretability
saliency
shortcut learning
vision transformer (ViT)
期刊
IF:
8.9
论文数:
7.6K
被引数:
7.2W
机构
暂无机构信息
引用论文
Mutation strategies for obtaining chitooligosaccharides with longer chains by transglycosylation reaction of family GH18 chitinase通过家族GH18几丁质酶的转糖基反应获得长链壳寡糖的突变策略
OAAI
OAAI分享收藏Types of Parent Verbal Responsiveness That Predict Language in Young Children With Autism Spectrum Disorder预测自闭症谱系障碍幼儿语言的父母言语反应类型OAAI
OAAI分享收藏Inflexibility of mental planning: A characteristic disorder with prefrontal lobe lesions?心理计划的僵化: 前额叶病变的特征性障碍？
PREAI
分享收藏INbreast: Toward a Full-field Digital Mammographic Database乳房: 迈向全视野数字乳腺摄影数据库.png)
ACADEMIC RADIOLOGYOAAI
ACADEMIC RADIOLOGYIF3.9
OAAI分享收藏Measuring the impact of outdated channel state information in interference alignment techniques
PREAI
分享收藏
查看更多内容