.png){kind=icon}
返回
ALBERTA: ALgorithm-Based Error Resilience in Transformer Architectures
.png){kind=icon}
OA
AI
DOI：10.1109/OJCS.2024.3400696.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Vision Transformers are being increasingly deployed in safety-critical applications that demand high reliability. Ensuring the correct execution of these models in GPUs is critical, despite the potential for transient hardware errors. We propose a novel algorithm-based resilience framework called ALBERTA that allows us to perform end-to-end resilience analysis and protection of transformer-based architectures. First, our work develops an efficient process of computing and ranking the resilience of transformers layers. Due to the large size of transformer models, applying traditional network redundancy to a subset of the most vulnerable layers provides high error coverage albeit with impractically high overhead. We address this shortcoming by providing a software-directed, checksum-based error detection technique aimed at protecting the most vulnerable general matrix multiply (GEMM) layers in the transformer models that use either floating-point or integer arithmetic. Results show that our approach achieves over 99% coverage for errors (single bit-flip fault model) that result in a mismatch with < 0.2% and < 0.01% computation and memory overheads, respectively. Lastly, we present the applicability of our framework in various modern GPU architectures under different numerical precisions. We introduce an efficient self-correction mechanism for resolving erroneous detection with an average of less than 2% overhead per error.
Keyword:
Transformers
Resilience
Computational modeling
Hardware
Computer architecture
Transient analysis
Measurement
Correction
detection
protection
resilience
robustness
safety
transformers
期刊
I
IF:
8.2
论文数:
411
被引数:
810
机构
引用论文
.png){kind=icon}
Inflexibility of mental planning: A characteristic disorder with prefrontal lobe lesions?心理计划的僵化: 前额叶病变的特征性障碍？
PREAI
分享收藏Normal Ranges of Renal Physiological Parameters for Technetium-99m Mercaptoacetyltriglycine and the Influence of Age and Sex Using a Camera-Based Method锝-99m巯基乙酰基三甘氨酸肾生理参数的正常范围及年龄和性别的影响：基于相机的方法
PREAI
分享收藏Do Perceptions of Competence Mediate The Relationship Between Fundamental Motor Skill Proficiency and Physical Activity Levels of Children in Kindergarten?能力的感知是否可以介导幼儿园儿童的基本运动技能熟练程度与身体活动水平之间的关系？
PREAI
分享收藏Community-Level Factors Associated with COVID-19 Cases and Testing Equity in King County, WashingtonOAAI
OAAI分享收藏
查看更多内容