.png){kind=icon}
Return
ALBERTA: ALgorithm-Based Error Resilience in Transformer Architectures
.png){kind=icon}
OA
AI
DOI：10.1109/OJCS.2024.3400696.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Vision Transformers are being increasingly deployed in safety-critical applications that demand high reliability. Ensuring the correct execution of these models in GPUs is critical, despite the potential for transient hardware errors. We propose a novel algorithm-based resilience framework called ALBERTA that allows us to perform end-to-end resilience analysis and protection of transformer-based architectures. First, our work develops an efficient process of computing and ranking the resilience of transformers layers. Due to the large size of transformer models, applying traditional network redundancy to a subset of the most vulnerable layers provides high error coverage albeit with impractically high overhead. We address this shortcoming by providing a software-directed, checksum-based error detection technique aimed at protecting the most vulnerable general matrix multiply (GEMM) layers in the transformer models that use either floating-point or integer arithmetic. Results show that our approach achieves over 99% coverage for errors (single bit-flip fault model) that result in a mismatch with < 0.2% and < 0.01% computation and memory overheads, respectively. Lastly, we present the applicability of our framework in various modern GPU architectures under different numerical precisions. We introduce an efficient self-correction mechanism for resolving erroneous detection with an average of less than 2% overhead per error.
Keywords:
Transformers
Resilience
Computational modeling
Hardware
Computer architecture
Transient analysis
Measurement
Correction
detection
protection
resilience
robustness
safety
transformers
Journal
I
IF:
8.2
Papers:
411
Citations:
810
Organization
Cited Papers
.png){kind=icon}
ShareSaveShareSaveShareSaveShareSaveCommunity-Level Factors Associated with COVID-19 Cases and Testing Equity in King County, WashingtonOAAI
OAAIShareSave
View more