.png){kind=icon}
.png){kind=icon}
Return
TPQA: Efficient attention architecture with task-aware pattern-guided quantization
.png){kind=icon}
PRE
AI
DOI：10.1016/j.future.2025.108352.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Attention mechanisms have become a cornerstone of modern deep learning models, yet their computational intensity poses significant deployment challenges for resource-limited devices. While quantization offers a potential solution, current approaches typically employ uniform precision assignment schemes across all attention heads, neglecting critical variations in head-specific contributions across different tasks. This oversight results in substantial computational redundancy for those attention heads with fewer contributions, impacting overall performance. Through systematic analysis of head pattern characteristics in transformer models, we reveal two key insights: different attention heads exhibit distinct task-aware patterns, and their varying contributions to model performance directly dictate differentiated quantization demands across heads. Building on these findings, we propose TPQA, a novel algorithm and accelerator co-design architecture for efficient deployment of transformer models. TPQA strategically assigns adaptive precision levels to each head based on pre-identified patterns, thereby reducing computational overhead while preserving model accuracy. Furthermore, TPQA employs a data reordering strategy to transform irregular workloads into structured formats and introduces a dedicated accelerator with an attention-weights-stationary dataflow to efficiently process these structured workloads. Comprehensive evaluations demonstrate TPQA's superior performance, achieving up to 2.1x speedup and 3.4x energy efficiency improvement over state-of-the-art accelerators while maintaining <1% accuracy loss on various tasks.
Keywords:
Multi-head attention
Quantization
Accelerator
Head pattern
Systolic array
Attention architecture
Journal
F
IF:
6.1
Papers:
6.8K
Citations:
2.3W