.png){kind=icon}
返回
Memory-Efficient Batch Normalization by One-Pass Computation for On-Device Training
.png){kind=icon}
PRE
AI
DOI：10.1109/TCSII.2024.3354738.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Batch normalization (BN) has become ubiquitous in modern deep learning architectures because of its remarkable improvement in deep neural network (DNN) training performance. However, the two-pass computation of statistical estimation and element-wise normalization in BN training requires two accesses to the input data, resulting in a huge increase in off-chip memory traffic during DNN training. In this brief, we propose a novel accelerator, named one-pass normalizer (OPN) to achieve memory-efficient BN for on-device training. Specifically, in terms of dataflow, we propose one-pass computation based on sampling-based range normalization and sparse data recovery techniques to reduce BN off-chip memory access. Regarding the OPN circuit, we propose channel-wise constant extraction to achieve a compact design. Experimental results show that the one-pass computation reduces off-chip memory access of BN by 2.0 similar to 3.8x compared with the previous state-of-the-art designs while maintaining training performance. Moreover, the channel-wise constant extraction saves the gate count and power consumption of OPN by 56% and 73%, respectively.
Keyword:
Training
Systolic arrays
Backpropagation
Artificial neural networks
Micromechanical devices
Feedforward systems
Memory management
Memory-efficient accelerator
deep neural networks
batch normalization
on-device training
one-pass computation
期刊
I
IF:
4.9
论文数:
8.8K
被引数:
2.5W
机构
引用论文
Estimation method of earthwork excavation using shield tunneling data -- a case study of Chengdu Metro盾构隧道数据在土方开挖估算方法中的应用——以成都地铁为例
OAAI
OAAI分享收藏A Low-Power Deep Neural Network Online Learning Processor for Real-Time Object Tracking Application
PREAI
分享收藏Damage Detection and Level Classification of Roof Damage After Typhoon Faxai Based on Aerial Photo and Deep Learning基于航拍照片和深度学习的Faxai台风后顶板损伤检测与等级分类.png)
OAAI
IF0
OAAI分享收藏The power of obfuscation techniques in malicious JavaScript code: A measurement study恶意JavaScript代码中混淆技术的功能: 一项测量研究
OAAI
OAAI分享收藏分享收藏
查看更多内容