.png){kind=icon}
返回
Diminishing Batch Normalization
.png){kind=icon}
OA
AI
DOI：10.1109/TNNLS.2022.3210840.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
In this article, we propose a generalization of the batch normalization (BN) algorithm, diminishing BN (DBN), where we update the BN parameters in a diminishing moving average way. BN is very effective in accelerating the convergence of a neural network training phase that it has become a common practice. Our proposed DBN algorithm retains the overall structure of the original BN algorithm while introducing a weighted averaging update to some trainable parameters. We provide an analysis of the convergence of the DBN algorithm that converges to a stationary point with respect to the trainable parameters. Our analysis can be easily generalized to the original BN algorithm by setting some parameters to constant. To the best of our knowledge, this analysis is the first of its kind for convergence with BN. We analyze a two-layer model with arbitrary activation functions. Common activation functions, such as ReLU and any smooth activation functions, meet our assumptions. In the numerical experiments, we test the proposed algorithm on complex modern CNN models with stochastic gradients (SGs) and ReLU activation on regression, classification, and image reconstruction tasks. We observe that DBN outperforms the original BN algorithm and benchmark layer normalization (LN) on the MNIST, NI, CIFAR-10, CIFAR-100, and Caltech-UCSD Birds-200-2011 datasets with modern complex CNN models such as Resnet-18 and typical FNN models.
Keyword:
Convergence
Training
Analytical models
Standards
Optimization
Linear programming
Neurons
Batch normalization (BN)
convergence analysis
learning theory
neural network
optimization
期刊
IF:
8.9
论文数:
7.6K
被引数:
7.2W
机构
引用论文
分享收藏分享收藏Gradient-based learning applied to document recognition基于梯度的学习在文档识别中的应用.png)
PROCEEDINGS OF THE IEEE
PROCEEDINGS OF THE IEEEIF25.9
PREAI
分享收藏The sub‐zero temperature chromatographic isolation of transient intermediates of a multi‐step cycle: Purification of the substrate‐bound oxy‐ferrous cytochrome P450FEBS Letters
OAAI
FEBS LettersIF0
OAAI分享收藏分享收藏
查看更多内容