.png){kind=icon}
Return
Learning Rate Dropout
.png){kind=icon}
PRE
AI
DOI：10.1109/TNNLS.2022.3155181.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Optimization algorithms are of great importance to efficiently and effectively train a deep neural network. However, the existing optimization algorithms show unsatisfactory convergence behavior, either slowly converging or not seeking to avoid bad local optima. Learning rate dropout (LRD) is a new gradient descent technique to motivate faster convergence and better generalization. LRD aids the optimizer to actively explore in the parameter space by randomly dropping some learning rates (to 0); at each iteration, only parameters whose learning rate is not 0 are updated. Since LRD reduces the number of parameters to be updated for each iteration, the convergence becomes easier. For parameters that are not updated, their gradients are accumulated (e.g., momentum) by the optimizer for the next update. Accumulating multiple gradients at fixed parameter positions gives the optimizer more energy to escape from the saddle point and bad local optima. Experiments show that LRD is surprisingly effective in accelerating training while preventing overfitting.
Keywords:
Training
Convergence
Optimization
Neural networks
Perturbation methods
Informatics
Adaptation models
Neural network
optimization algorithm
regularization
Journal
IF:
8.9
Papers:
7.5K
Citations:
7.2W
Organization
Cited Papers
.png){kind=icon}
ShareSaveSynthesis of n-type semiconducting diamond film using diphosphorus pentaoxide as the doping source
PREAI
ShareSave
ShareSaveShareSave
View more