.png){kind=icon}
返回
Incremental PID Controller-Based Learning Rate Scheduler for Stochastic Gradient Descent
.png){kind=icon}
PRE
AI
DOI：10.1109/TNNLS.2022.3213677.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
As we all know, the learning rate plays a vital role in deep neural network (DNN) training. This study introduces an incremental proportional-integral-derivative (PID) controller widely used in automatic control as a learning rate scheduler for stochastic gradient descent (SGD). To automatically calculate the current learning rate, we utilize feedback control to determine the relationship between training losses and learning rates, named incremental PID learning rates, which include PID-Base and PID-Warmup. The new schedulers reduce the dependence on the initial learning rate and achieve higher accuracy. Compared with multistep learning rates (MSLR), cyclical learning rates (CLR), and SGD with warm restarts (SGDR), incremental PID learning rates based on feedback control obtain higher accuracy on CIFAR-10, CIFAR-100, and Tiny-ImageNet-200. We believe that our methods can improve the performance of SGD.
Keyword:
Training
PD control
PI control
Feedback control
Convergence
Oscillators
Optimization
incremental proportional-integral-derivative (PID) controller
learning rate scheduler
stochastic gradient descent (SGD)
期刊
IF:
8.9
论文数:
7.6K
被引数:
7.2W
机构
引用论文
.png){kind=icon}
Synthesis of n-type semiconducting diamond film using diphosphorus pentaoxide as the doping source以五氧化二磷为掺杂源合成n型半导体金刚石膜
PREAI
分享收藏
Research on the Efficiency of Wireless Power Transfer System Based on Multi-Auxiliary Transmitting Coils基于多辅助发射线圈的无线电能传输系统效率研究
PREAI
分享收藏PID Controller-Based Stochastic Optimization Acceleration for Deep Neural Networks基于PID控制器的深度神经网络随机优化加速
PREAI
分享收藏The power of obfuscation techniques in malicious JavaScript code: A measurement study恶意JavaScript代码中混淆技术的功能: 一项测量研究
OAAI
OAAI分享收藏
查看更多内容