.png){kind=icon}
返回
DPLRS: Distributed Population Learning Rate Schedule
.png){kind=icon}
PRE
AI
DOI：10.1016/j.future.2022.02.001.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Deep neural network models perform very brightly in the field of artificial intelligence, but their success is affected by hyperparameters, and the learning rate schedule is one of the most important hyperparameters, while the search for the learning rate schedule is often time-consuming and compu-tationally resource-intensive. In this paper, we proposed Distributed Population Learning Rate Schedule (DPLRS) based on population joint optimization, which uses distributed data parallel deep neural network training to implement a dynamic learning rate schedule optimization strategy based on the population idea, with almost no loss of test accuracy. DPLRS is able to dynamically refine the learning rate schedule during model training instead of following the usual suboptimal strategy. We conducted experiments on typical AlexNet, VGG16, and ResNet18 using the Tianhe-3 supercomputing prototype. The results illustrate that using DPLRS to dynamically update the learning rate can greatly reduce the searching time of the learning rate schedule and meanwhile, can ensure the close performance with the latest population hyperparameter algorithm. Also, In our experiments, DPLRS lead to 123.85x speedup maximum, which prove the effectiveness and robustness of DPLRS. (c) 2022 Elsevier B.V. All rights reserved.
Keyword:
Deep learning
Distributed training
Hyperparameter search
Data parallel
Population algorithm
期刊
F
IF:
6.1
论文数:
6.8K
被引数:
2.3W
机构
引用论文
An improved grid search algorithm to optimize SVR for prediction一种改进的网格搜索算法优化SVR预测.png)
SOFT COMPUTING
SOFT COMPUTINGIF2.5
PREAI
分享收藏
分享收藏分享收藏
查看更多内容