.png){kind=icon}
返回
AdaSwarm: Augmenting Gradient-Based Optimizers in Deep Learning With Swarm Intelligence
.png){kind=icon}
OA
AI
DOI：10.1109/TETCI.2021.3083428.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
This paper introduces AdaSwarm, a novel gradient-free optimizer which has similar or even better performance than the Adam optimizer adopted in neural networks. In order to support our proposed AdaSwarm, a novel Exponentially weighted Momentum Particle Swarm Optimizer (EMPSO), is proposed. The ability of AdaSwarm to tackle optimization problems is attributed to its capability to perform good gradient approximations. We show that, the gradient of any function, differentiable or not, can be approximated by using the parameters of EMPSO. This is a novel technique to simulate GD which lies at the boundary between numerical methods and swarm intelligence. Mathematical proofs of the gradient approximation produced are also provided. AdaSwarm competes closely with several state-of-the-art (SOTA) optimizers. We also show that AdaSwarm is able to handle a variety of loss functions during backpropagation, including the maximum absolute error (MAE).
Keyword:
Optimization
Particle swarm optimization
Artificial neural networks
Training
Search problems
Mathematical model
Deep learning
EMPSO
AdaSwarm
gradient equivalence
neural loss function
backpropagation
期刊
I
IF:
6.5
论文数:
1.4K
被引数:
4.5K
机构
引用论文
.png){kind=icon}
查看更多内容