.png){kind=icon}
.png){kind=icon}
Return
Convex-Concave Programming: An Effective Alternative for Optimizing Shallow Neural Networks
.png){kind=icon}
PRE
AI
DOI：10.1109/TETCI.2024.3502463.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
In this study, we address the challenges of non-convex optimization in neural networks (NNs) by formulating the training of multilayer perceptron (MLP) NNs as a difference of convex functions (DC) problem. Utilizing the basic convex-concave algorithm to solve our DC problems, we introduce two alternative optimization techniques, DC-GD and DC-OPT , for determining MLP parameters. By leveraging the non-uniqueness property of the convex components in DC functions, we generate strongly convex components for the DC NN cost function. This strong convexity enables our proposed algorithms, DC-GD and DC-OPT , to achieve an iteration complexity of O(log(1/epsilon)) , surpassing that of other solvers, such as stochastic gradient descent ( SGD ), which has an iteration complexity of O(1/epsilon) . This improvement raises the convergence rate from sublinear ( SGD ) to linear (ours) while maintaining comparable total computational costs . Furthermore, conventional NN optimizers like SGD , RMSprop , and Adam are highly sensitive to the learning rate, adding computational overhead for practitioners in selecting an appropriate learning rate. In contrast, our DC-OPT algorithm is hyperparameter-free (i.e., it requires no learning rate), and our DC-GD algorithm is less sensitive to the learning rate, offering comparable accuracy to other solvers. Additionally, we extend our approach to a convolutional NN architecture, demonstrating its applicability to modern NNs. We evaluate the performance of our proposed algorithms by comparing them to conventional optimizers such as SGD , RMSprop , and Adam across various test cases. The results suggest that our approach is a viable alternative for optimizing shallow MLP NNs.
Keywords:
Neural network
difference convex function
convex-concave algorithm
convergence rate
complexity analysis
Journal
I
IF:
6.5
Papers:
1.4K
Citations:
4.5K