.png){kind=icon}
返回
Adaptive Morphing Activation Function for Neural Networks
.png){kind=icon}
OA
AI
DOI：10.3390/fractalfract8080444.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
A novel morphing activation function is proposed, motivated by the wavelet theory and the use of wavelets as activation functions. Morphing refers to the gradual change of shape to mimic several apparently unrelated activation functions. The shape is controlled by the fractional order derivative, which is a trainable parameter to be optimized in the neural network learning process. Given the morphing activation function, and taking only integer-order derivatives, efficient piecewise polynomial versions of several existing activation functions are obtained. Experiments show that the performance of polynomial versions PolySigmoid, PolySoftplus, PolyGeLU, PolySwish, and PolyMish is similar or better than their counterparts Sigmoid, Softplus, GeLU, Swish, and Mish. Furthermore, it is possible to learn the best shape from the data by optimizing the fractional-order derivative with gradient descent algorithms, leading to the study of a more general formula based on fractional calculus to build and adapt activation functions with properties useful in machine learning.
Keyword:
fractional derivative
activation function
machine learning
wavelets
期刊
IF:
3.3
论文数:
4.3K
被引数:
7.6K
机构
引用论文
Real-time and offline techniques for identifying obstructive sleep apnea patients用于识别阻塞性睡眠呼吸暂停患者的实时和离线技术
PREAI
分享收藏Fractional-order gradient descent learning of BP neural networks with Caputo derivative.png)
NEURAL NETWORKS
NEURAL NETWORKSIF6.3
PREAI
分享收藏分享收藏TanhSoft-Dynamic Trainable Activation Functions for Faster Learning and Better PerformanceIEEE ACCESSOAAI
IEEE ACCESSIF3.6
OAAI分享收藏Improved Convolutional Neural Network Based on Fast Exponentially Linear Unit Activation FunctionIEEE ACCESS
OAAI
IEEE ACCESSIF3.6
OAAI分享收藏
查看更多内容