.png){kind=icon}
Return
Adaptive Morphing Activation Function for Neural Networks
.png){kind=icon}
OA
AI
DOI：10.3390/fractalfract8080444.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
A novel morphing activation function is proposed, motivated by the wavelet theory and the use of wavelets as activation functions. Morphing refers to the gradual change of shape to mimic several apparently unrelated activation functions. The shape is controlled by the fractional order derivative, which is a trainable parameter to be optimized in the neural network learning process. Given the morphing activation function, and taking only integer-order derivatives, efficient piecewise polynomial versions of several existing activation functions are obtained. Experiments show that the performance of polynomial versions PolySigmoid, PolySoftplus, PolyGeLU, PolySwish, and PolyMish is similar or better than their counterparts Sigmoid, Softplus, GeLU, Swish, and Mish. Furthermore, it is possible to learn the best shape from the data by optimizing the fractional-order derivative with gradient descent algorithms, leading to the study of a more general formula based on fractional calculus to build and adapt activation functions with properties useful in machine learning.
Keywords:
fractional derivative
activation function
machine learning
wavelets
Journal
IF:
3.3
Papers:
4.4K
Citations:
7.6K
Organization
Cited Papers
Fractional-order gradient descent learning of BP neural networks with Caputo derivative.png)
NEURAL NETWORKS
NEURAL NETWORKSIF6.3
PREAI
ShareSaveShareSaveTanhSoft-Dynamic Trainable Activation Functions for Faster Learning and Better PerformanceIEEE ACCESSOAAI
IEEE ACCESSIF3.6
OAAIShareSaveImproved Convolutional Neural Network Based on Fast Exponentially Linear Unit Activation FunctionIEEE ACCESS
OAAI
IEEE ACCESSIF3.6
OAAIShareSave
View more