.png){kind=icon}
返回
Adaptively Customizing Activation Functions for Various Layers
.png){kind=icon}
OA
AI
DOI：10.1109/TNNLS.2021.3133263.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
To enhance the nonlinearity of neural networks and increase their mapping abilities between the inputs and response variables, activation functions play a crucial role to model more complex relationships and patterns in the data. In this work, a novel methodology is proposed to adaptively customize activation functions only by adding very few parameters to the traditional activation functions such as Sigmoid, Tanh, and rectified linear unit (ReLU). To verify the effectiveness of the proposed methodology, some theoretical and experimental analysis on accelerating the convergence and improving the performance is presented, and a series of experiments are conducted based on various network models (such as AlexNet, VggNet, GoogLeNet, ResNet and DenseNet), and various datasets (such as CIFAR10, CIFAR100, miniImageNet, PASCAL VOC, and COCO). To further verify the validity and suitability in various optimization strategies and usage scenarios, some comparison experiments are also implemented among different optimization strategies (such as SGD, Momentum, AdaGrad, AdaDelta, and ADAM) and different recognition tasks such as classification and detection. The results show that the proposed methodology is very simple but with significant performance in convergence speed, precision, and generalization, and it can surpass other popular methods such as ReLU and adaptive functions such as Swish in almost all experiments in terms of overall performance.
Keyword:
Neural networks
Training
Shape
Adaptive systems
Mathematical models
Learning systems
Deep learning
Adaptable parameters
adaptive activation function
deep learning
various layers
期刊
IF:
8.9
论文数:
7.6K
被引数:
7.2W
机构
引用论文
A joint residual network with paired ReLUs activation for image super-resolution一种用于图像超分辨率的配对ReLUs激活联合残差网络.png)
NEUROCOMPUTING
NEUROCOMPUTINGIF6.5
PREAI
分享收藏Investigation of Ti mesh-supported anodes for direct borohydride fuel cells直接硼氢化物燃料电池用钛网支撑阳极的研究
PREAI
分享收藏Structural study of lanthanides(III) in aqueous nitrate and chloride solutions by EXAFS通过EXAFS对硝酸盐和氯化物水溶液中镧系元素 (III) 的结构研究
PREAI
分享收藏分享收藏
Recycling of NdFeB Magnets from Electric Drive Motors of (Hybrid) Electric Vehicles从 (混合动力) 电动汽车的驱动电机中回收NdFeB磁体
PREAI
分享收藏Photograph aesthetical evaluation and classification with deep convolutional neural networks基于深度卷积神经网络的照片审美评价与分类NEUROCOMPUTING
NEUROCOMPUTINGIF6.5
PREAI
分享收藏
查看更多内容