.png){kind=icon}
返回
Shape autotuning activation function
.png){kind=icon}
PRE
AI
DOI：10.1016/j.eswa.2020.114534.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
The choice of activation function is essential for building state-of-the-art neural networks. At present, the most widely-used activation function with effectiveness is ReLU. However, ReLU suffers from the weakness including non-zero mean, negative missing, and unbounded output, thus it has potential disadvantages in the optimization process. In this paper, we propose a novel activation function, namely ?Shape Autotuning Activation Function? (SAAF), to overcome these three challenges simultaneously. The SAAF inherits merits of smooth activation functions (such as Sigmoid and Tanh) and piecewise activation functions (such as ReLU and its variants), and avoids their deficiencies. Specifically, the SAAF adaptively adjusts a pair of independent trainable parameters to capture negative information and provide a near-zero mean output, resulting in better generalization performance and faster learning speed. At the same time, it provides bounded outputs to ensure a more stable distribution of output during network training. We evaluated SAAF on deep networks applied to a variety of tasks, including image classification, machine translation, and generative modeling. Comprehensive comparison study shows that the proposed SAAF is superior to state-of-the-art activation functions.
Keyword:
Activation functions
Deep learning
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
7.5
论文数:
3.0W
被引数:
10.2W
机构
引用论文
分享收藏Enhancing batch normalized convolutional networks using displaced rectifier linear units: A systematic comparative study使用位移整流器线性单元增强批量归一化卷积网络: 系统的比较研究
PREAI
分享收藏Improving deep neural network with Multiple Parametric Exponential Linear Units.png)
NEUROCOMPUTING
OAAI
NEUROCOMPUTINGIF6.5
OAAI分享收藏
查看更多内容