.png){kind=icon}
.png){kind=icon}
Return
Hardware-Efficient Unified Approximation for Implementing Diverse Smooth Activation Functions
.png){kind=icon}
PRE
AI
DOI：10.1109/TC.2026.3661496.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Smooth activation functions such as Swish, GELU, and Tanhexp have emerged as effective alternatives to ReLU, alleviating issues such as dying ReLU and gradient vanishing while improving training stability and accuracy. However, their computational complexity poses significant challenges for efficient hardware realization. Moreover, despite their structural similarity, existing hardware implementations treat each smooth activation independently, resulting in redundant and costly designs. This paper proposes a unified, hardware-efficient approximation framework that models multiple smooth activation functions using a single parameterized polynomial form. By adjusting only a few parameters, the framework can approximate diverse smooth activations while maintaining their functional characteristics, including accuracy and convergence speed. The proposed implementation requires only two multipliers and one adder, achieving up to 87% reduction in area and 86% energy savings compared to conventional LUT-based designs. Experimental validation on several neural networks demonstrates that the approximated activations do not degrade the accuracy. Owing to its generality and efficiency, the proposed framework is well suited for mixed-activation neural architectures and hardware-aware neural architecture search.
Keywords:
Hardware-efficient accelerator
Smooth activation approximation
Unified approximation framework
Journal
IF:
3.8
Papers:
5.3K
Citations:
9.8K