High-Precision Method and Architecture for Base-2 Softmax Function in DNN Training

Yuan Zhang

doi:10.1109/TCSI.2023.3277247

High-Precision Method and Architecture for Base-2 Softmax Function in DNN Training

2023-08-01

8

PRE

AI

DOI：10.1109/TCSI.2023.3277247

Original

Original request for help

Save

Abstract

En 中文

Softmax is a common and complex activation function in Deep Neural Networks (DNN). However, it is a challenge to apply it efficiently in DNN training hardware accelerator. Therefore, we propose a high precision calculation method and architecture based on base-2 softmax, which has low hardware complexity than base-2 softmax but can still be useful in DNN training. First, we simplify the hardware implementation complexity of calculating base-2 softmax. Second, we use the base-TEXPRESERVE4 hyperbolic COordinate Rotation Digital Computer (CORDIC) to implement the core computation. Finally, we show that the proposed method can be used in DNN training through experiments. Moreover, with the same order of the magnitude of high precision, our hardware cost is lower than traditional base-e softmax or other alternative design methods. Under TSMC 28nm CMOS technology, an example design of our architecture has the area of 98787.43 mu m(2) and the power consumption of 24.72 mW for circuit synthesis at the frequency of 1GHz.

Keywords:

Softmax

DNN

training

high precision

hyperbolic CORDIC

High-Precision Method and Architecture for Base-2 Softmax Function in DNN Training

Abstract

Journal

Organization