Efficient Precision-Adjustable Architecture for Softmax Function in Deep Learning

Danyang Zhu

doi:10.1109/TCSII.2020.3002564

Return

Efficient Precision-Adjustable Architecture for Softmax Function in Deep Learning

2020-12-01

PRE

DOI：10.1109/TCSII.2020.3002564

Original

Original request for help

Save

Abstract

En 中文

The softmax function has been widely used in deep neural networks (DNNs), and studies on efficient hardware accelerators for DNN have also attracted tremendous attention. However, it is very challenging to design efficient hardware architectures for softmax because of the expensive exponentiation and division calculations in it. In this brief, the softmax function is firstly simplified by exploring algorithmic strength reductions. Afterwards, a hardware-friendly and precision-adjustable calculation method for softmax is proposed, which can meet different precision requirements in various deep learning (DL) tasks. Based on the above innovations, an efficient architecture for softmax is presented. By tuning the parameter P, the system accuracy and complexity can be adjusted dynamically to achieve a good tradeoff between them. The proposed design is coded using hardware description language (HDL) and evaluated on two platforms, Xilinx Zynq-7000 ZC706 development board and TSMC 28-nm CMOS technology, respectively. Hardware implementation results show that our architecture significantly outperforms other works in speed and area, and that by adjusting P, the accuracy can be further increased with little hardware overhead.

Keywords:

Hardware

Computer architecture

Complexity theory

Neural networks

Circuits and systems

Task analysis

Mathematical model

Softmax

hardware architecture

deep neural network

transformer

VLSI

AI Summary

Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.