Diluted binary neural network

Yuhan Lin

doi:10.1016/j.patcog.2023.109556

Diluted binary neural network

2023-08-01

2

PRE

AI

DOI：10.1016/j.patcog.2023.109556

Original

Original request for help

Save

Abstract

En 中文

Binary neural networks (BNNs) are promising on resource-constrained devices because they reduce mem-ory consumption and accelerate inference effectively. However, they are still potential on performance improvement. Prior studies attribute performance degradation of BNNs to limited representation ability and gradient mismatch. In this paper, we find that it also results from the mandatory representation of small full-precision auxiliary weights to large values. To tackle with this issue, we propose an approach dubbed as Diluted Binary Neural Network (DBNN). Besides avoiding mandatory representation effectively, the proposed DBNN also alleviates sign flip problem to a large extent. For activations, we jointly min-imize quantization error and maximize information entropy to develop the binarization scheme. Com-pared with existing sparsity-binarization approaches, DBNN trains network from scratch without other procedures and achieves larger sparsity. Experiments on several datasets with various networks demon-strate the superiority of our approach. (c) 2023 Elsevier Ltd. All rights reserved.

Keywords:

Model compression

Network quantization

Binary neural network

Ternary neural network

Sparse regularization

Diluted binary neural network

Abstract

Journal

Organization