Basis scaling and double pruning for efficient inference in network-based transfer learning

doi:10.1016/j.patrec.2023.11.026

Basis scaling and double pruning for efficient inference in network-based transfer learning

2024-01-01

1

PRE

AI

DOI：10.1016/j.patrec.2023.11.026

Original

Original request for help

Save

Abstract

En 中文

Network-based transfer learning allows the reuse of deep learning features with limited data, but the resulting models can be unnecessarily large. Although network pruning can improve inference efficiency, existing algorithms usually require fine-tuning that may not be suitable for small datasets. In this paper, using the singular value decomposition, we decompose a convolutional layer into two layers: a convolutional layer with the orthonormal basis vectors as the filters, and a BasisScalingConvlayer which is responsible for rescaling the features and transforming them back to the original space. As the filters in each decomposed layer are linearly independent, when using the proposed basis scaling factors with the Taylor approximation of importance, pruning can be more effective and fine-tuning individual weights is unnecessary. Furthermore, as the numbers of input and output channels of the original convolutional layer remain unchanged after basis pruning, it is applicable to virtually all architectures and can be combined with existing pruning algorithms for double pruning to further increase the pruning capability. When transferring knowledge from ImageNet pre-trained models to different target domains, with less than 1% reduction in classification accuracies, we can achieve pruning ratios up to 74.6% for CIFAR-10 and 98.9% for MNIST in model parameters.

Keywords:

Network pruning

Transfer learning

Efficient inference

Singular value decomposition

Double pruning

Basis scaling and double pruning for efficient inference in network-based transfer learning

Abstract

Journal

Organization