Filter Sketch for Network Pruning

doi:10.1109/TNNLS.2021.3084206

Filter Sketch for Network Pruning

2022-12-01

55

OA

AI

DOI：10.1109/TNNLS.2021.3084206

Original

Save

View PDF

Abstract

En 中文

We propose a novel network pruning approach by information preserving of pretrained network weights (filters). Network pruning with the information preserving is formulated as a matrix sketch problem, which is efficiently solved by the off-the-shelf frequent direction method. Our approach, referred to as FilterSketch, encodes the second-order information of pretrained weights, which enables the representation capacity of pruned networks to be recovered with a simple fine-tuning procedure. FilterSketch requires neither training from scratch nor data-driven iterative optimization, leading to a several-orders-of-magnitude reduction of time cost in the optimization of pruning. Experiments on CIFAR-10 show that FilterSketch reduces 63.3% of floating-point operations (FLOPs) and prunes 59.9% of network parameters with negligible accuracy cost for ResNet-110. On ILSVRC-2012, it reduces 45.5% of FLOPs and removes 43.0% of parameters with only 0.69% accuracy drop for ResNet-50. Our code and pruned models can be found at https://github.com/lmbxmu/FilterSketch.

Keywords:

Information filters

Optimization

Training

Computational modeling

Covariance matrices

Hardware

Informatics

Filter pruning

information preserving

network compression and acceleration

network pruning

sketch

structured pruning

Filter Sketch for Network Pruning

Abstract

AI Summary

Journal

Organization