.png){kind=icon}
.png){kind=icon}
Return
Learnable attention driven structured compression technique for neural networks
.png){kind=icon}
OA
AI
DOI：10.1038/s41598-026-49893-2.png)
.png){kind=icon}
.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
The recent involvement of neural networks in modern applications demands high accuracy even in resource constraint environments. Conventionally, high accuracy requires more computation, memory and energy, especially for models that perform substantially complex tasks. In this paper, we introduce a powerful compression technique named, “Attention Driven Structured Compression (ADSC)” that has the ability to significantly reduce model size while preserving accuracy. ADSC uses learnable attention-inspired MLP to compute filter importance using lightweight neural network making it computationally efficient unlike other compression techniques that require intensive retraining. Furthermore, Instead of relying on fixed parameters, ADSC learns to pick the right filters based on how the patterns actually activate. We have conducted numerous experiments on diverse datasets including MNIST, CIFAR 10/100 and California Housing to demonstrate the effectiveness of ADSC. The ADSC attains 50-60% compression with minimal performance degradation on all the tested datasets. The performance of ADSC is compared with other popular compression methods like L1 Channel Pruning, Lottery Ticket and NISP (Neuron Importance Score Propagation) based on compression ratios, accuracy loss and computational overhead. These comparisons showed that ADSC outperformed existing techniques. It maintains high accuracy even under heavy compression and keeps a balance that is crucial for deploying deep neural networks on mobile hardware.
Keywords:
Engineering
Mathematics and computing
Attention
Compression
Neural Network
Structured Pruning
Science
Humanities and Social Sciences
multidisciplinary
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.
Journal
IF:
3.9
Papers:
27.1W
Citations:
83.5W