Multi-scale adaptive networks for efficient inference

doi:10.1007/s13042-023-01908-4

Multi-scale adaptive networks for efficient inference

2023-07-05

1

PRE

AI

DOI：10.1007/s13042-023-01908-4

Original

Original request for help

Save

Abstract

En 中文

The success of deep neural networks has been impressive in many areas. However, the increase in model performance is usually accompanied by an increase in depth and width, which is not conducive to the model being deployed at the edge. To address this problem, a new inference framework, multi-scale adaptive networks (MSAN), is proposed. Specifically, several branches are added at different stages of the network, and a scalable attention as well as self-distillation are used to improve the performance of shallow branches. To enhance the distillation effect and to reuse features efficiently, the knowledge from shallow and deep layers is fused through selective feature connections. In addition, two adaptive distillation strategies are proposed to further improve the performance of self-distillation. MSAN can be used to promote the performance of networks, static model compression and dynamic inference. Extensive experiments have demonstrated the superior performance of MSAN in these three aspects.

Keywords:

Attention mechanism

Adaptive self-distillation

Selective dense feature connections

Model compression

Inference acceleration

Multi-scale adaptive networks for efficient inference

Abstract

Journal

Organization