.png){kind=icon}
返回
An Efficient and Robust Cloud-Based Deep Learning With Knowledge Distillation
.png){kind=icon}
PRE
AI
DOI：10.1109/TCC.2022.3160129.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
In recent years, deep neural networks have shown extraordinary power in various practical learning tasks, especially in object detection, classification, natural language processing. However, deploying such large models on resource-constrained devices or embedded systems is challenging due to their high computational cost. Efforts such as model partition, pruning, or quantization have been used at the expense of accuracy loss. Knowledge distillation is a technique that transfers model knowledge from a well-trained model (teacher) to a smaller and shallow model (student). Instead of using a learning model on the cloud, we can deploy distilled models on various edge devices, significantly reducing the computational cost, memory usage and prolonging the battery lifetime. In this work, we propose a novel neuron manifold distillation (NMD) method, where the student models imitate the teacher's output distribution and learn the feature geometry of the teacher model. In addition, to further improve the cloud-based learning system reliability, we propose a confident prediction mechanism to calibrate the model predictions. We conduct experiments with different distillation configurations over multiple datasets. Our proposed method demonstrates a consistent improvement in accuracy-speed trade-offs for the distilled model.
Keyword:
Deep learning
transfer learning
knowledge distillation
期刊
I
IF:
5
论文数:
1.8K
被引数:
4.3K
机构
引用论文
.png){kind=icon}
A survey of federated learning for edge computing: Research problems and solutions面向边缘计算的联邦学习综述: 研究问题与解决方案
OAAI
OAAI分享收藏Privacy-Preserving Computation Offloading for Parallel Deep Neural Networks Training面向并行深度神经网络训练的隐私保护计算卸载
OAAI
OAAI分享收藏
查看更多内容