arrow
返回

Accelerating DNN Training Through Selective Localized Learning

delete2022-01-11
delete6
delete
OA
AI
S
Sarada Krithivasan *
S
Sanchari Sen
S
Swagath Venkataramani
A
Anand Raghunathan
DOI:10.3389/fnins.2021.759807delete
delete原文链接
delete分享
delete收藏
查看原文
摘要

摘要

En 中文
Training Deep Neural Networks (DNNs) places immense compute requirements on the underlying hardware platforms, expending large amounts of time and energy. We propose LoCal+SGD, a new algorithmic approach to accelerate DNN training by selectively combining localized or Hebbian learning within a Stochastic Gradient Descent (SGD) based training framework. Back-propagation is a computationally expensive process that requires 2 Generalized Matrix Multiply (GEMM) operations to compute the error and weight gradients for each layer. We alleviate this by selectively updating some layers' weights using localized learning rules that require only 1 GEMM operation per layer. Further, since localized weight updates are performed during the forward pass itself, the layer activations for such layers do not need to be stored until the backward pass, resulting in a reduced memory footprint. Localized updates can substantially boost training speed, but need to be used judiciously in order to preserve accuracy and convergence. We address this challenge through a Learning Mode Selection Algorithm, which gradually selects and moves layers to localized learning as training progresses. Specifically, for each epoch, the algorithm identifies a Localized -> SGD transition layer that delineates the network into two regions. Layers before the transition layer use localized updates, while the transition layer and later layers use gradient-based updates. We propose both static and dynamic approaches to the design of the learning mode selection algorithm. The static algorithm utilizes a pre-defined scheduler function to identify the position of the transition layer, while the dynamic algorithm analyzes the dynamics of the weight updates made to the transition layer to determine how the boundary between SGD and localized updates is shifted in future epochs. We also propose a low-cost weak supervision mechanism that controls the learning rate of localized updates based on the overall training loss. We applied LoCal+SGD to 8 image recognition CNNs (including ResNet50 and MobileNetV2) across 3 datasets (Cifar10, Cifar100, and ImageNet). Our measurements on an Nvidia GTX 1080Ti GPU demonstrate upto 1.5x improvement in end-to-end training time with ~0.5% loss in Top-1 classification accuracy.
Keyword:
Deep Neural Networks (DNNs)
localized learning
runtime efficiency
graphics process unit (GPU)
stochastic gradient decent algorithm
AI总结

AI总结

对已上传原文的论文进行重点信息的提取,主要内容包括:简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。

期刊

Frontiers in Neuroscience 封面图
Frontiers in Neuroscience
IF:
3.2
论文数:
1.6W
被引数:
6.8W

机构

Purdue University System 封面图
Purdue University System
学者数:
4.0W
论文数: 3.6W
被引数: 4.7W
P
Purdue University
学者数:
2.0K
论文数: 706
被引数: 1.5W
引用论文

引用论文

err分享
err收藏
EyeContext
err2013-04-27
err0
PREAI
errAndreas Bulling; Christian Weichel; Hans Gellersen
err分享
err收藏
Preparation of zinc oxide free, transparent rubber nanocomposites using a layered double hydroxide filler
err2011-01-01
err0
PREAI
errAmit Das; De-Yi Wang; Andreas Leuteritz; Kalaivani Subramaniam; H. Chris Greenwell; Udo Wagenknecht; Gert Heinrich
err分享
err收藏
Three-dimensional numerical study of flow structure in channel confluences
err2010-05-01
err0
PREAI
errAhmad Shakibainia; Mohammad Reza Majdzadeh Tabatabai; Amir Reza Zarrati
err分享
err收藏
Quantifying the uncertainty in change points
err2012-03-14
err0
errOAAI
errChristopher F. H. Nam; John A. D. Aston; Adam M. Johansen
err分享
err收藏
学者 查看更多内容