Parareal Neural Networks Emulating a Parallel-in-Time Algorithm

Youngkyu Lee

doi:10.1109/TNNLS.2022.3206797

Parareal Neural Networks Emulating a Parallel-in-Time Algorithm

2024-05-01

2

OA

AI

DOI：10.1109/TNNLS.2022.3206797

Original

Original request for help

Save

Abstract

En 中文

As deep neural networks (DNNs) become deeper, the training time increases. In this perspective, multi-CPU parallel computing has become a key tool in accelerating the training of DNNs. In this article, we introduce a novel methodology to construct a parallel neural network that can utilize multiple GPUs simultaneously from a given DNN. We observe that layers of DNN can be interpreted as the time steps of a time-dependent problem and can be parallelized by emulating a parallel-in-time algorithm called parareal. The parareal algorithm consists of fine structures which can be implemented in parallel and a coarse structure that gives suitable approximations to the fine structures. By emulating it, the layers of DNN are torn to form a parallel structure, which is connected using a suitable coarse network. We report accelerated and accuracy-preserved results of the proposed methodology applied to VGG-16 and ResNet-1001 on several datasets.

Keywords:

Deep neural network (DNN)

parallel computing

parareal algorithm

time-dependent problem

Parareal Neural Networks Emulating a Parallel-in-Time Algorithm

Abstract

Journal

Organization