NetPlacer+: Model Parallelism Based on Load Balance in Distributed Deep Learning

doi:10.1109/TETCI.2025.3543765

NetPlacer+: Model Parallelism Based on Load Balance in Distributed Deep Learning

2025-01-01

0

PRE

AI

Mahdi Boloursaz Mashhadi

DOI：10.1109/TETCI.2025.3543765

Original

Original request for help

Save

Abstract

En 中文

The importance of Model Parallelism in Distributed Deep Learning continues to grow due to the increase in the Deep Neural Network (DNN) scale and the demand for higher training speed. Different from all the existing works, we propose a model-parallel strategy called NetPlacer+ based on load balance. The major idea in NetPlacer+ is to partition the DNN model into multiple devices by balancing each device's computation and communication load. We build the mathematical model of NetPlacer+. We transform the mathematical model of NetPlacer+ and obtain its approximate optimal solution using the interior point method. Extensive experiments in two GPU clusters and eight modern DNNs are conducted to verify the effectiveness of NetPlacer+. Experimental results show that the model-parallel strategy of NetPlacer+ achieves up to 1.25x speedup compared to NVIDIA's DLPlacer.

Keywords:

Distributed deep learning

model parallelism

load balance

NetPlacer+: Model Parallelism Based on Load Balance in Distributed Deep Learning

Abstract

Journal

Organization