.png){kind=icon}
.png){kind=icon}
Return
Asynchronous Control Based Aggregation Transport Protocol for Distributed Deep Learning
.png){kind=icon}
PRE
AI
DOI：10.1109/TC.2025.3525604.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
With the rapid growth scale of dataset and model, the training of deep neural networks (DNN) tends to be deployed in a distributed manner. In the large-scale distributed training, the bottlenecks have gradually moved from computational resources to communication process. Recent researches adopt in-network aggregation (INA) that offloads the gradient aggregation process to programmable switches, thereby reducing network traffic amount and transmission latency. Unfortunately, due to the bandwidth competition in shared training clusters, the straggler will slow down the training efficiency of INA. To address this issue, we propose an Asynchronous Control based Aggregation Transport Protocol (AC-ATP), which makes full use uncongested links to transmit gradients and the switch memory to cache gradients from the fast workers to accelerate the gradient aggregation. Meanwhile, AC-ATP performs congestion control according to the transmission progress of worker and the remaining completion time of the job. The evaluation results of real testbed and large-scale simulations show that AC-ATP reduces the aggregate time by up to 68% and speeds up training in real-world benchmark models.
Keywords:
Switches
Training
Bandwidth
Computers
Transport protocols
Memory management
Control systems
Artificial neural networks
Deep learning
Servers
Distributed training
in-network aggregation
straggler
transport protocol
Journal
IF:
3.8
Papers:
5.3K
Citations:
9.8K