Layer-wise Principal Gradient Compression for Distributed Machine Learning

Zehua Guo

doi:10.1109/tc.2026.3727583

Layer-wise Principal Gradient Compression for Distributed Machine Learning

2026-08-26

0

PRE

AI

DOI：10.1109/tc.2026.3727583

Original

Original request for help

Save

Abstract

En 中文

Distributed Machine Learning (DML) can accelerate model training by using multiple training nodes and large datasets. As model and data sizes grow, DML often uses more training nodes, and frequent model information exchange creates high communication overhead. To reduce this overhead, we propose PrincipalCatcher, a layer-wise gradient compression method based on Principal Component Analysis (PCA) and error feedback. PrincipalCatcher uses PCA to capture the main gradient components in each layer and transmits only the compressed gradient representation during iterative training. To limit the accuracy loss caused by lossy compression, each worker accumulates compression errors and adds them to gradients in the next epoch. We evaluate PrincipalCatcher on image classification tasks under distributed training settings. The results show that PrincipalCatcher can reduce gradient transmission by up to 96.4% while keeping accuracy close to conventional stochastic gradient descent in most tested settings.

Keywords:

Distributed Machine Learning

Principal Component Analysis

Gradient Compression

Error Feedback

Layer-wise Principal Gradient Compression for Distributed Machine Learning

Abstract

Journal

Organization

Cited Papers