Coded Deep Learning: Framework and Algorithm

En‐hui Yang

doi:10.1109/TIT.2025.3610095

Coded Deep Learning: Framework and Algorithm

2025-11-01

0

PRE

AI

E

En‐hui Yang *

S

Shayan Mohajer Hamidi

DOI：10.1109/TIT.2025.3610095

Original

Original request for help

Save

Abstract

En 中文

The success of deep learning (DL) is often achieved at the expense of large model sizes and high computational complexity during both training and post-training inferences, making it difficult to train and run large models in a resource-limited environment. To alleviate these issues, this paper introduces a new framework dubbed coded deep learning (CDL), which integrates information-theoretic coding concepts into the inner workings of DL, aiming to substantially compress model weights and activations, reduce computational complexity at both training and post-training inference stages, and enable efficient model/data parallelism. Specifically, within CDL, 1) we first propose a novel probabilistic method for quantizing both model weights and activations, and its soft differentiable variant which offers an analytic formula for gradient calculation during training; 2) both the forward and backward passes during training are executed over quantized weights and activations, which eliminates a majority of floating-point operations and reduces the training computation complexity; 3) during training, both weights and activations are entropy constrained so that they are compressible in an information-theoretic sense at any stage of training, which in turn reduces communication costs in cases where model/data parallelism is adopted; and 4) the trained model in CDL is by default in a quantized format with compressible quantized weights, reducing post-training inference complexity and model storage complexity. Additionally, a variant of CDL, namely relaxed CDL (R-CDL), is presented to further improve the trade-off between validation accuracy and compression at the disadvantage of full precision operation involved in forward and backward passes during training with other advantageous features of CDL intact. Extensive empirical results show that CDL and R-CDL outperform the state-of-the-art algorithms in DNN compression in the literature.

Keywords:

Training

Computational modeling

Quantization (signal)

Entropy

Parallel processing

Information theory

Data models

Probabilistic logic

Analytical models

Deep learning

entropy

Huffman coding

model and data parallelism

quantization-aware training

Coded Deep Learning: Framework and Algorithm

Abstract

Journal

Organization

Cited Papers