Understanding How Pretraining Regularizes Deep Learning Algorithms

doi:10.1109/TNNLS.2021.3131377

Understanding How Pretraining Regularizes Deep Learning Algorithms

2023-09-01

7

PRE

AI

DOI：10.1109/TNNLS.2021.3131377

Original

Original request for help

Save

Abstract

En 中文

Deep learning algorithms have led to a series of breakthroughs in computer vision, acoustical signal processing, and others. However, they have only been popularized recently due to the groundbreaking techniques developed for training deep architectures. Understanding the training techniques is important if we want to further improve them. Through extensive experimentation, Erhan et al. (2010) empirically illustrated that unsupervised pretraining has an effect of regularization for deep learning algorithms. However, theoretical justifications for the observation remain elusive. In this article, we provide theoretical supports by analyzing how unsupervised pretraining regularizes deep learning algorithms. Specifically, we interpret deep learning algorithms as the traditional Tikhonov-regularized batch learning algorithms that simultaneously learn predictors in the input feature spaces and the parameters of the neural networks to produce the Tikhonov matrices. We prove that unsupervised pretraining helps in learning meaningful Tikhonov matrices, which will make the deep learning algorithms uniformly stable and the learned predictor will generalize fast w.r.t. the sample size. Unsupervised pretraining, therefore, can be interpreted as to have the function of regularization.

Keywords:

Deep learning

Training

Neural networks

Prediction algorithms

Signal processing algorithms

Decoding

Approximation algorithms

Classification

deep learning

regularization

representation learning

unsupervised pretrain

Understanding How Pretraining Regularizes Deep Learning Algorithms

Abstract

Journal

Organization