Revisiting Internal Covariate Shift for Batch Normalization

doi:10.1109/TNNLS.2020.3026784

Return

Revisiting Internal Covariate Shift for Batch Normalization

2021-11-01

PRE

Muhammad Awais

Md. Tauhid Bin Iqbal

Sung‐Ho Bae *

DOI：10.1109/TNNLS.2020.3026784

Original

Original request for help

Save

Abstract

En 中文

Despite the success of batch normalization (BatchNorm) and a plethora of its variants, the exact reasons for its success are still shady. The original BatchNorm article explained it as a mechanism that reduces the internal covariate shift (ICS), i.e., the distribution shifts in the input of the layers during training. Recently, some articles manifested skepticism on this hypothesis and provided alternative explanations for the success of BatchNorm, such as the applicability of very high learning rates and the ability to smooth the landscape in optimization. In this work, we counter these alternative arguments by demonstrating the importance of reduction in ICS following an empirical approach. We demonstrated various ways to achieve the abovementioned alternative properties without any performance boost. In this light, we explored the importance of different BatchNorm parameters (i.e., batch statistics and affine transformation parameters) by visualizing their effectiveness in the performance and analyzed their connections with ICS. Afterward, we showed a different normalization scheme that fulfills all the alternative explanations except reduction in ICS. Despite having all the alternative properties, we observed its poor performance, which nullifies the alternative claims, rather signifies the importance of the ICS reduction. We performed comprehensive experiments on many variants of BatchNorm, finding that all of them similarly reduce ICS.

Keywords:

Integrated circuits

Training

Neural networks

Standards

Optimization

Switches

Learning systems

Batch normalization (BatchNorm)

deep learning

internal covariate shift (ICS)

AI Summary

Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.