DropDim: A Regularization Method for Transformer Networks

Hao Zhang

doi:10.1109/LSP.2022.3140693

DropDim: A Regularization Method for Transformer Networks

2022-01-01

4

OA

AI

DOI：10.1109/LSP.2022.3140693

Original

Original request for help

Save

Abstract

En 中文

We introduce DropDim, a structured dropout method designed for regularizing the self-attention mechanism, which is a key component of the transformer. In contrast to the general dropout method, which randomly drops neurons, DropDim drops part of the embedding dimensions. In this way, the semantic information can be completely discarded. Thus, the excessive co-adapting between different embedding dimensions can be broken, and the self-attention is forced to encode meaningful features with a certain number of embedding dimensions erased. Experiments on a wide range of tasks executed on the MUST-C English-Germany dataset show that DropDim can effectively improve model performance, reduce over-fitting, and show complementary effects with other regularization methods. When combined with label smoothing, the WER can be reduced from 19.1% to 15.1% on the ASR task, and the BLEU value can be increased from 26.90 to 28.38 on the MT task. On the ST task, the model can reach a BLEU score of 22.99, an increase by 1.86 BLEU points compared to the strong baseline.

Keywords:

Task analysis

Transformers

Smoothing methods

Semantics

Training

Neurons

Decoding

End-to-end

transformer

regularization

dropout

DropDim: A Regularization Method for Transformer Networks

Abstract

Journal

Organization