.png){kind=icon}
返回
DropDim: A Regularization Method for Transformer Networks
.png){kind=icon}
OA
AI
DOI：10.1109/LSP.2022.3140693.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
We introduce DropDim, a structured dropout method designed for regularizing the self-attention mechanism, which is a key component of the transformer. In contrast to the general dropout method, which randomly drops neurons, DropDim drops part of the embedding dimensions. In this way, the semantic information can be completely discarded. Thus, the excessive co-adapting between different embedding dimensions can be broken, and the self-attention is forced to encode meaningful features with a certain number of embedding dimensions erased. Experiments on a wide range of tasks executed on the MUST-C English-Germany dataset show that DropDim can effectively improve model performance, reduce over-fitting, and show complementary effects with other regularization methods. When combined with label smoothing, the WER can be reduced from 19.1% to 15.1% on the ASR task, and the BLEU value can be increased from 26.90 to 28.38 on the MT task. On the ST task, the model can reach a BLEU score of 22.99, an increase by 1.86 BLEU points compared to the strong baseline.
Keyword:
Task analysis
Transformers
Smoothing methods
Semantics
Training
Neurons
Decoding
End-to-end
transformer
regularization
dropout
期刊
IF:
9.6
论文数:
1.1W
被引数:
1.7W
机构
暂无机构信息
引用论文
.png){kind=icon}
分享收藏Ichnological analysis as a tool for assessing deep-sea circulation in the westernmost Mediterranean over the last Glacial Cycle遗迹学分析作为一种评估地中海最西端在最后一个冰期旋回中深部环流状况的工具
PREAI
分享收藏Transcriptome analysis of human dorsal striatum implicates attenuated canonical WNT signaling in neuroinflammation and in age-related impairment of striatal neurogenesis and synaptic plasticity人类背侧纹状体转录组分析表明，减弱的canonical WNT信号通路参与神经炎症以及与年龄相关的纹状体神经发生和突触可塑性的损害。
PREAI
分享收藏Sucrose Preference and Novelty-Induced Hypophagia Tests in Rats using an Automated Food Intake Monitoring System使用自动化食物摄入监测系统进行的蔗糖偏好和 novelty-induced hypophagia 测试
PREAI
分享收藏
查看更多内容