.png){kind=icon}
返回
Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability
.png){kind=icon}
OA
AI
DOI：10.1162/tacl_a_00708.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
How do language models learn to make predictions during pre-training? To study this, we extract learning curves from five autoregressive English language model pre-training runs, for 1M unseen tokens in context. We observe that the language models generate short repetitive phrases before learning to generate longer and more coherent text. We also find that individual tokens often exhibit sudden increases or decreases in loss that are surprisingly consistent across pre-training runs. To better understand these fluctuations, we quantify the final surprisal, within-run variability, age of acquisition, forgettability, and cross-run variability of learning curves for individual tokens in context. More frequent tokens reach lower final surprisals, exhibit less variability within and across pre-training runs, are learned earlier, and are less likely to be forgotten'' during pre-training. Higher n-gram probabilities further accentuate these effects. Independent of the target token, shorter and more frequent contexts correlate with marginally more stable and quickly acquired predictions. Based on our results, we argue for the existence of sequential learning dependencies between different model capabilities, and we characterize language model learning as early n-gram learning before gradual refinement of tail n-gram predictions.
期刊
T
IF:
6.9
论文数:
486
被引数:
5.7K
机构
引用论文
.png){kind=icon}
Matrix Effects in the Detection of Pb and Ba in Soils Using Laser-Induced Breakdown Spectroscopy使用激光诱导击穿光谱检测土壤中Pb和Ba的基质效应
PREAI
分享收藏Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond在实践中利用LLMs的力量: 关于ChatGPT及以后的调查OAAI
OAAI分享收藏Strong Prediction: Language Model Surprisal Explains Multiple N400 Effects强大的预测: 语言模型惊喜解释了多个N400效应
OAAI
OAAI分享收藏
查看更多内容