.png){kind=icon}
返回
Bayesian interpolation with deep linear networks
.png){kind=icon}
OA
AI
DOI：10.1073/pnas.2301345120.png)
.png){kind=icon}
.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Characterizing how neural network depth, width, and dataset size jointly impact model quality is a central problem in deep learning theory. We give here a complete solution in the special case of linear networks with output dimension one trained using zero noise Bayesian inference with Gaussian weight priors and mean squared error as a negative log-likelihood. For any training dataset, network depth, and hidden layer widths, we find nonasymptotic expressions for the predictive posterior and Bayesian model evidence in terms of Meijer-G functions, a class of meromorphic special functions of a single complex variable. Through asymptotic expansions of these Meijer-G functions, a rich new picture of the joint role of depth, width, and dataset size emerges. We show that linear networks make provably optimal predictions at infinite depth: the posterior of infinitely deep linear networks with data-agnostic priors is the same as that of shallow networks with evidence-maximizing data-dependent priors. This yields a principled reason to prefer deeper networks when priors are forced to be data -agnostic. Moreover, we show that with data-agnostic priors, Bayesian model evidence in wide linear networks is maximized at infinite depth, elucidating the salutary role of increased depth for model selection. Underpinning our results is an emergent notion of effective depth, given by the number of hidden layers times the number of data points divided by the network width; this determines the structure of the posterior in the large-data limit.
Keyword:
deep learning
Bayesian inference
neural networks
linear networks
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
P
IF:
9.1
论文数:
10.8W
被引数:
73.5W
机构
引用论文
Arsenic Removal from Natural Water Using Low Cost Granulated Adsorbents: A Review使用低成本颗粒状吸附剂从天然水中去除砷: 综述
OAAI
OAAI分享收藏Early Cretaceous dioritic rocks in the Tongling region, eastern China: Implications for the tectonic settings.png)
Lithos
LithosIF0
PREAI
分享收藏分享收藏
查看更多内容