.png){kind=icon}
返回
Sample complexity of classification with compressed input
.png){kind=icon}
PRE
AI
DOI：10.1016/j.neucom.2020.07.043.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
One of the most studied problems in machine learning is finding reasonable constraints that guarantee the generalization of a learning algorithm. These constraints are usually expressed as some simplicity assumptions on the target. For instance, in the Vapnik-Chervonenkis (VC) theory the space of possible hypotheses is considered to have a limited VC dimension One way to formulate the simplicity assumption is via information theoretic concepts. In this paper, the constraint on the entropy H(X) of the input variable X is studied as a simplicity assumption. It is proven that the sample complexity to achieve an epsilon-delta Probably Approximately Correct (PAC) hypothesis is bounded by 2(cH(X)/epsilon) +log1/2/alpha epsilon(2) which is sharp up to the 1/epsilon(2) factor (a and c are constants). Moreover, it is shown that if a feature learning process is employed to learn the compressed representation from the dataset, this bound no longer exists. These findings have important implications on the Information Bottleneck (IB) theory which had been utilized to explain the generalization power of Deep Neural Networks (DNNs), but its applicability for this purpose is currently under debate by researchers. In particular, this is a rigorous proof for the previous heuristic that compressed representations are exponentially easier to be learned. However, our analysis pinpoints two factors preventing the IB, in its current form, to be applicable in studying neural networks. Firstly, the exponential dependence of sample complexity on 1/epsilon., which can lead to a dramatic effect on the bounds in practical applications when epsilon is small. Secondly, our analysis reveals that arguments based on input compression are inherently insufficient to explain generalization of methods like DNNs in which the features are also learned using available data. (C) 2020 Elsevier B.V. All rights reserved.
Keyword:
Compressed representation
Generalization bound
Information bottleneck
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
6.5
论文数:
2.5W
被引数:
6.5W
机构
引用论文
Situación de la infección por SARS-CoV-2 en pacientes en tratamiento renal sustitutivo. Informe del Registro COVID-19 de la Sociedad Española de Nefrología (SEN).png)
Nefrología
OAAI
NefrologíaIF0
OAAI分享收藏分享收藏OVERTAKEN BY REALITY: THE TEMPORAL INTERPLAY BETWEEN RHETORIC AND COLLECTIVE ACTION被现实超越：修辞与集体行动之间的时间性互动
PREAI
分享收藏 没有更多内容
没有更多内容