.png){kind=icon}
返回
On Consensus-Optimality Trade-offs in Collaborative Deep Learning
.png){kind=icon}
OA
AI
DOI：10.3389/frai.2021.573731.png)
.png){kind=icon}
.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
In distributed machine learning, where agents collaboratively learn from diverse private data sets, there is a fundamental tension between consensus and optimality. In this paper, we build on recent algorithmic progresses in distributed deep learning to explore various consensus-optimality trade-offs over a fixed communication topology. First, we propose the incremental consensus-based distributed stochastic gradient descent (i-CDSGD) algorithm, which involves multiple consensus steps (where each agent communicates information with its neighbors) within each SGD iteration. Second, we propose the generalized consensus-based distributed SGD (g-CDSGD) algorithm that enables us to navigate the full spectrum from complete consensus (all agents agree) to complete disagreement (each agent converges to individual model parameters). We analytically establish convergence of the proposed algorithms for strongly convex and nonconvex objective functions; we also analyze the momentum variants of the algorithms for the strongly convex case. We support our algorithms via numerical experiments, and demonstrate significant improvements over existing methods for collaborative deep learning.
Keyword:
distributed optimization
consensus-optimality
collaborative deep learning
sgd
convergence
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
F
IF:
4.7
论文数:
2.5K
被引数:
4.4K
机构
引用论文
Distributed estimation over a low-cost sensor network: A Review of state-of-the-art低成本传感器网络上的分布式估计: 最新技术综述.png)
INFORMATION FUSIONOAAI
INFORMATION FUSIONIF15.5
OAAI分享收藏Gradient-based learning applied to document recognition基于梯度的学习在文档识别中的应用PROCEEDINGS OF THE IEEE
PROCEEDINGS OF THE IEEEIF25.9
PREAI
分享收藏分享收藏Distributed optimization for the multi-robot system using a neurodynamic approachNEUROCOMPUTING
NEUROCOMPUTINGIF6.5
PREAI
分享收藏
查看更多内容