.png){kind=icon}
返回
FEDBERT: When Federated Learning Meets Pre-training
.png){kind=icon}
PRE
AI
DOI：10.1145/3510033.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
The fast growth of pre-trained models (PTMs) has brought natural language processing to a new era, which has become a dominant technique for various natural language processing (NLP) applications. Every user can download the weights of PTMs, then fine-tune the weights for a task on the local side. However, the pre-training of a model relies heavily on accessing a large-scale of training data and requires a vast amount of computing resources. These strict requirements make it impossible for any single client to pre-train such a model. To grant clients with limited computing capability to participate in pre-training a large model, we propose a new learning approach, FEDBERT, that takes advantage of the federated learning and split learning approaches, resorting to pre-training BERT in a federated way. FEDBERT can prevent sharing the raw data information and obtain excellent performance. Extensive experiments on seven GLUE tasks demonstrate that FEDBERT can maintain its effectiveness without communicating to the sensitive local data of clients.
Keyword:
Federated learning
pre-training
BERT
NLP
期刊
IF:
6.6
论文数:
1.5K
被引数:
6.2K
机构
引用论文
.png){kind=icon}
分享收藏Cobalt and Copper Composite Oxides as Efficient Catalysts for Preferential Oxidation of CO in H2-Rich Stream钴和铜复合氧化物作为H2-Rich流中CO优先氧化的有效催化剂
PREAI
分享收藏
查看更多内容