Federated Split Learning via Mutual Knowledge Distillation

doi:10.1109/TNSE.2023.3348461

Federated Split Learning via Mutual Knowledge Distillation

2024-05-01

1

PRE

AI

L

Luo, Linjun

张

张幸林 (Xinglin Zhang) *

DOI：10.1109/TNSE.2023.3348461

Original

Original request for help

Save

Abstract

En 中文

Federated learning (FL) and split learning (SL) can coordinate multiple clients (e.g., end devices in mobile/IoT networks) to collaboratively train deep learning models without exposing clients' raw data. Generally, FL converges faster than SL, while SL requires less client-side resource than FL. To excavate their complementary advantages, recent works mostly apply FL to improve SL. These one-way synergy approaches hence face the same limitation as SL, i.e., the clients rely on an online server or download the entire large model to make inferences. Furthermore, they cannot handle the requirements on client-side model personalization. To address these, we propose a novel Federated Split learning framework via Mutual Knowledge Distillation (FSMKD), which intertwines FL with SL in a two-way manner, i.e., we enable FL and SL to boost each other's performance. Specifically, we design a two-body structure including the head:personalized-local-body:tail network as the local model and the head:shared-server-body:tail network as the global model. Thus, FSMKD can support personalized local models and exploit information across heterogeneous learning tasks for the global model training through deep mutual learning. Extensive experiments show that FSMKD outperforms the existing FL-SL synergy frameworks on the server-side and obtains a personalized model that outperforms FedAvg on the client-side.

Keywords:

Servers

Computational modeling

Data models

Training

Load modeling

Deep learning

Task analysis

Federated learning

split learning

knowledge distillation

Federated Split Learning via Mutual Knowledge Distillation

Abstract

Journal

Organization