[SAINT_Lab.]Accelerating Federated Learning with Split Learning.pdf
Federated learning (FL) (McMahan et al., 2017; Konecnˇ y`et al., 2016b) is being regarded as a promising direction for distributed learning, as it enables clients to collaboratively train a global model without directly uploading their privacy-sensitive data to the server.
However, in FL, each client should repeatedly download the entire model from the server, update the model, and upload it back to the server.
This training process causes significant computation/communication burdens especially with deep neural networks having large numbers of model parameters.
Moreover, when the computing powers and the transmission rates of the clients are low (e.g., mobile/IoT devices), FL requires significant computation/communication delays.
These issues can limit the application of FL in practical scenarios aiming to train a large-scale model using local data of clients given low computing powers and low transmission rates.
Split learning (SL) (Gupta & Raskar, 2018; Vepakomma et al., 2018; Thapa et al., 2020) is another recent approach for this setup, which can reduce the computation burden at the clients by splitting the model w into two parts: the first few layers (client-side model wC ) are allocated to the clients, and the remaining layers (server-side model wS) are allocated to the server.
Since each client only need to train the first few layers of the model, the computational burden at each client is reduced compared to FL.
However, existing SL-based ideas still have two critical issues in terms of latency and communication efficiency.
First, existing SL solutions still require significant time delay, since each participating client should wait for the backpropagated gradients from the server in order to update its model.
Moreover, the communication burden can still be substantial for transmitting the forward/backward signals via uplink/downlink communications at each global round.
Contributions: In this paper, we propose a fast and communication-efficient solution that provides a new direction to federated/split learning, by addressing the high latency requirement and high communication resource requirement of current FL and SL-based approaches. Motivated by the idea of local-loss-based training (Nøkland & Eidnes, 2019; Belilovsky et al., 2020), instead of considering the conventional loss function that is computed at the output of the model w, we introduce alternative local loss functions specifically geared to the split learning setup. We develop an algorithm where the client-side models can be updated without receiving the backpropagated signals from the server, significantly improving latency and communication efficiency. Fig. 1 compares our idea with FL and the state-of-the-art SL approach, termed SplitFed (Thapa et al., 2020). Our main contributions are summarized as follows: • We propose a new federated split learning algorithm that addresses the latency and communication efficiency issues of current FL and SL approaches, via local-loss-based training geared to split learning. • We provide latency analysis and provide an optimal solution on splitting the model to minimize the latency. We also provide theoretical analysis to guarantee convergence of our scheme. • Experimental results show that our approach outperforms existing FL and SL-based ideas in practice where clients having low computing powers and low transmission rates collaborate to train a global model.