通过近似梯度反馈，让客户端在低内存下高效训练大模型。
FSL-SAGE: Accelerating Federated Split Learning via Smashed Activation Gradient Estimation
- 用辅助模型模拟服务器梯度，避免逐轮通信
- 收敛速度达 $\mathcal{O}(1/\sqrt{T})$，与 FedAvg 相当
- 适合资源受限设备上的分布式训练，兼顾效率与精度
联邦学习（FL）和分片学习（SL）等协作训练方法可在不共享原始数据的情况下实现分布式机器学习。然而，FL要求客户端能训练完整模型，对大规模模型不可行；而SL虽通过将大部分训练任务移至服务器缓解了客户端内存压力，但其串行特性导致网络延迟增加。现有方法尝试通过本地损失函数实现客户端并行训练以提升效率，但缺乏服务器反馈，可能影响准确率。本文提出FSL-SAGE（基于击碎激活梯度估计的联邦分片学习），通过辅助模型周期性地适应本地数据，模拟服务器行为来估算梯度反馈。我们证明该方法收敛速率为 $\mathcal{O}(1/\sqrt{T})$，与 FedAvg 一致，同时显著降低通信开销与客户端内存需求。实验表明，FSL-SAGE 在多项指标上优于现有最先进的FSL方法，兼具通信效率与高精度。
原文摘要 · Abstract (English)
Collaborative training methods like Federated Learning (FL) and Split Learning (SL) enable distributed machine learning without sharing raw data. However, FL assumes clients can train entire models, which is infeasible for large-scale models. In contrast, while SL alleviates the client memory constraint in FL by offloading most training to the server, it increases network latency due to its sequential nature. Other methods address the conundrum by using local loss functions for parallel client-side training to improve efficiency, but they lack server feedback and potentially suffer poor accuracy. We propose FSL-SAGE (Federated Split Learning via Smashed Activation Gradient Estimation), a new federated split learning algorithm that estimates server-side gradient feedback via auxiliary models. These auxiliary models periodically adapt to emulate server behavior on local datasets. We show that FSL-SAGE achieves a convergence rate of $\mathcal{O}(1/\sqrt{T})$, where $T$ is the number of communication rounds. This result matches FedAvg, while significantly reducing communication costs and client memory requirements. Our empirical results also verify that it outperforms existing state-of-the-art FSL methods, offering both communication efficiency and accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。