提升分布式学习通信效率，兼顾隐私与性能。
Strategies for Improving Communication Efficiency in Distributed and Federated Learning: Compression, Local Training, and Personalization
- 提出统一压缩框架，保证收敛性并降低通信开销。
- Scafflix平衡全局与个性化目标，加速收敛且适应非独立同分布数据。
- 引入隐私保护剪枝方法，减少设备间通信，适合大规模模型部署。
分布式和联邦学习是基于去中心化数据训练模型并保障隐私的关键范式，但通信开销仍是主要瓶颈。本文聚焦模型压缩、本地训练与个性化策略，建立兼具偏差与无偏压缩算子的统一框架，并提供收敛性保证；提出自适应本地训练策略，结合个性化缓解客户端漂移问题。其中，Scafflix 在独立同分布（IID）与非独立同分布（non-IID）场景下均表现优异。进一步设计隐私保护剪枝框架，通过层级聚合优化稀疏性并最小化通信成本，如 Cohort-Squeeze 有效降低跨设备通信量。最后，对称后训练剪枝方法 SymWanda 在高稀疏度下仍保持鲁棒性与精度，无需重新训练。在基准数据集与大规模语言模型上的大量实验表明，该方法在准确率、收敛速度与通信开销之间取得良好权衡，为可扩展、高效的分布式学习提供理论与实践洞见。
原文摘要 · Abstract (English)
Distributed and federated learning are essential paradigms for training models across decentralized data sources while preserving privacy, yet communication overhead remains a major bottleneck. This dissertation explores strategies to improve communication efficiency, focusing on model compression, local training, and personalization. We establish a unified framework for biased and unbiased compression operators with convergence guarantees, then propose adaptive local training strategies that incorporate personalization to accelerate convergence and mitigate client drift. In particular, Scafflix balances global and personalized objectives, achieving superior performance under both IID and non-IID settings. We further introduce privacy-preserving pruning frameworks that optimize sparsity while minimizing communication costs, with Cohort-Squeeze leveraging hierarchical aggregation to reduce cross-device overhead. Finally, SymWanda, a symmetric post-training pruning method, enhances robustness under high sparsity and maintains accuracy without retraining. Extensive experiments on benchmarks and large-scale language models demonstrate favorable trade-offs among accuracy, convergence, and communication, offering theoretical and practical insights for scalable, efficient distributed learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。