arXiv:2409.15216cs.LGcs.CV2024-09

FLeNS用压缩海森矩阵加速联邦学习,通信量小且收敛快。

FLeNS: Federated Learning with Enhanced Nesterov-Newton Sketch

  • 结合奈斯特罗夫加速与自适应海森压缩,降低通信开销。
  • 理论证明通信轮次呈超线性收敛,比现有方法更快。
  • 适合隐私敏感、边缘计算等资源受限场景使用。

联邦学习在平衡通信效率与快速收敛方面面临挑战,尤其针对二阶方法。尽管牛顿类算法可在通信轮次中实现线性收敛,但传输完整的海森矩阵因二次复杂度而通常不切实际。我们提出联邦学习增强型奈斯特罗夫-牛顿压缩方法(FLeNS),同时利用奈斯特罗夫方法的加速能力与海森压缩的降维优势。FLeNS在无需精确海森矩阵的情况下逼近中心化牛顿法,显著降低通信开销。通过将奈斯特罗夫加速与自适应海森压缩结合,FLeNS保留了关键的二阶信息,同时保持快速收敛特性。理论分析基于统计学习,证明FLeNS在通信轮次上实现超线性收敛——这是联邦优化领域的显著进展。我们提供了严格的收敛保证,并刻画了加速、压缩尺寸与收敛速度之间的权衡关系。大量实验验证了理论结果,展示了FLeNS在减少通信需求方面的最先进性能,尤其适用于隐私敏感和边缘计算场景。代码已公开于https://github.com/sunnyinAI/FLeNS。

原文摘要 · Abstract (English)

Federated learning faces a critical challenge in balancing communication efficiency with rapid convergence, especially for second-order methods. While Newton-type algorithms achieve linear convergence in communication rounds, transmitting full Hessian matrices is often impractical due to quadratic complexity. We introduce Federated Learning with Enhanced Nesterov-Newton Sketch (FLeNS), a novel method that harnesses both the acceleration capabilities of Nesterov's method and the dimensionality reduction benefits of Hessian sketching. FLeNS approximates the centralized Newton's method without relying on the exact Hessian, significantly reducing communication overhead. By combining Nesterov's acceleration with adaptive Hessian sketching, FLeNS preserves crucial second-order information while preserving the rapid convergence characteristics. Our theoretical analysis, grounded in statistical learning, demonstrates that FLeNS achieves super-linear convergence rates in communication rounds - a notable advancement in federated optimization. We provide rigorous convergence guarantees and characterize tradeoffs between acceleration, sketch size, and convergence speed. Extensive empirical evaluation validates our theoretical findings, showcasing FLeNS's state-of-the-art performance with reduced communication requirements, particularly in privacy-sensitive and edge-computing scenarios. The code is available at https://github.com/sunnyinAI/FLeNS

联邦学习二阶优化通信压缩加速收敛

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。