提出高效实现的联邦牛顿学习框架，训练速度提升千倍。
Unlocking FedNL: Self-Contained Compute-Optimized Implementation
- 自包含设计，支持单机与多机部署
- 实验启动时间从4.8小时缩短至毫秒级
- 适配资源受限场景，适合实际系统集成
联邦学习（FL）是一种新兴范式，使智能体在不共享本地数据的前提下协同训练机器学习模型。近期工作（arXiv:2106.02969）提出一类联邦牛顿学习（FedNL）算法，标志着二阶优化方法在联邦学习与大规模优化中的重要进展。然而，现有参考原型存在三大实践缺陷：(i) 在服务器级工作站中单次实验需4.8小时启动；(ii) 仅支持多节点模拟；(iii) 难以集成到资源受限应用。为弥合理论与实践差距，我们提出FedNL-LS与FedNL-PP，实现单机与多机场景下的自包含实现。本工作将墙时（wall clock time）降低1000倍。在单机场景下，该实现优于CVXPY（arXiv:1603.00943）；在多机场景下，优于Apache Spark（arXiv:1505.06807）与Ray/Scikit-Learn（arXiv:1712.05889）。最后，我们提出两种面向实际的压缩器——自适应TopLEK与缓存感知RandSeqK，满足FedNL理论要求。
原文摘要 · Abstract (English)
Federated Learning (FL) is an emerging paradigm that enables intelligent agents to collaboratively train Machine Learning (ML) models in a distributed manner, eliminating the need for sharing their local data. The recent work (arXiv:2106.02969) introduces a family of Federated Newton Learn (FedNL) algorithms, marking a significant step towards applying second-order methods to FL and large-scale optimization. However, the reference FedNL prototype exhibits three serious practical drawbacks: (i) It requires 4.8 hours to launch a single experiment in a sever-grade workstation; (ii) The prototype only simulates multi-node setting; (iii) Prototype integration into resource-constrained applications is challenging. To bridge the gap between theory and practice, we present a self-contained implementation of FedNL, FedNL-LS, FedNL-PP for single-node and multi-node settings. Our work resolves the aforementioned issues and reduces the wall clock time by x1000. With this FedNL outperforms alternatives for training logistic regression in a single-node -- CVXPY (arXiv:1603.00943), and in a multi-node -- Apache Spark (arXiv:1505.06807), Ray/Scikit-Learn (arXiv:1712.05889). Finally, we propose two practical-orientated compressors for FedNL - adaptive TopLEK and cache-aware RandSeqK, which fulfill the theory of FedNL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。