提出首个理论支持的加速分布式优化算法，适配数据相似场景。
Accelerated Methods with Compressed Communications for Distributed Optimization Problems under Data Similarity
- 结合无偏/有偏压缩与本地迭代，利用方差缩减与误差反馈。
- 在多种数据集上实现更低平均损失，性能创纪录。
- 适合数据分布相似的分布式训练场景，如联邦学习。
近年来，随着数据和问题规模增大，分布式学习已成为训练高性能模型的关键工具。然而，通信瓶颈（尤其在高维数据下）仍是主要挑战。已有技术包括通信压缩和局部更新，当本地数据样本具有相似性时效果更佳。本文研究这些方法的协同效应，提出首个理论支持的加速算法，在数据相似条件下同时使用无偏与有偏压缩，结合方差缩减与误差反馈机制。实验结果验证了其在不同平均损失与数据集上的卓越性能，达到当前最优记录。
原文摘要 · Abstract (English)
In recent years, as data and problem sizes have increased, distributed learning has become an essential tool for training high-performance models. However, the communication bottleneck, especially for high-dimensional data, is a challenge. Several techniques have been developed to overcome this problem. These include communication compression and implementation of local steps, which work particularly well when there is similarity of local data samples. In this paper, we study the synergy of these approaches for efficient distributed optimization. We propose the first theoretically grounded accelerated algorithms utilizing unbiased and biased compression under data similarity, leveraging variance reduction and error feedback frameworks. Our results are of record and confirmed by experiments on different average losses and datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。