提出新方法解决分布式优化中压缩通信下的复合优化难题
Composite Optimization with Error Feedback: the Dual Averaging Approach
- 将对偶平均与误差控制结合，改进传统误差反馈机制
- 首次实现复合优化下带压缩通信的强收敛性理论保证
- 适合研究分布式机器学习通信效率的学者和工程师
通信效率是分布式机器学习训练中的核心挑战，消息压缩是一种常用解决方案。然而，标准误差反馈（EF）方法尽管在平滑无约束优化中有效（Karimireddy等，2019），在更广泛且实际重要的复合优化场景下失效，该场景包含平滑损失与非平滑正则化项或约束的组合。现有研究尚未建立EF在一般复合设置下的理论基础。本文针对带误差反馈的复合优化问题，指出基础EF机制及其分析方法在引入复合部分后不再成立，根源在于方法本身的固有缺陷。为此，我们提出一种新算法，融合对偶平均与最新版误差控制（EControl, Gao等，2024），首次实现复合优化中误差反馈的强收敛性分析。同时，我们还提出一种新颖的不精确对偶平均分析框架，具有独立研究价值。实验结果验证了理论发现。
原文摘要 · Abstract (English)
Communication efficiency is a central challenge in distributed machine learning training, and message compression is a widely used solution. However, standard Error Feedback (EF) methods (Seide et al., 2014), though effective for smooth unconstrained optimization with compression (Karimireddy et al., 2019), fail in the broader and practically important setting of composite optimization, which captures, e.g., objectives consisting of a smooth loss combined with a non-smooth regularizer or constraints. The theoretical foundation and behavior of EF in the context of the general composite setting remain largely unexplored. In this work, we consider composite optimization with EF. We point out that the basic EF mechanism and its analysis no longer stand when a composite part is involved. We argue that this is because of a fundamental limitation in the method and its analysis technique. We propose a novel method that combines Dual Averaging with EControl (Gao et al., 2024), a state-of-the-art variant of the EF mechanism, and achieves for the first time a strong convergence analysis for composite optimization with error feedback. Along with our new algorithm, we also provide a new and novel analysis template for inexact dual averaging method, which might be of independent interest. We also provide experimental results to complement our theoretical findings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。