根据优化曲率动态调整同步频率，大幅减少大模型训练通信开销。
CurvaDion: Curvature-Adaptive Distributed Orthonormalization
- 用动量变化检测曲率高低，决定是否同步
- 在160M到1.3B参数模型上实现99%通信减少
- 无需额外计算，适合大规模语言模型训练
随着语言模型参数量达万亿级别，跨多张GPU的分布式训练成为必要，但梯度同步仍受限于高带宽、低延迟网络。尽管近期方法如Dion通过低秩更新减少了每步通信量，但仍每步同步，未考虑优化路径的差异。我们发现：平坦区域中各工作节点梯度相近，频繁同步冗余；而高曲率区域需协调以避免发散。为此提出CurvaDion，利用相对最大动量变化（RMMC）检测需同步的高曲率区域。RMMC基于优化中已计算的动量动态，作为方向曲率的计算高效代理，每层仅增加$/mathcal{O}(d)$操作。理论上建立了RMMC与损失曲率的联系，并证明CurvaDion在160M至1.3B参数模型上实现99%通信减少，且收敛性能与基线相当。
原文摘要 · Abstract (English)
As language models scale to trillions of parameters, distributed training across many GPUs becomes essential, yet gradient synchronization over high-bandwidth, low-latency networks remains a critical bottleneck. While recent methods like Dion reduce per-step communication through low-rank updates, they synchronize at every step regardless of the optimization landscape. We observe that synchronization requirements vary dramatically throughout training: workers naturally compute similar gradients in flat regions, making frequent synchronization redundant, while high-curvature regions require coordination to prevent divergence. We introduce CurvaDion, which uses Relative Maximum Momentum Change (RMMC) to detect high-curvature regions requiring synchronization. RMMC leverages momentum dynamics which are already computed during optimization as a computationally tractable proxy for directional curvature, adding only $\mathcal{O}(d)$ operations per layer. We establish theoretical connections between RMMC and loss curvature and demonstrate that CurvaDion achieves 99\% communication reduction while matching baseline convergence across models from 160M to 1.3B parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。