通过动态控制分歧误差，让去中心化训练找到更平坦的解，提升模型泛化能力。
Decentralized SGD with Controlled Disagreement Finds Flatter Minima
- 用时变缩放机制调节各节点间共识误差，平衡收敛与泛化
- 在图像分类任务中测试准确率超越标准去中心化和集中式SGD
- 适合追求高泛化性能的去中心化机器学习系统设计者
去中心化训练常被认为劣于集中式训练，因工作节点间的共识误差会影响收敛与泛化。本文提出自适应共识去中心化SGD（DSGD-AC），采用随时间变化的缩放机制，在训练过程中维持共识误差。我们证明，自适应共识通过弱化图阻尼来保持误差幅值，同时仍允许基于曲率的阻尼调控分歧方向，从而在部署模型周围产生更强的海森加权损失包络惩罚，即使归一化海森对齐弱于标准DSGD。图像分类实验表明，DSGD-AC能获得更平坦的解，且测试准确率高于标准DSGD甚至集中式SGD。结果表明共识误差可作为有效的隐式正则项，为去中心化学习算法设计提供新视角。
原文摘要 · Abstract (English)
Decentralized training is often regarded as inferior to centralized training because the consensus errors between workers are thought to undermine convergence and generalization. This work challenges this view by introducing decentralized SGD with Adaptive Consensus (DSGD-AC), which uses a time-dependent scaling mechanism to maintain consensus errors throughout the training. We show that adaptive consensus changes the stationary variance of disagreement modes by balancing two effects: it preserves consensus-error magnitude through weaker graph damping while still allowing curvature-dependent damping to shape the disagreement directions. This balance can produce a stronger Hessian-weighted loss-envelope penalty around the deployed model, even when normalized Hessian alignment is weaker than in standard DSGD. Empirical results on image classification show that DSGD-AC reaches flatter solutions and higher test accuracy than standard DSGD and even centralized SGD. Together, these results support consensus errors as a useful implicit regularizer and open a new perspective on the design of decentralized learning algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。