解决分布式学习中的标签分布偏移问题，提升模型在数据隔离下的泛化能力。
Addressing Label Shift in Distributed Learning via Entropy Regularization
- 通过熵正则化优化测试与训练标签密度比的极大似然估计。
- 在不均衡设置下，相比基线方法性能提升最高达20%。
- 适用于数据无法集中、标签分布多变的分布式机器学习场景。
我们针对多节点分布式学习中最小化真实风险的挑战提出解决方案。此类系统常面临节点间与节点内标签分布偏移，严重阻碍模型性能优化，且需确保数据保留在各节点内。为此，我们提出通用鲁棒标签偏移（VRLS）方法，通过改进测试到训练标签密度比的最大似然估计来应对该问题。VRLS引入香农熵正则化，并在训练中动态调整密度比，以更好适应测试时的标签偏移。在多节点环境下，该方法还能跨节点学习并适应密度比，有效缓解标签偏移，提升整体性能。在MNIST、Fashion MNIST和CIFAR-10上的实验表明，该方法在数据不平衡设置下性能优于基线最多20%。理论分析进一步建立了估计误差的高概率界，验证了其有效性。
原文摘要 · Abstract (English)
We address the challenge of minimizing true risk in multi-node distributed learning. These systems are frequently exposed to both inter-node and intra-node label shifts, which present a critical obstacle to effectively optimizing model performance while ensuring that data remains confined to each node. To tackle this, we propose the Versatile Robust Label Shift (VRLS) method, which enhances the maximum likelihood estimation of the test-to-train label density ratio. VRLS incorporates Shannon entropy-based regularization and adjusts the density ratio during training to better handle label shifts at the test time. In multi-node learning environments, VRLS further extends its capabilities by learning and adapting density ratios across nodes, effectively mitigating label shifts and improving overall model performance. Experiments conducted on MNIST, Fashion MNIST, and CIFAR-10 demonstrate the effectiveness of VRLS, outperforming baselines by up to 20% in imbalanced settings. These results highlight the significant improvements VRLS offers in addressing label shifts. Our theoretical analysis further supports this by establishing high-probability bounds on estimation errors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。