提出可处理分组样本加权的自适应优化算法，提升模型鲁棒性。
Aligning Distributionally Robust Optimization with Practical Deep Learning Needs
- 基于改进的分布鲁棒优化，实现对样本组的自适应加权。
- 在表格学习与分割学习等任务中均优于传统优化器和现有DRO方法。
- 适用于需要群体加权的实际深度学习场景，如类别级样本处理。
传统深度学习优化方法对所有训练样本一视同仁，而分布鲁棒优化（DRO）则自适应地为不同样本分配重要性权重。然而，当前DRO方法与实际深度学习实践之间存在显著差距：现代优化器需具备自适应性和处理随机梯度的能力，且应支持对单个样本及样本组（如同一类的所有样本）进行加权。本文提出ALSO——自适应损失缩放优化器，一种针对修改后的DRO目标的自适应算法，可实现对样本组的加权。我们证明了该算法在非凸目标下的收敛性，这正是深度学习模型的典型情况。在从表格深度学习到分割学习等多种任务上的实证评估表明，ALSO在性能上优于传统优化器和现有DRO方法。
原文摘要 · Abstract (English)
While traditional Deep Learning (DL) optimization methods treat all training samples equally, Distributionally Robust Optimization (DRO) adaptively assigns importance weights to different samples. However, a significant gap exists between DRO and current DL practices. Modern DL optimizers require adaptivity and the ability to handle stochastic gradients, as these methods demonstrate superior performance. Additionally, for practical applications, a method should allow weight assignment not only to individual samples, but also to groups of objects (for example, all samples of the same class). This paper aims to bridge this gap by introducing ALSO $\unicode{x2013}$ Adaptive Loss Scaling Optimizer $\unicode{x2013}$ an adaptive algorithm for a modified DRO objective that can handle weight assignment to sample groups. We prove the convergence of our proposed algorithm for non-convex objectives, which is the typical case for DL models. Empirical evaluation across diverse Deep Learning tasks, from Tabular DL to Split Learning tasks, demonstrates that ALSO outperforms both traditional optimizers and existing DRO methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。