小旋转让Adam丢掉优势，新方法可恢复其学习复杂边界的能力
How do simple rotations affect the implicit bias of Adam?
- 通过重参数化使优化对数据旋转不变
- 微小旋转会导致Adam收敛到更差的线性边界
- 适合关注优化器泛化性能的研究者
自适应梯度方法如Adam和Adagrad在机器学习中广泛应用，但其对模型泛化能力的影响仍不明确。已有研究指出，对于二分类任务，Adam具有‘丰富性偏差’，能学习更接近贝叶斯最优决策边界的非线性边界，优于梯度下降。然而，Adam采用的坐标预条件机制使其对特征空间的正交变换敏感。本文发现，即使数据分布发生微小旋转，Adam也会丧失这一优势，转而收敛至远离贝叶斯最优的线性边界。为解决此问题，我们引入一种近期提出的重参数化方法——对优化目标施加正交变换，使任意一阶优化器具备对数据旋转的等变性，并通过实验验证该方法可有效恢复Adam对复杂决策边界的偏好。
原文摘要 · Abstract (English)
Adaptive gradient methods such as Adam and Adagrad are widely used in machine learning, yet their effect on the generalization of learned models -- relative to methods like gradient descent -- remains poorly understood. Prior work on binary classification suggests that Adam exhibits a ``richness bias,'' which can help it learn nonlinear decision boundaries closer to the Bayes-optimal decision boundary relative to gradient descent. However, the coordinate-wise preconditioning scheme employed by Adam renders the overall method sensitive to orthogonal transformations of feature space. We show that this sensitivity can manifest as a reversal of Adam's competitive advantage: even small rotations of the underlying data distribution can make Adam forfeit its richness bias and converge to a linear decision boundary that is farther from the Bayes-optimal decision boundary than the one learned by gradient descent. To alleviate this issue, we show that a recently proposed reparameterization method -- which applies an orthogonal transformation to the optimization objective -- endows any first-order method with equivariance to data rotations, and we empirically demonstrate its ability to restore Adam's bias towards rich decision boundaries.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。