Adam的性能受参数空间旋转影响，传统理论解释不足。
Understanding Adam Requires Better Rotation Dependent Assumptions
- 发现Adam对参数空间旋转敏感，随机旋转会降低其性能。
- 验证更新方向正交性是衡量Adam基底敏感性的关键指标。
- 适合研究优化器理论或改进自适应方法的读者。
尽管广泛使用，但对Adam相较于随机梯度下降（SGD）的优势尚无全面的理论解释。本文研究了Adam在参数空间旋转下的敏感性，发现随机旋转会显著降低其在训练Transformer时的性能，表明其对基底选择具有关键依赖性。这说明传统的旋转不变假设无法充分解释Adam的优势。我们进一步识别出能保持甚至提升其性能的结构化旋转，并检验了文献中常见的旋转依赖性假设，发现它们难以解释Adam在不同旋转类型下的表现。相比之下，我们验证了更新方向的正交性作为衡量基底敏感性的有效指标，提示其可能是构建更准确旋转依赖性理论框架的核心要素。
原文摘要 · Abstract (English)
Despite its widespread adoption, Adam's advantage over Stochastic Gradient Descent (SGD) lacks a comprehensive theoretical explanation. This paper investigates Adam's sensitivity to rotations of the parameter space. We observe that Adam's performance in training transformers degrades under random rotations of the parameter space, indicating a crucial sensitivity to the choice of basis in practice. This reveals that conventional rotation-invariant assumptions are insufficient to capture Adam's advantages theoretically. To better understand the rotation-dependent properties that benefit Adam, we also identify structured rotations that preserve or even enhance its empirical performance. We then examine the rotation-dependent assumptions in the literature and find that they fall short in explaining Adam's behaviour across various rotation types. In contrast, we verify the orthogonality of the update as a promising indicator of Adam's basis sensitivity, suggesting it may be the key quantity for developing rotation-dependent theoretical frameworks that better explain its empirical success.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。