利用权重空间对称性，用单次梯度估算神经网络曲率，更高效准确。
Exploiting weight-space symmetries for approximating curvature
- 通过分析损失函数的对称性，仅用一次梯度就构造出结构化海森矩阵近似
- 在多种网络架构上验证，比现有方法更快且精度更高，支持小语言模型优化
- 框架可统一解释已有方法，适合做二阶优化、模型压缩和不确定性估计
许多机器学习技术依赖于损失函数曲率的近似，但在现代深度网络规模下仍极难实现。然而，以往研究从未利用损失景观中已知的权重空间对称性带来的曲率约束。本文通过解析地对保持损失不变的群作用进行平均，仅需单个梯度即可构建可计算、可存储、可求逆的结构化海森矩阵近似。用户指定的对称群类型直接控制近似精度与计算成本之间的权衡。此外，该框架为理解现有方法提供了统一的理论视角；特定对称群选择可恢复 Shampoo/Muon 类似曲率估计。我们在多种网络架构上验证了该方法，并应用于二阶优化基准，包括一个小语言模型。该曲率估计框架还可推广至不确定性估计、持续学习、压缩/剪枝、训练数据溯源等任务。
原文摘要 · Abstract (English)
Many machine learning techniques rely on approximating a loss function's curvature, but this is notoriously hard to do at the scale of modern deep networks. Surprisingly, no previous work has exploited the curvature constraints that arise from well known weight-space symmetries in loss landscapes. By analytically averaging over group actions that leave the loss invariant, we construct structured Hessian approximations from single gradients that can be tractably estimated, stored, and inverted. The choice of user-specified symmetry group directly governs the trade-off between approximation accuracy and computational cost. Moreover, our framework provides a unifying theoretical lens for viewing existing methods; in particular, a specific choice of symmetry group recovers Shampoo/Muon-like curvature estimates. We validate our method on a range of network architectures, and deploy it to second-order optimization benchmarks, including a small language model. Our curvature estimation framework might find applications in other machine learning problems such as uncertainty estimation, continual learning, compression/pruning, training data attribution, and more.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。