提出HAM方法，让模型更稀疏且训练更快。
Hyperbolic Aware Minimization: Implicit Bias for Sparsity
- 用轻量镜像步替代过度参数化，保持双曲几何优势
- 实验显示在视觉任务中提升性能，即使训练密集也有效
- 适合需要稀疏化与高效训练的深度学习场景
理解优化算法的隐式偏差对解释和提升深度模型泛化能力至关重要。点式过度参数化引发的双曲隐式偏差虽促进稀疏性，但在零附近产生小的逆黎曼度量，减缓参数更新并阻碍有意义的符号翻转。为克服此瓶颈，我们提出双曲感知最小化（HAM），其交替执行标准优化步骤与轻量级双曲镜像步。该镜像步计算与内存开销低于点式过度参数化，重现其有益的双曲几何以支持特征学习，同时缓解小逆度量瓶颈。我们在欠定线性回归背景下对隐式偏差的分析揭示了HAM持续提升性能的机制——即便在密集训练情况下，我们在标准视觉基准上已验证其有效性。HAM与多种稀疏化方法结合尤为有效，推动了当前技术水平。
原文摘要 · Abstract (English)
Understanding the implicit bias of optimization algorithms is key to explaining and improving the generalization of deep models. The hyperbolic implicit bias induced by pointwise overparameterization promotes sparsity, but also yields a small inverse Riemannian metric near zero, slowing down parameter movement and impeding meaningful parameter sign flips. To overcome this obstacle, we propose Hyperbolic Aware Minimization (HAM), which alternates a standard optimizer step with a lightweight hyperbolic mirror step. The mirror step incurs less compute and memory than pointwise overparameterization, reproduces its beneficial hyperbolic geometry for feature learning, and mitigates the small-inverse-metric bottleneck. Our characterization of the implicit bias in the context of underdetermined linear regression provides insights into the mechanism how HAM consistently increases performance --even in the case of dense training, as we demonstrate in experiments with standard vision benchmarks. HAM is especially effective in combination with different sparsification methods, advancing the state of the art.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。