研究优化算法如何影响多分类数据的解结构，揭示低秩约束下的隐式偏好。
Towards The Implicit Bias on Multiclass Separable Data Under Norm Constraints
- 提出NucGD优化器，通过核范数约束实现低秩解。
- 证明在小批量采样下，噪声会引导收敛至最大间隔解。
- 设计无需SVD的高效更新规则，适合大规模训练。
梯度优化算法引发的隐式偏差对过参数化模型的泛化能力至关重要，但其机制可能十分微妙。本文基于归一化最陡下降（NSD）框架，研究优化几何如何影响多分类可分数据上的解结构。提出一种几何感知的优化器NucGD，通过核范数约束强制解具有低秩特性。除算法本身外，还将NucGD与新兴的低秩投影方法统一关联。为支持可扩展训练，通过异步幂迭代推导出高效的无SVD更新规则。此外，通过实验剖析随机优化动态的影响，揭示小批量采样和动量引入的梯度噪声如何调节收敛至期望的最大间隔解的过程。代码已开源：https://github.com/Tsokarsic/observing-the-implicit-bias-on-multiclass-seperable-data。
原文摘要 · Abstract (English)
Implicit bias induced by gradient-based algorithms is essential to the generalization of overparameterized models, yet its mechanisms can be subtle. This work leverages the Normalized Steepest Descent} (NSD) framework to investigate how optimization geometry shapes solutions on multiclass separable data. We introduce NucGD, a geometry-aware optimizer designed to enforce low rank structures through nuclear norm constraints. Beyond the algorithm itself, we connect NucGD with emerging low-rank projection methods, providing a unified perspective. To enable scalable training, we derive an efficient SVD-free update rule via asynchronous power iteration. Furthermore, we empirically dissect the impact of stochastic optimization dynamics, characterizing how varying levels of gradient noise induced by mini-batch sampling and momentum modulate the convergence toward the expected maximum margin solutions.Our code is accessible at: https://github.com/Tsokarsic/observing-the-implicit-bias-on-multiclass-seperable-data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。