提出新矩阵分解架构，让梯度下降稳定生成低秩解。
Implicit Bias in Matrix Factorization and its Explicit Realization in a New Architecture
- 用约束范数的因子加可变对角矩阵，显式实现低秩偏好
- 实验显示模型始终产出真正低秩解，而非近似解
- 适用于需要轻量低秩表示的回归与分类任务
矩阵分解的梯度下降表现出向近似低秩解的隐式偏差。尽管现有理论常假设迭代序列有界，但实际中即使序列无界，这种偏差依然存在。这反映出因子在增大幅度的同时发展出低秩结构，并趋向特定方向。为稳定捕捉该行为，我们提出新分解模型 $X\approx UDV^\top$：其中 $U$ 与 $V$ 被约束于范数球内，而 $D$ 为对角因子，使模型能覆盖整个搜索空间。实验表明该模型始终呈现强隐式偏差，生成真正（非近似）低秩解。将该思想扩展至神经网络，我们设计一种含约束层与对角组件的新模型，在多个回归与分类任务上达到竞争性性能，同时产生轻量、低秩表示。
原文摘要 · Abstract (English)
Gradient descent for matrix factorization exhibits an implicit bias toward approximately low-rank solutions. While existing theories often assume the boundedness of iterates, empirically the bias persists even with unbounded sequences. This reflects a dynamic where factors develop low-rank structure while their magnitudes increase, tending to align with certain directions. To capture this behavior in a stable way, we introduce a new factorization model: $X\approx UDV^\top$, where $U$ and $V$ are constrained within norm balls, while $D$ is a diagonal factor allowing the model to span the entire search space. Experiments show that this model consistently exhibits a strong implicit bias, yielding truly (rather than approximately) low-rank solutions. Extending the idea to neural networks, we introduce a new model featuring constrained layers and diagonal components that achieves competitive performance on various regression and classification tasks while producing lightweight, low-rank representations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。