揭示监督学习中特征表示的低秩压缩机制。
Task-Driven Kernel Flows: Label Rank Compression and Laplacian Spectral Filtering
- 通过核微分方程建模特征学习，发现谱演化具水填特性。
- 证明稳定状态下核矩阵秩不超过类别数 $C$，实现压缩。
- 解释梯度噪声也呈低秩（$O(C)$），适合任务相关学习者。
我们提出了宽L2正则化网络中特征学习的理论，表明监督学习本质上具有压缩性。推导出预测'水填'谱演化的核常微分方程，并证明任意稳定稳态下，核秩被限制在类别数 $C$ 内。进一步证明，SGD噪声同样为低秩（$O(C)$），将动态限制在任务相关子空间中。该框架统一了确定性和随机性对齐视角，凸显监督学习的低秩特性与自监督学习高秩、扩张表示的本质差异。
原文摘要 · Abstract (English)
We present a theory of feature learning in wide L2-regularized networks showing that supervised learning is inherently compressive. We derive a kernel ODE that predicts a "water-filling" spectral evolution and prove that for any stable steady state, the kernel rank is bounded by the number of classes ($C$). We further demonstrate that SGD noise is similarly low-rank ($O(C)$), confining dynamics to the task-relevant subspace. This framework unifies the deterministic and stochastic views of alignment and contrasts the low-rank nature of supervised learning with the high-rank, expansive representations of self-supervision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。