arXiv:2605.13612cs.LGcond-mat.dis-nn2026-05

提出神经低阶滤波模型,揭示深度网络分层特征学习的谱机制。

Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning

论文配图:Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning
图 1 · 摘自论文原文
  • 将梯度训练简化为分层低阶相关性筛选,逐层选择与标签相关性高的方向。
  • 在真实数据上预测出与早期梯度下降发现一致的结构化表示,且优于随机特征基线。
  • 适合研究深度网络表征演化、特征构成机制的理论与实验研究者。

理解深度神经网络如何从数据中学习有用内部表示,仍是深度学习理论的核心难题。本文提出神经低阶滤波(Neural LoFi),一种基于梯度训练的简化极限,使分层特征学习成为明确的迭代谱过程。在此极限下,各层动态解耦:给定当前表示后,下一层选择与标签具有最大可访问低阶相关性的方向。该模型提供了一个可处理的深度学习替代机制,并具有自然的核空间解释。Neural LoFi 为懒惰区域之外的多层特征学习提供了数学明确的框架,可预测逐层表示的选择方式,解释概念在特定样本复杂度下的涌现,并给出深度通过低阶组合性从旧特征构建新特征的具体机制。我们通过全连接和卷积架构的机制实验验证了该理论,结果表明其优于懒惰随机特征基线,能恢复有意义的结构化滤波器,并准确预测真实数据上早期梯度下降发现的表示。

原文摘要 · Abstract (English)

Understanding how deep neural networks learn useful internal representations from data remains a central open problem in the theory of deep learning. We introduce Neural Low-Degree Filtering (Neural LoFi), a stylized limit of gradient-based training in which hierarchical feature learning becomes an explicit iterative spectral procedure. In this limit, the dynamics at each layer decouple: given the current representation, the next layer selects directions with maximal accessible low-degree correlation to the label. This yields a tractable surrogate mechanism for deep learning, together with a natural kernel-space interpretation. Neural LoFi provides a mathematically explicit framework for studying multi-layer feature learning beyond the lazy regime. It predicts how representations are selected layer by layer, explains how emergence of concepts arises with given sample complexity,and gives a concrete mechanism by which depth progressively constructs new features from old ones through low-degree compositionality. We complement the theory with mechanistic experiments on fully connected and convolutional architectures, showing that Neural LoFi improves over lazy random-feature baselines, recovers meaningful structured filters, and predicts representations aligned with early gradient-descent feature discovery with real datasets.

深度学习理论特征学习谱分析神经网络机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。