揭示镜流优化如何影响神经网络的稀疏与密集特征学习
Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learning
- 基于对偶理论推导镜流平衡方程，刻画边际增长机制
- 发现收敛可极慢，甚至呈指数级；不同镜映射产生迥异表征
- 统一解释稀疏/密集特征学习，适合研究优化几何的学者
我们研究具有齐次激活函数的深度神经网络中镜流达到的最大边缘解。在经典梯度流结果基础上，通过凸对偶推导出镜流的新平衡方程，从而刻画决定边际的尖端函数。进一步建立了最大边缘特性、收敛速率与范数增长估计。实验验证于合成数据集和标准视觉任务：（1）不同的非齐次镜映射可诱导相同的最大边缘解；（2）收敛可能极慢，包括指数级慢速；（3）尽管所有镜映射均实现特征学习，但产生的表示差异显著，从稀疏到密集神经元激活不等。这些结果为齐次神经网络中的稀疏与密集特征学习提供了统一视角，凸显镜映射对优化动态与分类器几何结构的影响。
原文摘要 · Abstract (English)
We study the max-margin solutions reached by mirror flow in deep neural networks with homogeneous activation functions. Extending classical results on gradient flow, we derive a novel balance equation for mirror flow from convex duality, enabling a characterization of the horizon function governing the induced margin. We further establish max-margin characterizations together with convergence rates and norm growth estimates. Finally, we support our theory through experiments on synthetic datasets and standard vision tasks. Concretely, we show that: (1) distinct non-homogeneous mirror maps can induce the same max-margin solution; (2) convergence can be extremely slow, including exponentially slow regimes; and (3) although all considered mirror maps exhibit feature learning, they can produce markedly different representations, ranging from sparse to dense neuron activations. Together, these results provide a unified perspective on sparse and dense feature learning in homogeneous neural networks, highlighting how mirror maps shape both optimization dynamics and the geometry of the learned classifiers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。