arXiv:2505.21791stat.MLcs.LG2025-05NeurIPS

用ℓ^p正则化训练,可直接得到最稀疏的ReLU网络。

Global Minimizers of $\ell^p$-Regularized Objectives Yield the Sparsest ReLU Neural Networks

  • 通过最小化ℓ^p权重范数(0 < p < 1)构造连续可导目标函数
  • 全局最优解恰好对应最稀疏的单层ReLU神经网络
  • 为梯度方法实现稀疏网络训练提供理论基础,适合研究模型压缩

过参数化神经网络能以多种方式拟合数据,这引发了一个根本问题:应选择哪种解?何种显式正则化策略能保证获得该解?本文解决寻找最稀疏插值ReLU网络的问题——即具有最少非零参数或神经元的网络,这一目标在效率、泛化、可解释性、理论分析和模型压缩方面具有广泛意义。不同于后期剪枝方法,本文提出一种连续、几乎处处可导的训练目标函数,其全局极小值被证明对应于能够拟合数据的最稀疏单隐层ReLU网络。这一结果实现了概念性突破:将稀疏插值的组合问题转化为光滑优化任务,可能使梯度方法得以应用。该目标基于在0 < p < 1下最小化权重的ℓ^p伪范数,这是有限维情形中经典的促稀疏策略。然而,将其应用于神经网络面临新挑战:函数类是无限维的,且权重通过高度非凸目标学习。本文证明,在该公式下,全局极小值与最稀疏解精确对应。这项工作为理解何时以及如何通过训练利用连续稀疏诱导目标恢复稀疏网络奠定了基础。

原文摘要 · Abstract (English)

Overparameterized neural networks can interpolate a given dataset in many different ways, prompting the fundamental question: which among these solutions should we prefer, and what explicit regularization strategies will provably yield these solutions? This paper addresses the challenge of finding the sparsest interpolating ReLU network--i.e., the network with the fewest nonzero parameters or neurons--a goal with wide-ranging implications for efficiency, generalization, interpretability, theory, and model compression. Unlike post hoc pruning approaches, we propose a continuous, almost-everywhere differentiable training objective whose global minima are guaranteed to correspond to the sparsest single-hidden-layer ReLU networks that fit the data. This result marks a conceptual advance: it recasts the combinatorial problem of sparse interpolation as a smooth optimization task, potentially enabling the use of gradient-based training methods. Our objective is based on minimizing $\ell^p$ quasinorms of the weights for $0 < p < 1$, a classical sparsity-promoting strategy in finite-dimensional settings. However, applying these ideas to neural networks presents new challenges: the function class is infinite-dimensional, and the weights are learned using a highly nonconvex objective. We prove that, under our formulation, global minimizers correspond exactly to sparsest solutions. Our work lays a foundation for understanding when and how continuous sparsity-inducing objectives can be leveraged to recover sparse networks through training.

稀疏神经网络ℓ^p正则化可训练稀疏性理论分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。